METHOD AND SYSTEM FOR AUTOMATIC ANNOTATION OF SENSOR DATA

Description

Field of invention

[0001] The present invention relates to methods and computer systems for automatically annotating sensor data frames, in particular data frames from an image acquisition sensor. background

[0002] Autonomous driving promises unprecedented levels of comfort and safety in everyday traffic. Despite enormous investments by various companies, existing approaches are only applicable under limited conditions and / or only provide for a subset of truly autonomous behavior. One reason for this is the lack of a sufficient quantity and variety of available driving scenarios. Thus, further progress is limited by the need for enormous amounts of sufficiently diverse training data as well as validation data (i.e., independent ground-truth data). Preparing training data generally requires recording many different driving scenarios by a vehicle equipped with a set of sensors, particularly image acquisition sensors such as one or more cameras, a lidar sensor, and / or a radar sensor. Before using these recorded scenarios as training data, they must be annotated.

[0003] This is often done by annotation service providers who receive the recorded sensor data and divide it into work packages for a variety of human workers, also known as labelers. The exact annotations required (e.g., the object classes to be distinguished) depend on each project and are specified in the detailed labeling specification. The client provides the raw data to the annotation service provider and expects high-quality annotations according to their specifications within a short timeframe. The number of labelers required to complete the annotation project increases with the amount of data supplied and also increases with the timeframe for a fixed data set decreasing. For this reason, larger annotation projects, which, for example,Providing enough ground truth data to validate an autonomous vehicle would not be feasible with human labor alone, but would require automation of the annotation process.

[0004] Automation approaches use neural networks to label the recorded sensor data. An initial set of received data is manually labeled and then used to train dedicated neural networks. Once sufficiently trained, the dedicated neural networks can annotate the mass of recorded image acquisition sensor data.

[0005] From the document "Vehicle Type Recognition from Surveillance Data Based on Deep Active Learning" by Xinghao Ding et al., a method for training a neural network is known in which images are incrementally selected based on an entropy or a confidence measure, manually labeled and used for training.

[0006] Compared to a purely manual approach, this significantly reduces the effort. However, maintaining high annotation quality still requires time-consuming quality checks by people. Since the quality assurance process must still be applied to all annotations, there is a linear relationship between the project volume and the workload required to meet project requirements.

[0007] Therefore, improved methods for the automatic annotation of sensor data, especially image acquisition sensor data, are needed; it would be particularly desirable to ensure high annotation quality with a reduced number of manual quality checks. Summary of the invention

[0008] The invention is defined in the appended claims. An objective of the present invention is to provide methods and computer systems for the automatic annotation of sensor data frames, in particular video frames or lidar point clouds.

[0009] In a first aspect of the invention, a computer-implemented method for automatically annotating sensor data frames is provided; the method comprises Receiving a multitude of sensor data frames, grouping the frames into a multitude of packets based on at least one condition attribute, wherein the condition attribute describes an environmental condition that existed during the recording of the sensor data frame, annotating the frames from a first packet using a neural network, wherein the annotation includes assigning at least one data point to each frame, the first packet comprising frames for which the at least one condition attribute is within a selected range of values, selecting a first sample of one or more frames from the first packet and determining a quality measure for the data points, wherein, if the computer determines that the quality measure for at least one frame in the first sample is below a predefined threshold, the procedure further comprises receiving corrected annotations for the frames in the first sample.Retraining the neural network based on the frames in the first sample, selecting a second sample of one or more frames from the frames of the first package that were not included in the first sample, annotating the frames of the second sample with the retrained neural network, receiving a quality measure for the data points and ensuring that the quality measure for the frames in the second sample is above a predefined threshold, annotating the remaining frames of the first package with the retrained neural network, and exporting the annotated frames of the first package.

[0010] The host computer can be a single, standard computer comprising a processor, such as a general-purpose microprocessor, a display device, and an input device. Alternatively, the host computer system can comprise one or more servers containing a variety of processing elements, with the servers connected via a network to a client comprising a display device and an input device. Thus, the annotation software can run partially or entirely on a remote server, such as a cloud computing environment, so that only a graphical user interface needs to run locally. Exporting the annotated frames can, for example, involve saving the frames to an external storage device and / or converting or combining them into a predefined data format.

[0011] By grouping sensor data frames based on a condition attribute that describes environmental conditions present at the time of recording, potential correlations between the condition attribute and annotation accuracy can be considered. An environmental condition that existed while the frame was being recorded can influence the accuracy of the annotations. For annotations encompassing multiple data points, the influence can vary depending on the data point. If the sensor data includes camera images captured at night, the position and / or class of an object may be more difficult to determine. However, an attribute of a car, such as the state of a warning light, may be more easily perceived than in full daylight.The invention makes it possible to identify environmental conditions that impair annotation accuracy and to improve the neural network under these conditions through selective retraining. Since the retraining is specifically targeted at problematic environmental conditions, the overall training effort is reduced. This further reduces the computing power required for training and thus also the energy consumption.

[0012] The term "neural network" can refer to a single neural network, a combination of different neural networks according to a predefined architecture, or any type of machine learning-based technique that learns from sample data in a supervised, semi-supervised, or unsupervised manner. Different neural networks can be used for different data points; the object's position and / or classification can be determined with a first neural network, while attributes of the object can be determined with at least one other neural network.

[0013] Because manual work is only used to create training, test, and / or validation data for systematically improving the neural network or other machine learning-based automation component for frame annotation, the effort required for large annotation projects can be considerably reduced. Typically, after a few iterations of retraining the neural network, the quality level is sufficient to deliver automation results, i.e., annotations by the neural network, without further manual checks. The method according to the invention further reduces the necessary manual effort and time by focusing the retraining on those conditions where annotation quality is still lacking.

[0014] As a quality measure, for example, the area coverage between an automatically generated bounding box and a bounding box created manually as part of quality control can be used. A maximum number and / or a maximum proportion of incorrectly assigned object classes and / or false positives and / or false negatives can also be required. The quality measure would then be below the predefined threshold, for example, if the bounding boxes have insufficient coverage. Another quality measure could be that a sample of a given number of frames may contain a maximum of a specified number of false positives (incorrectly identified objects) and / or false negatives (incorrectly unidentified objects). The quality measure would then be below the predefined threshold, for example, if the maximum allowed number of unidentified objects in the sample was exceeded.

[0015] The steps of selecting a second sample of frames from the first package and annotating the remaining frames of the first package with the retrained network can be reversed. For example, all remaining frames of the first package can be annotated with the retrained network before a second sample is selected. Annotating only the frames of the second sample with the retrained network and postponing the annotation of further frames until sufficient annotation quality is ensured reduces the computational effort in cases where the neural network needs to be retrained more than once, thus accelerating the retraining and annotation process.

[0016] In one embodiment, the received sensor data comprises frames from at least one image acquisition sensor, such as one or more cameras, a lidar sensor, and / or a radar sensor. The received sensor data may also include additional sensor data recorded simultaneously with the image acquisition sensor data, such as GPS position, vehicle acceleration, or data from a rain sensor. For image frames, i.e., frames containing image data or data frames from an image acquisition sensor, the condition attribute is preferably a geographic location, time of day, weather condition, visibility condition, road type, distance to an object, and / or traffic density.The distance to an object can be the distance to the nearest object, the distance to the furthest object, or the average distance to a multitude of objects detected in the frame. By considering the distance of an object as an environmental condition during recording, its influence on the object detection and / or classification performance of a neural network can be quantified. For an image frame, the at least one data point preferably comprises the position of an object, the class of an object, the position of an edge of a bounding frame, the degree of an object's coverage by other objects, a correlation of an object in the image frame with an object in a preceding or subsequent image frame (as a result of tracking the object), and / or the activation of a light indicator, such as a turn signal or brake light.The number of data points can depend on the content of the image frame, for example, many cars and pedestrians in a big city scene with a corresponding number of object positions, object classifications, and possible attributes for the corresponding object class.

[0017] In one embodiment, the received sensor data comprises audio frames recorded by at least one microphone. For audio frames, i.e., frames containing audio data, the condition attribute is preferably a geographical location, the gender and / or age of a speaker who was recorded, a room size, and / or a level of background noise. For audio frames, the at least one data point comprises one or more words of text recognized from the audio frame. Words can be recognized from a multitude of subsequent audio frames, so that a data point can be derived from a multitude of audio frames. The difficulty in recognizing speech can depend, for example, on the frequency range produced by a speaker, the presence of reverberation or echo from the room, and / or the level of existing background noise.

[0018] Preferably, the step of receiving a multitude of sensor data frames includes a preprocessing step of the frames, wherein at least one of the condition attributes for a frame is determined by a dedicated neural network based on the frame, and / or at least one of the condition attributes for a frame is determined based on additional sensor data recorded concurrently with the frames. The additional sensor data can be combined and / or used for queries to various services that specify, for example, weather conditions or a type of lighting condition based on time and geographic location.

[0019] In one embodiment, the first sample comprises two or more frames selected from the first package. Preferably, once the computer determines that the quality measure for the first sample is below the predefined threshold, no further calculations are performed on the frames from the first package until corrected annotations for the frames in the first sample are received. Additional frames from the first group can be manually annotated and added to the frames of the first sample, thus providing a larger dataset for retraining the model. By deferring further processing until the neural network has been retrained, significant amounts of time and energy are saved.

[0020] Preferably, the selection of frames for the initial sample depends on the data points for which the quality measure is to be determined, in particular a random selection of single frames for object detection and / or a random selection of batches of consecutive frames for object tracking. Applying an intelligent sampling strategy maximizes the improvement achievable through retraining. An object detector, such as one used for traffic sign recognition, benefits from training data with high variance, so a random selection of single frames is a useful initial sample. On the other hand, a tracking component benefits from continuous data, since only then can the same object be tracked between consecutive frames. In this case, a sample would expediently be a random selection of consecutive frames—for example, always 10—for a variety of objects.For example, an intelligent sampling process would take frames 10 to 20, as well as frames 100 to 110 and 235 to 245, for the initial sample when determining a quality measure for a tracking component. To ensure high variance in the sample, the software component performing the sampling can prescribe a minimum time interval between samples to guarantee that different frames were captured under varying environmental conditions. Additionally or alternatively, one or more attributes can be considered during sampling. For instance, if a sample is being selected to quantify the object detector's nighttime capabilities, different environments, such as a large city, rural areas, or highways, can be specified. Random selection would then be performed among all samples that meet the prescribed criterion.

[0021] In one embodiment, the steps of selecting a current sample from one or more frames from the first package, determining a quality measure for the data points, receiving corrected annotations for the frames in the current sample, and retraining the neural network based on the frames in the current sample are repeated until the quality measure for the frames in the current sample exceeds a predefined threshold or the first package contains no remaining frames. Advantageously, the neural network is retrained until it can also correctly handle adverse environmental conditions affecting the annotation process.

[0022] Preferably, sensor data annotation and recording are performed alternately or simultaneously. If it is determined that the quality measure for at least one frame in the first sample is below a predefined threshold, the computer requests the recording of additional sensor data for which at least one condition attribute lies within the selected value range of the first sample. A value range for the condition attribute can be selected by equipping a test vehicle with an automated recording device that executes a selection program triggering a recording as soon as a predefined recording condition is met, or by instructing a test driver to drive under specific conditions, e.g., at night. Thus, new data is recorded, at least primarily, for those environmental conditions for which the neural network requires further training.By carefully selecting training data, improvement per training effort is maximized. This reduces the computing power and energy consumption required for training.

[0023] A second aspect of the invention provides a computer-implemented method for automatically annotating sensor data comprising frames, such as video or audio frames. At least one processor of a host computer performs the method, which comprises: a) Receiving a large number of sensor data frames, b) Grouping the frames into packets based on at least one condition attribute, wherein the condition attribute describes an environmental condition that existed during the recording of the sensor data frame, c) Annotating the frames from a first packet using a neural network, wherein the annotation includes assigning at least one data point to each frame, the first packet comprising frames for which the at least one condition attribute lies within a selected range of values, d) Selecting an initial sample of one or more frames from the first packet and determining a quality measure for the data points, e) Determining that the quality measure for at least one frame in the first sample is below a predefined threshold.f) Receiving corrected annotations for the frames in the first sample and retraining the neural network with the frames in the first sample, g) Annotating at least one of the remaining frames of the first package with the retrained neural network, h) Selecting a second sample of one or more frames from the at least one remaining annotated frame of the first package and determining a quality measure for the data points, i) Determining that the quality measure for the frames in the second sample is above a predefined threshold, j) Annotating the remaining frames from the first package with the retrained neural network, and k) Exporting the annotated frames.

[0024] One aspect of the invention also relates to a non-volatile, computer-readable medium containing instructions which, when executed by a microprocessor of a computer system, cause the computer system to execute the method according to the invention as described above or in the attached claims.

[0025] In a further aspect of the invention, a computer system is provided comprising a host computer, which includes a processor, main memory, a display, a device for human input, and non-volatile memory, in particular a hard disk or a solid-state drive. The non-volatile memory contains instructions which, when executed by the processor, cause the computer system to execute the method according to the invention.

[0026] The processor can be a general-purpose microprocessor, typically used as the central processing unit of a personal computer, or it can comprise one or a plurality of processing elements designed to perform specific calculations, such as a graphics processing unit. In alternative embodiments of the invention, the processor can be replaced or supplemented by a programmable logic device, such as an FPGA configured to provide a fixed set of functions, and / or it can comprise an IP core microprocessor. Brief description of the drawings

[0027] A better understanding of the present invention can be obtained by considering the following detailed description of the preferred embodiment in conjunction with the following drawings, in which: Fig. 1 an exemplary sketch of a computer system; Fig. 2An example of a video frame with a schematic diagram of possible data points in the inset at the top left; Fig. 3 a schematic diagram of an exemplary package of video frames; Fig. 4 a schematic diagram of an exemplary package of video frames grouped according to a time of day and weather information; Fig. 5 a schematic diagram that illustrates the correlation between environmental conditions and the quality of the annotations; Fig. 6 a schematic diagram of an automation system that performs a method according to the invention.

[0028] Similar elements in the figures are designated with the same letters. While the invention is open to various modifications and alternative forms, certain embodiments are shown in the drawing as examples and are described in detail here. It is understood, however, that the drawing and the detailed description are not intended to limit the invention to the specific form disclosed. On the contrary, the invention is intended to cover all the following modifications, equivalents, and alternatives within the scope and scope of the present invention, as defined by the attached claims. Detailed description

[0029] Fig. 1 represents an exemplary embodiment of a computer system.

[0030] The embodiment shown comprises a host computer PC with a display ANZ and user interface devices, such as a keyboard TAS and a mouse MAU; furthermore, an external server may be connected via a network, as indicated by a cloud symbol.

[0031] The host computer (PC) comprises at least one processor (CPU) with one or more cores, RAM, and a number of devices connected to a local bus (such as PCI Express) that exchanges data with the CPU via a bus controller (BSC). These devices include, for example, a graphics processing unit (GPU) for driving the display, a USB port for connecting peripherals, non-volatile storage (HDD), such as a hard disk drive (HDD), and a network interface (NC). Furthermore, the host computer may include a dedicated accelerator for neural networks. The accelerator can be implemented as a programmable logic device (PLD), such as an FPGA, as a graphics processor suitable for general-purpose computing, or as an application-specific integrated circuit.Preferably, the non-volatile memory contains instructions which, when executed by one or more cores of the processor CPU, cause the computer system to execute a method according to the invention.

[0032] In alternative embodiments, indicated in the figure as a cloud, the host computer can comprise one or more servers, each comprising one or more processing elements, with the servers connected via a network to a client comprising a display device and an input device. Thus, the annotation environment can be run partially or entirely on a remote server, such as in a cloud computing environment. A personal computer can be used as a client comprising a display device and an input device connected via a network. Alternatively, a graphical user interface of the annotation environment can be displayed on a portable computing system, such as, in particular, a smartphone or a tablet with a touchscreen interface.

[0033] Fig. 2presents an example video frame with a schematic diagram of possible data points in the inset at the top left.

[0034] The figure represents a photograph or a frame of a cityscape. Such a frame could be part of a video recording. Generally, a recording provided by a client might consist of video or audio data representing a sequential context, such as a 5-minute drive recorded by a camera and LiDAR sensor, or a 10-minute voice recording. Video recordings, for example, could consist of a series of consecutive frames, each capturing a series of objects. The neural network processes the recording to create annotations that can include a variety of data points, with each data point describing a specific aspect.

[0035] A data point is a parameter that describes a specific property of a recording and can be applied to all levels of detail. Levels of detail can be the entire recording, a series of consecutive or random frames, a single frame, or an object within a frame. A specific example would be an annotation for a car, consisting of a bounding box that describes the car's position within a certain accuracy, a vertical line marking the car's edge, a classification to describe the car's type, attributes for clipping or obscuring, turn signals, brake lights, color, and so on.Data points can be classes, frames, segments, polygons, lines, attributes such as turn signals, brake lights, colors, subclasses, tracking information, occlusion level, clipping level, complex classes that describe the relevance of an object / frame / clip, sound, text, feeling, or any other automatically determined information.

[0036] The inset in the upper left of the figure shows various data points for a car. Cars can be of different types, such as a van, an SUV, or a sports car. The position, or rather the dimensions, of a car are generally specified by a bounding box, that is, a rectangular frame or cuboid that encloses the car. Vertical lines indicate the car's boundaries. Another possible data point for a car is the activation of a light indicator, such as the turn signal shown in the inset.

[0037] A large number of cars are present in the frame, each enclosed by a bounding box. Cars may be fully visible, such as the one driving directly in front of the camera, or they may be obscured. The traffic density of the urban scene can affect the annotation quality, for example, by making it difficult to accurately determine the boundaries of the bounding box due to obscuration.

[0038] Fig. 3 represents a schematic diagram of an exemplary package of video frames.

[0039] A common way to generate sensor data for training or validating an autonomous vehicle is to have a test driver drive around while all relevant sensor data is recorded, such as camera, lidar, and / or GPS data. The data is unsorted, so the first recording (Recording 1) might be taken in full daylight on a highway, while the next recording (Recording 2) might also be taken during the day, but during a rain shower. The next recording (Recording 3) might be taken at night. Subsequent recordings may involve unpredictable changes in environmental conditions.

[0040] Fig. 4This is a schematic diagram of an example package of video frames grouped according to additional time-of-day and weather information. Because the visibility of an object depends heavily on the time of day and weather conditions, the annotation quality for an object detector correlates with these environmental conditions.

[0041] It is advantageous to group the recorded frames into packets or clusters according to the time of day and weather conditions. In the example shown, recordings 1, 5, and 6 were recorded on a dry day, while recordings 2, 4, and 7 were recorded during a wet day due to rainy weather. Recording 4 was recorded under wet conditions during the night.

[0042] Further criteria could be used to group recorded frames into packets or clusters. For example, in the context of autonomous driving, customer-provided data can be grouped not only based on day / night and wet / dry conditions, but also on road type, e.g., inner-city street versus highway.

[0043] Frames recorded under similar environmental conditions are processed together to provide groups of frames with consistent annotation quality. In one embodiment, different neural networks can be used to annotate the frames based on at least one environmental condition that existed when the respective frame was recorded.

[0044] Fig. 5 This is an example diagram illustrating the correlation between environmental conditions and the quality of annotations.

[0045] An initial group of frames, Cluster 1, comprising recordings 2, 4, and 7, was recorded on a rainy or wet day. Based on a manual quality check, the accuracy of Cluster 1 is close to 90%. Therefore, the annotations still need to be verified, but the neural network can generate sufficiently accurate data after a few retraining iterations.

[0046] A second group of frames, Cluster 2, comprising recordings 1 and 5, was recorded on a dry day. Based on a manual quality check, the accuracy of Cluster 2 is 99%. Since this is sufficiently accurate, quality checks for groups of frames recorded under the same environmental conditions can be omitted entirely.

[0047] A third group of frames, Cluster 3, comprising recordings 3 and 8, was recorded on a dry night. Based on a manual quality check, the accuracy of Cluster 3 is 50%, which is clearly unacceptable. Frames recorded under the same environmental conditions require extensive manual review and improved training of the neural network.

[0048] Because the frames were grouped according to environmental conditions, human effort is allocated to those groups of frames where it is most needed. Frames captured under favorable conditions can be processed fully automatically. Similarly, the computing power or energy required for retraining a neural network is deployed where this has a noticeable impact on the quality of the annotations.

[0049] Fig. 6Figure 1 is a schematic diagram of an automation system that performs a method according to the invention. The automation system executes various steps of the method in dedicated components and is well adapted for execution in a cloud computing environment.

[0050] In the first step, "data acquisition," unsorted recordings are received from a customer. These recordings can then be normalized, for example, by splitting them into frames to enable smooth processing.

[0051] In a second step, "enrichment," the frames from the recordings are analyzed and automatically enriched with metadata relevant for measuring the quality of automation. This step is shown as a prerequisite for automation, but in alternative implementations, and depending on the desired metadata, enrichment could also be performed after automation based on information collected during annotation, such as traffic density or the object's distance from the sensor. In the context of autonomous driving, metadata or condition attributes relevant to annotation quality could include geography, weather conditions, road type, lighting conditions, and / or time of day. For the efficiency of automation, it is beneficial to process a group of frames in their entirety in the following steps.For projects with nested frame recording and processing, it can be advantageous to add frames recorded under the same environmental conditions until a predefined cluster size is reached before proceeding with further processing steps. Thus, enrichment and clustering encompass techniques for adding static or dynamic metadata to recordings, as well as techniques for inserting individual recordings into larger clusters of a definable size based on metadata enrichment.

[0052] In a third step, the "Scheduler," different groups of frames are assigned to annotation by an automation engine, which operates one or more automation components to annotate the frames with one or more data points. The scheduler selects the group of frames for processing based on the availability of new versions of automation components. An automation component can generate a single data point, such as a vertical line, or a number of related data points, such as a bounding box and an object class. Automation components can be neural networks or any other type of machine learning-based technique that learns from data samples in a supervised, semi-supervised, or unsupervised manner.

[0053] In a fourth step, the "automation engine," a group of frames is processed by at least one automation component, which assigns data points to the frames. The automation system generates any type of data point via automation components; these components are a central part of the annotation system's workflow. Preferably, data points carry metadata that precisely describes the version of the automation component used to generate the result. The automation engine includes techniques for accurately storing the relevant metadata about the automation components.

[0054] In a fifth step, "sample verification," a sample of frames is selected for quality control. During quality control, a human annotator can be shown the frame with relevant annotations, such as a bounding box, and asked whether the bounding box is correct. Alternatively, they can be shown a user interface to adjust the bounding box and / or add one if an object was missed by the neural network. The automation system determines a quality measure based on the type and number of corrections made by the human annotator.

[0055] In a sixth step, "Sample verification passed?", the system determines whether the annotation quality or quality measure is above a predefined threshold. If the automation system determines that it is (yes), the group of frames comprising the selected sample is exported and delivered to the customer. If at least one group of frames recorded under a specific set of environmental conditions passes the sample verification, the automation system may determine that the annotation quality for all groups of frames with the same environmental conditions can be exported without further quality checks, thus skipping steps five and six.In one embodiment, the automation system can count the number of groups with those environmental conditions that exhibited sufficient annotation quality and can skip sample verification once a predefined number of groups have passed the sample verification. If the automation system determines that the group of frames has not passed the sample verification (no), execution continues at an eighth step in which the automation system determines whether frames recorded under the environmental conditions of the selected sample are necessary for the dataset. Whether this is the case can depend on the number of frames recorded under the same conditions that were already used to train the model.If a sufficient number of frames have already been used for training, the group of frames can simply be inserted into the third step, "Scheduler", to be processed again once a retrained neural network is available.

[0056] In a seventh step, "Customer Sample Verification," the customer can review a sample of the exported frames to ensure that the annotations meet their specifications and the required annotation quality. If the customer rejects the group of frames, a sample or the entire group of frames is processed manually in the "Correction" step. Preferably, the automation system enforces sample verification for all subsequent groups under the same environmental conditions until a new group of frames passes the sample verification of the sixth step and / or the customer sample verification of the seventh step.

[0057] In a ninth step, "Correction," a manual annotation is performed on the sample of frames that failed the test, or on a sample or the entire group of frames that were rejected by the customer. The manually annotated frames are exported and delivered to the customer for the seventh step, Customer Sample Review. The manually annotated frames are also used to retrain the neural network by feeding the corrected data into the training, validation, or test datasets. These datasets are symbolically represented by a cylinder.

[0058] In a tenth step, "Flywheel," the at least one neural network or automation component that generated the data points rejected during sample verification is retrained. Retraining the neural network improves the automation quality. Preferably, the automation components are improved to a level where no manual inspection is required for as many metadata clusters (i.e., frames recorded under a specific set of environmental conditions) as possible. Iteration times for retraining must be as short as possible to enable rapid efficiency improvements.

[0059] Flywheel includes techniques for efficiently storing training datasets for each automation component (each data point) to monitor training dataset changes and automatically trigger retraining as soon as predefined or automatically determined thresholds of training dataset changes are detected. Furthermore, Flywheel includes techniques for automatically deploying retrained models into automation components and notifying the planner of version changes to automation components.

[0060] When new data is captured simultaneously or interleaved with frame annotation, an additional step of targeted data capture can be performed. Automation components are improved through numerous training iterations on a constantly refined dataset, which increasingly reflects real-world variance over time. Confidence levels per metadata cluster enable a systematic approach to capturing precisely those data samples where automation results are most compromised. With reference to Figure 5 The frames from Cluster 3 were recorded at night, and automatic annotation currently results in unacceptable annotation quality. Once this is discovered during the verification step, a targeted data acquisition can be requested, in which nighttime samples are specifically recorded to improve the training dataset of the automation component under these environmental conditions.

[0061] In a preferred embodiment, the level and amount of additional training data of a specific type (cluster) are determined based on confidence. All data recorded under the same conditions can be used for retraining. Once corrections are made to incorrectly annotated frames, they are fed directly into the training set of the dedicated automation component. However, it is usually not necessary to manually correct all data for a given cluster and data point. Instead, only samples up to the next retraining threshold level are taken and corrected. The remaining data is automatically scheduled for re-processing with a higher version of the automation component. Targeted data acquisition includes techniques for selecting samples of interest based on metadata clusters up to predefined amounts for manual correction.Furthermore, it preferably includes techniques for marking poor quality samples that are not needed for retraining for automation runs on higher versions of the respective automation component.

[0062] By using the correlation between the environmental conditions under which a frame was recorded and the resulting quality of the annotations, the method according to the invention makes it possible to use manual work, especially in the rapid improvement of neural networks, which are then used to create automatic annotations for delivery to the customer, thus considerably accelerating larger annotation projects that are necessary, for example, for validation.

[0063] Those skilled in the art will be aware that the sequence of at least some of the steps of the method according to the invention can be changed without deviating from the scope of the claimed invention. While the present invention is described with respect to a limited number of embodiments, those skilled in the art will be aware of numerous modifications and variations thereof.

Claims

1. A computer-implemented method for automatically annotating sensor data frames, the method comprising: Receiving a plurality of sensor data frames, Grouping the frames into a plurality of packets based on at least one condition attribute, wherein the condition attribute describes an environmental condition that existed during recording of the sensor data frame, Annotating the frames from a first packet using a neural network, wherein the annotating comprises assigning at least one data point to each frame, wherein the first packet comprises frames for which the at least one condition attribute is in a selected range of values, Selecting a first sample of one or more frames from the first packet and determining a quality measure for the data points, wherein the quality measure is determined from the type and number of corrections made by a human annotator, wherein if the computer determines that the quality measure for at least one frame in the first sample is below a predefined threshold, the method further comprises: receiving corrected annotations for the frames in the first sample, retraining the neural network based on the frames in the first sample, Selecting a second sample of one or more frames from the frames in the first packet that were not included in the first sample, Annotating the frames of the second sample with the re-trained neural network and determining a quality measure for the data points, Determining that the quality measure for the frames in the second sample is above a predefined threshold, Annotating the remaining frames of the first packet with the post-trained neural network, and Exporting the annotated frames of the first packet.

2. A method according to any preceding claim, wherein for frames comprising image data, the condition attribute is a geographical location, a time of day, a weather condition, a visibility condition, a road type, a distance to an object and / or a traffic density and / or wherein for audio frames, the condition attribute is a geographical location, a gender and / or an age of a speaker, a room size and / or a measure of background noise.

3. A method according to any preceding claim, wherein the step of receiving a plurality of sensor data frames comprises a step of pre-processing the frames, wherein at least one of the condition attributes for a frame is determined by a dedicated neural network based on the frame, and / or at least one of the condition attributes for a frame is determined based on additional sensor data recorded simultaneously with the frames.

4. A method according to any preceding claim, wherein for an image frame the at least one data point comprises a position of an object, a class of an object, a position of an edge of a bounding box, a correlation of an object in the image frame with an object in a preceding or subsequent image frame, and / or an activation of a light indicator, and / or for audio frames the at least one data point comprises one or more text words recognized from the audio frame, wherein a data point may be derived from a plurality of audio frames.

5. A method according to any preceding claim, wherein the first sample comprises two or more frames selected from the first packet, and wherein once the computer determines that the quality measure for the first sample is below the predetermined threshold, no further calculations are performed on the frames from the first packet until corrected annotations are received for the frames in the first sample.

6. A method according to any preceding claim, wherein the selection of frames for the first sample depends on the data points for which the quality measure is to be determined, in particular a random selection of individual frames for object detection and / or a random selection of batches of consecutive frames for object tracking.

7. A method according to any preceding claim, wherein the steps of selecting a current sample of one or more frames from the first packet and determining a quality measure for the data points and receiving corrected annotations for the frames in the current sample and retraining the neural network based on the frames in the current sample are repeated until the quality measure for the frames in the current sample is above a predefined threshold or the first packet comprises no remaining frames.

8. A method according to any preceding claim, wherein the annotating of sensor data and the recording of sensor data are performed alternately or simultaneously, and wherein, when it is determined that the quality measure for at least one frame in the first sample is below a predefined threshold, the computer requests the recording of additional sensor data for which the at least one condition attribute is within the selected range of values of the first packet.

9. A method for automatically annotating sensor data comprising frames, such as video or audio frames, the method being performed by at least one processor of a host computer, the method comprising: (a) Receiving a plurality of frames of sensor data, b) Grouping the frames into packets based on at least one condition attribute, wherein the condition attribute describes an environmental condition that existed during the recording of the sensor data frame, c) Annotating the frames from a first packet using a neural network, wherein the annotating comprises assigning at least one data point to each frame, wherein the first packet comprises frames for which the at least one condition attribute is in a selected range of values, d) Selecting a first sample of one or more frames from the first packet and determining a quality measure for the data points, e) Determining that the quality measure for at least one frame in the first sample is below a predefined threshold, wherein the quality measure is determined from the type and number of corrections made by a human annotator, f) Receiving corrected annotations for the frames in the first sample and retraining the neural network with the frames in the first sample, g) Annotating at least one of the remaining frames of the first packet with the retrained neural network, h) Selecting a second sample of one or more frames from the at least one annotated remaining frames of the first packet and determining a quality measure for the data points, i) Determining that the quality measure for the frames in the second sample is above a predefined threshold, j) Annotating the remaining frames from the first packet with the post-trained neural network, and k) Exporting the annotated frames.

10. A non-transitory computer-readable medium comprising instructions which, when executed by a microprocessor of a computer system, cause the computer system to perform a method according to any of the preceding claims.

11. A computer system comprising a host computer, the host computer comprising a microprocessor, a random access memory, a display, a human input device and a non-volatile memory, in particular a hard disk or a solid state drive, the non-volatile memory comprising instructions which, when executed by the microprocessor, cause the computer system to perform a method according to any of the preceding claims.