Method and system for automatically annotating sensor data - Patents.com

Abstract

The present invention relates to a computer-implemented method for automatically annotating sensor data frames, e.g. image or acoustic frames. Received sensor data frames are annotated, with each sensor data frame being associated with at least one data point and with each data point being associated with at least one state attribute. The data points are grouped based on the at least one state attribute, where the group includes a defined value range of the state attribute. A random sample of data points is selected from a first group, and a quality measure for the random sample is determined. If the quality measure for the first random sample is below a predefined threshold, a neural network is post-trained based on the corrected annotation for the first random sample. As soon as the quality measure exceeds the predefined threshold, the annotated sensor data frame is exported.

Description

[Technical field]

[0001] The present invention relates to a method and computer system for automatically annotating sensor data frames, in particular imaging sensor data frames. [Background technology]

[0002] Autonomous driving does not promise a sufficient level of comfort and safety in everyday traffic. Despite significant investments by various companies, existing approaches are only applicable to a limited extent and / or only present a subset of real autonomous behavior. This is due to the lack of a sufficient amount and variety of available driving scenarios. Further progress is hindered here by the need for a huge amount of sufficiently different training and validation data (i.e. independent ground truth data). The preparation of training data generally requires the recording of various driving scenarios by a vehicle equipped with a sensor set, in particular imaging sensors, such as one or more cameras, LiDAR sensors and / or radar sensors. Before these recorded scenarios can be used as training data, they must be annotated.

[0003] This is often performed by annotation service providers, also called labelers, who receive the recorded sensor data and divide it into work packets for multiple human workers. The exact annotations required (e.g. object classes to be distinguished) depend on each project and are specified as detailed labeling specifications. Customers provide the annotation service provider with raw data and expect high-value annotations according to their information in a short time frame. The number of labelers required to complete an annotation project increases as the amount of data provided increases and as the time frame for a fixed amount of data decreases. For these reasons, large-scale annotation projects that can provide sufficient ground truth data for validating, for example, autonomous vehicles, cannot be achieved by human work alone, and automation of the annotation process is required.

[0004] The automated approach utilizes a neural network for labeling the recorded sensor data. An initial set of received data is manually labeled and then used to train the neural network. Once the neural network is sufficiently trained, it can annotate large amounts of recorded imaging sensor data. This is a significant reduction in effort compared to a purely manual approach. However, time-consuming human quality checks are still required to maintain a high annotation quality. Since a quality control process must always be applied for every annotation, there is a linear relationship between the project volume and the cost of the work required to fulfill the project requirements.

[0005] Therefore, there is a need for improved methods for automatically annotating sensor data, especially imaging sensor data, and it is particularly desirable to ensure high annotation quality while reducing the number of manual quality checks. Summary of the Invention [Problem to be solved by the invention]

[0006] It is an object of the present invention to provide a method and a computer system for automatically annotating sensor data frames, in particular video frames or LiDAR point clouds. [Means for solving the problem]

[0007] In a first aspect of the invention, a computer-implemented method for automatically annotating sensor data is provided, the method comprising the steps of receiving a plurality of sensor data frames, annotating the plurality of sensor data frames using at least one neural network, where the annotating comprises associating at least one data point with each sensor data frame and associating at least one state attribute with each data point, grouping the data points based on the at least one state attribute, where a first group comprises data points for which the at least one state attribute is within a defined value range, and selecting a first random sample of one or more data points from the first group and determining a quality measure for the data points in the first random sample. The method further includes, if the computer detects that the quality measure of the first random sample is below a predetermined threshold, receiving corrected annotations for the data points in the first random sample, post-training a neural network based on the data points in the first random sample, selecting a second random sample from one or more data points of the first group that were not in the first random sample, annotating the sensor data frames of the second random sample with the post-trained neural network and determining a quality measure for the data points in the second random sample. The method further includes, upon the computer detecting that the quality measure of the first random sample and the quality measure of the second random sample are above the predetermined threshold, annotating the remaining sensor data frames of the first group with the neural network and exporting the annotated sensor data frames of the first group.

[0008] The computer system that executes the method according to the present invention can be realized as a single host computer that includes a processor, for example a general-purpose microprocessor, a monitor and an input device.Alternatively, the computer system can include one or more servers with multiple processing elements, for example processor cores or dedicated accelerators, where the server is connected to a client that includes a monitor and an input device via a network.In this way, the annotation software or automation software that includes the component for automatic annotation can be partially or completely executed on a remote server, for example in a cloud computing environment, where only one graphic user interface needs to be implemented locally.

[0009] The data points may describe objects or features in the sensor data frame, in particular in an image or a LiDAR point cloud, or may indicate properties of the sensor data frame. The examination of the data points is preferably performed in a sensor data frame that includes or has properties of the object or feature associated with the data point. The sensor data frame may include a number of data points, and the examination of a first data point in the sensor data frame may be performed independently of the examination of a second data point in the sensor data frame. For example, an object in a camera image may be annotated with a data point in the form of a bounding box and a class of the object, and depending on the class of the object, in particular as a passenger vehicle, the object may also be further annotated with another attribute, such as a turn signal status, as a data point. The number of data points to be associated may depend on the content of the sensor data frame, which may result in empty sensor data frames to which no data points are associated. The empty sensor data frames here are to be ignored in further processing. The ignoring of such empty sensor data frames should be included in the association of at least one data point to each sensor data frame.

[0010] The state attributes may describe the ambient conditions, or more generally the circumstances that prevailed when the object or feature to which the data point is associated was recorded. The state attributes may be static state attributes that in particular describe the environmental conditions that existed at the time of recording. The ambient conditions present during the recording of a frame may have different effects on the accuracy of the annotation depending on the type of data point. In general, for annotations that include multiple data points, the effect of the state attributes may differ depending on the data point or the type of data point. If the sensor data includes, for example, camera images recorded at night, it may be more difficult to determine the location and / or class of the object. However, attributes of a car, such as the status of the turn signal, may be more easily identified at night than in full daylight. The state attributes may be dynamic state attributes calculated in the framework of annotation using neural networks or may be independent data points. One or more of the state attributes of a type of data point may be state attributes of other types of data points. Even under the same environmental conditions, for example objects that are far away may be more difficult to identify, which not only makes classification difficult but also limits the accuracy of the bounding box. Note that the size of the first object has no effect on the annotation quality of the second object (but possible occlusion of the second object has a complete effect).

[0011] The grouping of data points based on at least one state attribute allows for taking into account possible correlations between state attributes and annotation accuracy. The grouping of data points can be performed for different kinds or types of data points without mutual dependency. Since state attributes may act differently on one type of data point from another type of data point, individual data points of a certain type are preferably grouped together, while data points of different types are preferably grouped according to different criteria. The present invention allows for identifying static and dynamic state attributes that negatively affect annotation quality and for improving the neural network under these conditions by selective post-training. Furthermore, it allows for reducing manual correction and quality inspection costs.

[0012] The concept "neural network" may relate to an individual neural network, a combination of different neural networks according to a set architecture, or any kind of machine learning based technique that learns from training data in a supervised, semi-supervised or unsupervised manner. Different neural networks can be used for different data points, i.e. the location of an object and / or its classification can be determined by a first neural network, while the attributes of the object can be determined by at least one other neural network.

[0013] The invention is based on the consideration that in many cases the quality of different components of an annotation is totally different. For example, an annotation may consist of a two-dimensional bounding box, an object class and other attributes, such as the blinker status. Here, a situation may arise where the quality of the object class is normal, but the positioning of the bounding box must be corrected. For this reason, the individual data points are the smallest units of the annotation, and their quality can be calculated independently of the other components of the annotation. In this case, it is advantageous to distinguish between different data points, i.e. a bounding box basically describes another characteristic of the object, such as an attribute like the blinker status. This allows the state attributes to act differently on the quality of different types of data points, i.e. to be ineffective for a number of state attributes for one type of data point, but decisive for another type of data point. The decomposition of a complex annotation into individual data points allows the influence of state attributes on the quality of the annotation to be determined in a fine-grained manner and to be taken into account during correction.

[0014] The steps of selecting the second random sample from one or more data points of the first group that were not in the first random sample and annotating the sensor data frames of the second random sample with the post-trained neural network are interchangeable, so that, for example, an entire batch of sensor data frames can be annotated with the post-trained network before the second random sample is selected.

[0015] In particular, if a neural network has to be post-trained multiple times, the computational burden is reduced because only the data points of each second or another random sample need to be annotated by the post-trained network. For a large portion of the sensor data frame, annotation can be postponed until it has been verified during sample inspection that the post-trained network provides annotations of sufficiently good quality.

[0016] Exporting the annotated frames may for example include storing the frames on an external data carrier and / or converting or summarizing the frames into a set data format. Depending on the fine granularity of the data points, in principle also a handover of partially annotated sensor data frames is possible. For more perspective, it may be advantageous to hand over a sensor data frame to a customer only if a sufficient post-trained neural network is available for all types of data points present, i.e. if the sensor data frame can be fully annotated.

[0017] Since manual work is used at least in large part only to create training data, test data and / or validation data for the neural network or another machine learning-based automation component to improve the system for annotation of sensor data frames, the cost for a large annotation project can be significantly reduced. Typically, after several iterations of post-training the neural network, the quality level is sufficient to provide an automated result, i.e., annotation by the neural network, without the need for further manual inspection. Moreover, annotation can be preferably continued with a small random sample size that is independent of the data volume. The method of the present invention further reduces the required manual effort and time by focusing the post-training on conditions where annotation quality is still lacking.

[0018] As a quality measure, for example, the overlap area between an automatically created bounding box and a bounding box created manually or adapted in the framework of a quality control can be used. It is also possible to require a maximum number and / or a maximum percentage of mismatched and / or false positive and / or false negative object classes. The quality measure is deemed to be below a predefined threshold, for example, if the overlap between the bounding boxes is too low. It can also be set as a quality measure that in a random sample from a set number of frames, there may be a maximum of a set number of false positive or incorrectly identified objects and / or a maximum of a set number of false negative or not incorrectly identified objects. Here, the quality measure is deemed to be below a predefined threshold if, for example, a maximum permissible number of unidentified objects in a random sample is exceeded. When determining the quality measure, combined conditions can also be used, for example by weighted summarization of the individual values.

[0019] If the computer detects that the quality measure of the second random sample is below a predetermined threshold, the method preferably includes the following further steps: receiving corrected annotations for the data points in the currently examined random sample, post-training the neural network on the data points in the currently examined random sample, selecting another random sample from one or more data points of the first group that were not part of the previous random sample, annotating the sensor data frames of the other random sample by the neural network, and determining a quality measure for the data points of the other random sample. These further steps are preferably repeated until the computer detects that the quality measure of the frames in the other random sample has exceeded the predetermined threshold or that no sensor data frames containing uncorrected data points remain for the random sample. The method further includes annotating the remaining sensor data frames of the first group with the (in this case fully post-trained) neural network as soon as the quality measure of the sensor data frames of the random sample exceeds a predetermined threshold, and exporting the annotated sensor data frames of the first group. The approach allows for rapid improvement of the neural network through a limited number of iterations.

[0020] Preferably, the step of receiving a plurality of sensor data frames includes a step of pre-processing the sensor data frames, where at least one of the state attributes is determined by a dedicated neural network based on the frames and / or at least one of the state attributes is determined based on additional sensor data recorded simultaneously with the sensor data frames. A dedicated neural network means in particular a neural network specially trained for each problem statement. The additional sensor data can be combined and / or used for queries in various services indicating, for example, a lighting condition type based on weather conditions, time of day and geographical region.

[0021] Thus, in one embodiment, the sensor data frame is an image data frame, i.e. it contains data from imaging sensors, such as one or more cameras, LiDAR sensors and / or radar sensors. These received sensor data may also contain additional sensor data recorded simultaneously with the image data frame, such as GPS position, vehicle acceleration or data from a rain sensor. For an image data frame, the state attributes preferably include the geographical region, the time of day, weather conditions, visibility conditions, road type, distance to an object and / or traffic density, size of the bounding box, extent of occlusion and / or clipping, ego-vehicle speed, camera parameters, color area and / or contrast measure of the area contained in the bounding box, direction of travel of the ego-vehicle or astronomical information such as the position of the sun relative to the direction of travel of the ego-vehicle. The distance to an object may be the distance to the nearest object, the distance to the farthest object or the average distance to a number of objects identified in the frame, and the impact on the object detection and / or classification performance of the neural network can be quantified by considering the distance to an object as an ambient condition during recording. For an image data frame, the at least one data point preferably includes object location, object class, bounding box coordinates, line coordinates, object clipping, object occlusion, correlation of an object in a previous image data frame with an object in a subsequent image data frame (as a result of object tracking), and / or activation state of a light indicator such as a turn signal or brake light. The number of data points may depend on the content of the image data frame, for example many cars and pedestrians in a large city scene with a corresponding number of object locations, object classifications and possible attributes for the corresponding object classes. For pedestrians, for example clothing, posture and / or gaze direction may be additional attributes or additional data points.

[0022] In one embodiment, the received sensor data frame is an acoustic frame, i.e. it contains data of an acoustic sensor such as a microphone. For an acoustic frame, the state attribute is preferably a geographical region, a gender and / or an age of the speaker, a spatial size and / or a measure of background noise. For an acoustic frame, at least one data point includes a phonon and / or one or more words of text identified from the acoustic frame. Each word is identifiable from multiple subsequent acoustic frames, and thus the data point can be derived from multiple acoustic frames. Difficulties in speech recognition may depend, for example, on the frequency domain in which the speaker is making noise, the presence of spatial reverberation or echo, and / or the level of background noise present.

[0023] Preferably, grouping the data points includes determining clusters in a multi-dimensional space, in particular using a nearest neighbor algorithm and / or an unsupervised learning approach and / or a machine learning classification model. Advantageously, a type of data point is accurately assigned by the machine learning classification model to one of at least two clusters, each having a different expected quality level. The assignment to a cluster can be performed by classification or grouping in a multi-dimensional space expanded by a plurality of state attributes. Thus, based on a combination of static and dynamic state attributes, each individual data point can be assigned to a different cluster. However, all or a set amount of state attributes can also be used as the context of the data point to determine which state attributes have a significant impact on the quality of the data point of this type.

[0024] In one embodiment, annotating the sensor data frames comprises associating at least one data point of a first type and at least one data point of a second type to each sensor data frame. Particularly preferably, the data points of the first type are grouped based on determining clusters in a first multidimensional space and the data points of the second type are grouped based on determining clusters in a second multidimensional space, the multidimensional space for a data point being expanded by a number of state attributes. In this case, this may be the state attributes associated with the type of data point here, but it may also be configured to use all or a set amount of state attributes as a context for the data points and further to calculate, based on the cluster determination, which state attributes significantly affect the quality of the data points of that type.

[0025] Preferably, the first group is defined based on a first cluster in which at least one state attribute is within a defined first value range, and the second group is defined based on a defined second value range, the first value range and the second value range being disjunctive with respect to the at least one state attribute and / or with respect to all state attributes associated with each data point. Basically, a division into a larger number of clusters is also possible.

[0026] Preferably, the error probability and thus the quality level of the cluster is detected based on the data points of the cluster by random sampling. For example, the first data point can be associated with the first cluster, and the second data point can be associated with the second cluster. In this example, one random sample will determine the quality level (or the corresponding inverse error probability) of the first cluster of 100% and the second cluster of 0%. Statistical methods can dynamically adapt the random sample size during the measurement and introduce predictions of the previous measurement quality of the same cluster. The aim here is to improve the quality level of various clusters above a desired threshold with minimal manual inspection and correction costs by iterative improvement of the automated labeling. For clusters with higher error probability or lower quality, preferably more random samples are taken and more data points are corrected for post-training.

[0027] Particularly preferably, an error probability is determined for each data point based on whether the data point is in the first group or the second group. In this way, a quality level prediction is performed. Based on a combination of static and dynamic state attributes, individual data points can be classified into different quality levels. For data points in a group with a higher error probability or a group with a lower quality, preferably more random samples are taken.

[0028] If the annotation of the sensor data frame with the first type of data points is based on a first neural network and the annotation of the sensor data frame with the second type of data points is based on a second neural network, then preferably the further method steps for the first type of data points and the further method steps for the second type of data points are performed independently of each other. The quality level calculation or statistical quality analysis may result in different error distributions for the various data types. The independent processing allows error correction and post-training to be performed as desired for each type of data point and each limited to the specifically required extent.

[0029] Preferably, the selection of frames for the first random sample, in particular the random selection of individual frames for object detection and / or the random selection of a batch of several consecutive frames for object tracking, depends on the data points for which the quality measure is to be determined. By using an intelligent strategy for random sample taking, the improvement achievable by post-training is maximized. Since object detectors for traffic sign identification, for example, have high variance training data, the random selection of individual frames is a useful first random sample. On the other hand, the tracking component of the consecutive data is obtained, since tracking of the same object can only be performed in this case between consecutive frames. In this case, preferably a series of consecutive frames is randomly selected as the random sample due to the diversity of the objects, for example always 10 frames. By way of example, if a quality measure for the tracking component is to be determined, frames 10 to 20 as well as frames 100 to 110 and 235 to 245 were taken for the first random sample by intelligent sample taking. To obtain a high degree of dispersion in the random samples, the software component performing the sample taking can specify a minimum time interval between random samples to ensure that different frames are recorded under different ambient conditions. Additionally or alternatively, the sample taking can take into account one or more attributes. For example, if random samples are selected to quantify the performance of an object detector at night, different environments can be specified, such as metropolitan, land or road. In this case, the random selection will be performed among all random samples that meet the specified criteria.

[0030] Preferably, annotating the sensor data and recording the sensor data are performed alternately or simultaneously, and if it is determined that the quality measure of at least one frame in the first random sample is below a predetermined threshold, the computer requests the recording of additional sensor data in which at least one condition attribute is within a selected value range of the first packet. The value range of the condition attribute can be selected by equipping the test vehicle with an automatic recording device that runs a selection program that triggers recording as soon as a predetermined recording condition is met, or at the request of a test driver who drives under specific conditions, for example at night. In this way, new data is recorded at least mainly for ambient conditions for which the neural network requires further training. By carefully selecting the training data, the degree of improvement per training cost is maximized. In this way, the computation time and energy consumption required for training are also reduced.

[0031] In one embodiment, receiving a corrected annotation for the data points includes receiving a plurality of provisional annotations and calculating a corrected annotation based on the plurality of provisional annotations, in particular the selection being made based on an average or majority vote. For data points of a certain type of bounding box, the average of a plurality of values ​​representing the coordinates and / or size of said bounding box can be calculated. For another type, a majority vote may be better suited. That is, to achieve a higher quality of annotation, provisional or partial annotations can be made from a plurality of labelers, on the basis of which the ground truth is calculated. This is particularly advantageous for annotation of a first batch of sensor data frames, since it also allows checking of the labeling specifications.

[0032] An aspect of the present invention also relates to a non-volatile computer readable medium comprising instructions which, when executed by a microprocessor of a computer system, cause the computer system to carry out a method according to the present invention as described above or in the appended claims.

[0033] In another aspect of the invention, a computer system is proposed, comprising a processor, a main memory, a display, a device for human input, and a non-volatile memory, in particular a hard disk or a hard disk drive device, which contains instructions that, when executed by the processor, cause the computer system to carry out the method of the invention.

[0034] The processor may be a general-purpose microprocessor, as is commonly used as the central unit of a personal computer, or may include one or more processing elements designed to perform special calculations, such as a graphics processor. In alternative embodiments of the invention, the processor may be replaced or supplemented by a programmable logic device, such as an FPGA, configured to provide a fixed range of functions, and / or may include an IP core microprocessor.

[0035] The invention will now be described in detail with reference to the drawings, in which like parts are given the same reference numerals, and in which: The illustrated embodiments are highly schematic, i.e. the spacing and lateral and longitudinal dimensions are not drawn to scale and, unless expressly stated, have no derivable geometric relationship to one another; [Brief description of the drawings]

[0036] [Figure 1] FIG. 1 illustrates an exemplary embodiment of a computer system. [Diagram 2] FIG. 1 shows an example video frame with a schematic diagram of possible data points inset at the top left. [Diagram 3] FIG. 1 is a schematic diagram showing an automated system for carrying out the method according to the invention. [Figure 4] FIG. 1 illustrates an example of data points grouped into clusters each having a different quality level. [Figure 5a] FIG. 2 is a schematic diagram showing a first step in batch processing of sensor data frames. [Figure 5b] FIG. 4 is a schematic diagram illustrating a second step in batch processing of sensor data frames. [Figure 5c] FIG. 11 is a schematic diagram illustrating a third step of batch processing of sensor data frames. [Figure 5d] FIG. 11 is a schematic diagram showing a fourth step of batch processing of sensor data frames. [Figure 5e] FIG. 11 is a schematic diagram showing a fifth step of batch processing of sensor data frames. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] FIG. 1 illustrates an exemplary embodiment of a computer system.

[0038] The illustrated embodiment includes a host computer PC including a display DIS and user interface devices, such as a keyboard KEY and a mouse MOU, and may further be connected to an external server via a network, as indicated by the cloud symbol.

[0039] The host computer PC includes at least one processor CPU, including one or more cores, a main memory RAM, and a number of devices connected to a local bus (e.g. PCI-Express) that exchanges data with the CPU via a bus controller BC. These devices include, for example, a graphic processor GPU for driving and controlling a display, a controller for connecting peripheral devices, a non-volatile memory HDD, for example a hard disk or a solid-state disk, and a network interface NC. Furthermore, the host computer can include an accelerator AI dedicated to neural networks. The accelerator AI can be configured as a programmable logic module, for example an FPGA, or as a graphic processor suitable for general calculations, or as an application-specific integrated circuit. Preferably, the non-volatile memory includes instructions that, when executed by one or more cores of the processor CPU, cause the computer system to execute the method according to the invention.

[0040] In an alternative embodiment, shown as a cloud in the figure, the computer system may include one or more servers including one or more processing elements, where the servers are connected to clients, e.g., a host computer PC, via a network. Thus, the annotation environment may be executed partially or completely on a remote server, e.g., in a cloud computing device. As a client, instead of a host computer, a mobile terminal device may also be used, i.e., the graphic user interface of the annotation environment may be executed, in particular, on a smartphone or on a tablet with a touch screen user interface.

[0041] FIG. 2 shows a camera image as an example sensor data frame with a schematic diagram of possible data points inset at the top left.

[0042] The illustrated photograph of a metropolitan scene may be an individual image or part of a video recording. Typically, the recording provided by the customer represents a sequential context and consists of video data, e.g., a 5-minute drive recorded via imaging sensors such as cameras and LiDAR sensors, or audio data, e.g., a 10-minute audio recording. The video recording may, for example, consist of a series of consecutive frames containing a series of objects. Such recordings are processed with at least one neural network to create annotations. The annotations may include multiple data points, where each data point describes a particular aspect.

[0043] A data point is a parameter that describes a particular characteristic of a recording and can be applied to all levels of detail. A level of detail can be the entire recording, a series of consecutive or random frames, an individual frame, or an object on one frame. Particular examples are annotations for cars consisting of a bounding box that represents the car's position within a certain accuracy, vertical lines marking the edges of the car, a classification that describes the type of car, and attributes that represent clipping or occlusion, i.e. turn signals, brake lights, color, etc. Data points can be classes, boxes, segments, polygons, lines, attributes such as turn signals, brake lights, color, subclasses, tracking information, occlusion ratio, clipping ratio, composite classes that describe the importance of an object / frame / clip, tone, text, or any other information that can be computed automatically.

[0044] In the inset at the top left of the figure, various data points for the car are shown. The car can be of different types, for example a vendor car, an SUV or a sports wagon. The position or even the dimensions of the car are generally shown by a bounding box, i.e. a rectangular frame or a cuboid that surrounds the car. The vertical lines indicate the boundaries of the car. Another possible data point for the car is the activation state of a light-emitting display, such as for example a driving direction indicator or a blinker, which is shown in the inset.

[0045] There are many cars in the frame, and each car is surrounded by a bounding box. Cars may be fully visible, e.g., driving directly in front of the camera, or they may be occluded. Traffic density in large urban scenes can impair annotation quality, e.g., by making it difficult to accurately determine the bounding box boundaries due to occlusions.

[0046] 3 is a schematic diagram of an automation system for carrying out the method according to the invention, which implements the different steps of the method in dedicated components and is well suited to execution in a cloud computing environment.

[0047] In the first step, i.e. "data recording", the unclassified recording by the customer is received. The recording can be normalized, e.g. split into sensor data frames or images, to allow uniform processing. This step can also include an enrichment phase, in which the sensor data frames of the recording are automatically enriched with metadata related to the measurement of the automation quality. Thus, for example, each image can be associated with the recorded geographical region, in particular based on GPS coordinates received at the same time as the image. In the context of autonomous driving, metadata or state attributes related to the quality of the annotation may include weather conditions, road type, light conditions and / or time of day. These state attributes represent the conditions during the recording of the sensor data frames and may also be referred to as static. Similarly, other state attributes that may affect the labeling quality of, for example, object recognition, such as the size of the bounding box (large objects are easier to identify), are only derived from the annotation performed and therefore may also be referred to as dynamic.

[0048] For efficiency of automation, it is advantageous to process batches of frames or individual images together in subsequent steps. In projects involving nested recording and processing of images, it may be advantageous to accumulate frames recorded under the same ambient conditions until a set batch size is reached before continuing to perform further processing steps.

[0049] In a second step, or "scheduler", various batches of sensor data frames or individual images are scheduled for annotation by the automation engine. The scheduler can now select one or more automated components for execution by the automation engine that annotate the frames with one or more data points. Furthermore, the scheduler can select a batch of frames to be processed based on the availability of new versions of the automated components. The automated components can generate the coordinates and object classes of individual data points, e.g. vertical lines, or multiple related data points, e.g. bounding boxes. The automated components can be neural networks or other machine learning based techniques that learn from data samples in the form of supervised, semi-supervised, or unsupervised learning.

[0050] In a third step, i.e. the "automation engine", the batch of sensor data frames is processed in at least one automation component that maps data points to multiple frames. The automation system generates each data point of any type via an automation component, so that the automation component is a central part of the automation system's workflow. Preferably, the data points are provided with metadata that precisely describes the version of the automation component used. The automation engine includes techniques, e.g. a special database, for precisely storing the relevant metadata via the automation component. Some of the state attributes associated with the data points can be determined by a dedicated automation component. The "context", i.e. state attributes of a data point, may include attributes that are themselves data points. Thus, for example, the accuracy of the placement of a vertical line may depend on the size of the bounding box in which the line is to be drawn.

[0051] In the fourth step, i.e. "clustering", individual data points of a certain type are grouped based on state attributes. A type of data point can be associated with a certain state attribute. State attributes for a bounding box can include, for example, the size of the bounding box, the time and / or weather conditions at the time of image recording and / or partial occlusion of the object. The values ​​of the state attributes of the individual bounding boxes can form multiple clusters in the multi-dimensional space expanded by the state attributes. Different clusters can be associated with different qualities of the annotation.

[0052] [Table 1]

[0053] Listing 1 shows an example context that includes multiple state attributes for two example data points, namely bounding boxes B01 and B02 that indicate the positions of the respective objects. Each state attribute describes a condition that may potentially affect the quality of the annotation.

[0054] In this way, an automated system can determine clusters in a multi-dimensional space based on a large number of individual data points of the same type, in particular using nearest neighbor algorithms and / or unsupervised learning approaches and / or machine learning classification models. The computed clusters can be analyzed to set criteria for grouping of data points and / or prediction of annotation quality by defining value ranges representing at least one of the state attributes of the data points.

[0055] Preferably, grouping based on defined value ranges is performed for several condition attributes. The grouping here can be based for example on bounding box dimensions, which can be accurately positioned for objects located close to the camera or for large objects. In contrast, the relative error in positioning the bounding box around small objects at a distance can be significant. Thus, a correlation can be obtained between larger dimensions and higher quality bounding boxes. Weather is another condition attribute that can be used to group data points, for example due to reduced contrast and / or image distortion caused by water droplets on the camera objective lens. Other condition attributes may possibly not significantly affect the quality variation of the data points and can be ignored, so for example the field of view or the viewing angle of the camera used to record the images can be constant for all images recorded by said camera. Grouping based on value ranges can also be performed by neural networks or machine learning classification models.

[0056] In the fifth step, i.e. "sample inspection", a quality check of the random sample of data points is performed. In the first phase, i.e. "random sampling", a number of data points for quality control are selected based on the random sample request. The frequency and / or size of the random samples taken for a group of data points can be selected depending on the predicted quality of the data points in that group, i.e., more frequent random sampling can be performed for data points associated with state attributes suggesting poor quality. In the second phase, i.e. "inspection and correction", the frames can be shown to a human annotator along with the corresponding annotations, e.g. bounding boxes, and the annotator can be queried whether the bounding boxes are correct. Alternatively, the annotator can be shown a user interface to adjust the bounding boxes and / or add bounding boxes in case of "false negatives" in order to annotate objects missed by the neural network. The automated system determines a quality measure from the type and number of corrections made by the human annotator. Preferably, the quality measure is selected such that missed objects are weighted more heavily than bounding boxes whose placement should be refined.

[0057] In the sixth step, "Did Sample Inspection Pass?", the system determines whether a quality measure of the random sample exceeds a predefined threshold, which indicates sufficient annotation quality. If the automated system detects that the threshold is exceeded (yes), the group of individual images containing the selected sample is exported and handed over to the customer. If the threshold is not exceeded (no), execution continues with the seventh step.

[0058] In the seventh step, i.e. "Necessary for the dataset?", it is calculated whether the manually corrected random sample should be used for a new training of the automation component on the data point. Whether this is the case may depend on how many images were recorded under the same conditions already used for training the model. If not (no), the group of data points from which the sample was taken is sent back to the scheduler (again automated by the post-trained model). As soon as a newly trained automation component is available for that data point, the scheduler sends the group of data points to the automation engine for new processing. If the corrected random sample should be used for post-training (yes), the manually annotated data point is supplied to the training / validation or test dataset for the respective neural network / automation component. These datasets are indicated by cylinders.

[0059] In the eighth step or "flywheel", the neural network or automation component that generated the data points rejected by the random sample inspection is post-trained. Additions to the neural network improve the quality of the automation. Preferably, the automation component is improved to the point where manual inspection is no longer required for as many clusters as possible. To allow rapid improvements in efficiency, the post-training iteration time should be as short as possible.

[0060] Flywheel includes techniques for efficiently storing and versioning the training datasets for each automated component or each data point type to monitor changes in the training datasets and to automatically trigger post-training as soon as a predefined or automatically calculated threshold for changes in the training dataset (e.g., a set number of new examples) is exceeded. Additionally, Flywheel includes techniques for automatically deploying post-trained neural networks to automated components and informing a scheduler regarding version changes.

[0061] An additional step of deliberate data detection can be performed when new data frames are recorded simultaneously or alternately with the annotation of the data frames. The automation component is improved by multiple training iterations on a data set that is constantly refined and that over time maps better and better to the real-world variance. A systematic approach can be pursued to collect data frames that represent situations where the automation results are most burdensome, based on the confidence level or the error probability per cluster, at least for static state attributes. For example, a situation may arise where the automatic annotation of sensor data frames recorded at night leads to an unacceptable annotation quality. As soon as this is detected in a sample inspection, deliberate recording of overnight data can be requested to achieve an improvement of the training data set of the automation component under the ambient conditions here. In particular, the collection of additional training data sets under each problematic ambient condition can be determined depending on the confidence level or the error probability calculated for the corresponding cluster. All data recorded under the same conditions can be used for training anew. As soon as the correction of the misannotated sensor data frame is performed, the correction is directly fed into the training data set of the corresponding automation component. However, it is usually not necessary to manually correct all data for a particular cluster and a particular data point. Instead, only random samples are collected and corrected for post-training up to a threshold value. The remaining part of the data is automatically planned for a new process in a more advanced version of the automation component. The intentional data collection includes a technique for selecting samples of interest based on clusters up to a predetermined amount for manual correction. Furthermore, the intentional data collection preferably includes a technique for identifying samples with poor quality that are not required for a new training of the automation process in the more advanced version of the respective automation component.

[0062] If the automatic annotation of the samples inspected in the sixth step is of sufficient quality, these annotations can be handed over to the customer. In the ninth step, i.e. "customer sample inspection", a random sample of the exported sensor data frames can be inspected by the customer to ensure that the annotation meets the customer's specifications and that the required annotation quality has been achieved. If the customer rejects a group of frames, in the tenth step, i.e. "correction", the random sample or the entire group of frames is processed manually. The ninth and tenth steps are optional and can therefore be omitted.

[0063] In the tenth step, "correction," manual annotation is performed on the entire group of random samples or sensor data frames rejected by the customer. Optionally, the manually annotated frames can be exported by the customer for new testing. The manually annotated frames are preferably used to post-train the neural network by providing the corrected data as a training, validation, or test dataset.

[0064] FIG. 4 shows an example of data points grouped into clusters with different quality levels.

[0065] An excerpt of the sensor data frames recorded by the camera is shown, each showing a vehicle. Around the vehicle, a bounding box is drawn that encloses the vehicle's outline. Additionally, the vehicles are annotated with vertical lines that indicate the vehicle's edges, allowing the relative angle between the vehicle and the camera to be inferred inversely. Two different types of data points are shown here, with the bounding boxes representing a first order data type that can exist independently in the image or in the sensor data frame. In contrast, the vertical lines are drawn only on the identified vehicle and are therefore second order data points.

[0066] The relative accuracy of the bounding box depends, for example, on the size of the objects contained within, since large objects can be better identified than small or distant objects. Note that the size of the bounding box also has a significant impact on the accuracy of vertical lines. Another influencing factor on the quality of annotation with vertical lines may include, for example, lighting conditions and occlusion rates, which may represent relevant state attributes.

[0067] The image sections or identified vehicles shown are clustered into three groups, each with a different predicted or calculated error probability. The left column shows an example of cluster 1, which contains high quality data points (or vertical lines), where a 2% error probability (Error WS) prevails. The center column shows an example of cluster 2, which contains medium quality data points, where an 8% error probability (Error WS) occurs. The right column shows an example of cluster 3, which contains low quality data points, where an 18% error probability (Error WS) occurs.

[0068] The formation of clusters allows the calculation of value ranges for relevant state attributes, e.g., occlusion rates above 30% are correlated with poor annotation quality. Although the form of the clusters can be complex, especially with many relevant state attributes, such clusters can be conveniently described by a trained neural network or machine learning classification model.

[0069] Figure 5a is a schematic diagram showing a first step in batch processing of sensor data frames, which may be performed in an automated system similar to that shown in Figure 3, with steps not shown being performed as part of the batch processing.

[0070] Splitting complex annotations into individual data points allows fine-grained observation of state attributes related to quality measures. Furthermore, the required computation time is reduced since only data points that are available to, respectively, e.g., a post-trained neural network need to be observed, while other data points in the sensor data frame can be left untouched. For a general treatment of data points, this document describes a method that, in the context of a simplified example, includes only one type of data points (e.g., bounding boxes around objects) and two clusters for that type of data points (cluster A; low quality, cluster B: good quality).

[0071] The sensor data frames acquired as input data, e.g. camera images, are divided into batches of fixed size to allow unified processing by the automation engine. The figure shows two batches of 500 frames each with sensor data. The automation engine executes a neural network for object recognition, which assigns bounding boxes to objects in the frames. After each batch is annotated and the individual data points are further associated with context from different state attributes, the data points are grouped into cluster A (dotted line) with poor data point quality and cluster B (dashed line) with good data point quality. From each camera image or frame, e.g. three data points in cluster A and two data points in cluster B are obtained. In the illustrated example, cluster A contains 2000 data points and cluster B contains 1100 data points (DP).

[0072] FIG. 5b is a schematic diagram illustrating a second step in the batch processing of sensor data frames.

[0073] A sample inspection is performed as soon as the cluster reaches a predetermined volume and / or as soon as a set time interval has elapsed. Based on a predefined random sample request, some data points are taken as a random sample. Here, manual steps of inspection and correction are performed (other steps can be fully automated by a computer), and the automated system receives the corrected data points for the random sample. For simplicity, each entire cluster is taken as a random sample here.

[0074] In the illustrated example, cluster A has reached the size threshold for sample inspection (indicated by the magnifying glass symbol), whereas cluster B is not inspected initially (indicated by the hourglass symbol). As a result, it turns out that, for example, 30% of the data points in cluster A must be corrected (e.g., correct 30%) to achieve the desired quality level. For simplicity, we assume that all of the data points of the corrected random sample are used for post-training the neural network. Thus, 600 corrected data points are provided for recording into the training data set.

[0075] FIG. 5c is a schematic diagram showing a third step in batch processing of sensor data frames.

[0076] In the example, another batch of sensor data frames or camera images is taken as input data and the latest batches 21 and 22 are shown to be processed. In the example shown, the amount of cluster B did not change, but the trigger condition for sample inspection of cluster B was met, i.e., a set number of batches (20) were processed, after which it was determined that a random sample should be taken and inspected or corrected in all clusters that had not been inspected before (correction / termination of all open clusters).

[0077] FIG. 5d is a schematic diagram illustrating a fourth step in batch processing of sensor data frames.

[0078] Sample inspection of cluster B (indicated by the magnifying glass symbol) showed that 10% of the cluster's data points must be corrected (e.g., correct 10% of the data points) to achieve the desired quality level. Here, all of the corrected random sample data points are used for post-training the neural network, so again 110 corrected data points are provided for recording in the training dataset. In batch processing, there is additionally the option to train on all annotated sensor data frames of the corrected batch, or only on the corrected data points.

[0079] Figure 5e is a schematic diagram showing the fifth and final step of batch processing of sensor data frames. Similar to Figure 3, here modules for data recording and a scheduler are also shown.

[0080] Batch 1 and batch 2 are shown, with many other batches indicated by the ellipses. After the data points of the batches were classified into clusters A and B, and the random samples were inspected and corrected, the neural network was post-trained. As soon as the desired quality level was achieved in further random samples, these batches can be delivered with statistical quality acceptance (delivery to customer). In this case, the majority of the annotated sensor data frames can be delivered without the need for manual post-processing.

[0081] The described method can be used for sensor data frames, i.e. point clouds, of LiDAR sensors or for multi-sensor setups, where independent grouping and correction is performed for different types of data points. Only samples required for training are manually corrected, allowing a large portion of the input data to be automatically annotated as soon as a post-trained neural network is available for each type of data point.

[0082] By utilizing the correlation between annotation quality and state attributes, the method according to the present invention can deliberately apply manual work to the rapid improvement of neural networks, which can then be used to create automatic annotations for delivery to customers.By processing different types of data points separately and creating annotations anew only after the existence of, for example, a post-trained neural network, the computation time is used particularly effectively.In total, large annotation projects, which are required for, for example, validation, are significantly accelerated.

Claims

1. 1. A computer-implemented method for automatically annotating sensor data, the method comprising: receiving a plurality of sensor data frames; annotating a plurality of sensor data frames using at least one neural network, said annotating comprising associating at least one data point with each sensor data frame and at least one state attribute with each data point; grouping the data points based on the at least one state attribute, a first group including data points for which the at least one state attribute is within a defined range of values; selecting a first random sample from one or more data points from the first group and determining a quality measure for the data points in the first random sample; Including, If the computer detects that the quality measure of the first random sample is below a predetermined threshold, the method further comprises: receiving corrected annotations for the data points in the first random sample, post-training the neural network based on the data points in the first random sample, and selecting a second random sample from one or more data points in the first group that were not in the first random sample; annotating the sensor data frames of the second random sample with a post-trained neural network and determining a quality measure for the data points in the second random sample; further comprising Upon the computer detecting that the quality measure of the first random sample and the quality measure of the second random sample exceed a predetermined threshold, the method further comprises: annotating the remaining sensor data frames of the first group with the neural network; exporting the first group of annotated sensor data frames; The method further comprises:

2. If the computer detects that the quality measure of the second random sample is below a predetermined threshold, the method further comprises: receiving corrected annotations for the data points in the currently examined random sample and post-training the neural network based on the data points in the currently examined random sample; selecting another random sample from one or more data points in the first group that were not part of a previous random sample; annotating the sensor data frames of the other random sample with the neural network and determining a quality measure for the data points of the other random sample; further comprising the further steps are repeated until the computer detects that the quality measure of frames in the further random sample exceeds the predetermined threshold or that no sensor data frames containing uncorrected data points remain for the random sample; As soon as a quality measure of the random sample of sensor data frames exceeds the predetermined threshold, the method comprises: annotating the remaining sensor data frames of the first group with the neural network; exporting the first group of annotated sensor data frames; further comprising: The method of claim 1.

3. At least one of the state attributes is determined by a dedicated neural network based on the sensor data frames; and / or at least one of the state attributes is determined based on additional sensor data recorded contemporaneously with the sensor data frame; The method of claim 1.

4. the sensor data frame comprises an image data frame, i.e., imaging sensor data; For an image data frame, the state attributes are geographical region, time of day, weather conditions, visibility conditions, road type, distance to objects and / or traffic density, size of the bounding box, extent of occlusion and / or clipping, ego-vehicle speed, camera parameters, color range and / or contrast measure of the area encompassed by the bounding box, direction of travel of the ego-vehicle, astronomical information such as the position of the sun relative to the direction of travel of the ego-vehicle, and / or For an image data frame, the at least one data point comprises an object position, an object class, bounding box coordinates, line coordinates, object clipping, object occlusion, correlation of an object in an image data frame with an object in a preceding or subsequent image data frame, and / or activation status of a light indicator such as a turn signal or brake light. The method of claim 1.

5. the sensor data frames include acoustic frames, i.e., acoustic sensor data; For an acoustic frame, the state attributes are a geographical region, a speaker's gender and / or age, a measure of the spatial size and / or background noise, and / or For an acoustic frame, the at least one data point comprises a phonon and / or one or more words of text identified from the acoustic frame. The method of claim 1.

6. wherein the step of grouping the plurality of data points comprises determining clusters in a multidimensional space, in particular using a nearest neighbor algorithm and / or an unsupervised learning approach and / or a machine learning classification model. The method of claim 1.

7. annotating the sensor data frames includes associating at least one data point of a first type and at least one data point of a second type with each sensor data frame; the first type of data points are grouped based on the determination of clusters in a first multidimensional space, and the second type of data points are grouped based on the determination of clusters in a second multidimensional space; The multidimensional space for a single data point is expanded by multiple state attributes. The method of claim 6.

8. the first group is defined based on a first cluster in which the at least one state attribute is within a defined first range of values, and the second group is defined based on a defined second range of values; the first value range and the second value range are disjunctive with respect to at least one state attribute and / or with respect to all state attributes associated with each data point; The method of claim 6.

9. determining an error probability for each data point based on whether the data point is in the first group or the second group; For data points in a group with a higher error probability, more random samples are drawn. The method of claim 8.

10. annotating the sensor data frames with the first type of data points based on a first neural network; annotating the sensor data frames with the second type of data points based on a second neural network; the further method steps for the first type of data points and the further method steps for the second type of data points are carried out independently of each other. The method of claim 1.

11. selecting the sensor data frames for the first random sample, in particular the random selection of individual images for data points related to object recognition and / or the random selection of a stack of consecutive frames for data points related to object tracking, depends on the type of data points for which a quality measure is to be determined; The method of claim 1.

12. the annotating the sensor data frames and the receiving the sensor data frames are performed alternately or simultaneously; If the computer detects that the quality measure of the random sample is below a predetermined threshold, transmission of sensor data frames in which the at least one state attribute is within a defined value range is requested. The method of claim 1.

13. receiving corrected annotations for the data points comprises receiving a plurality of provisional annotations; and calculating corrected annotations based on the plurality of provisional annotations; In particular, selection is based on average or majority vote; The method of claim 1.

14. A non-volatile computer readable medium comprising instructions which, when executed by a processor of a computer system, cause the computer system to perform the method of any one of claims 1 to 13.

15. 1. A computer system including a host computer, the host computer includes a processor, a main memory, a display, an input device, and a non-volatile memory; The non-volatile memory contains instructions that, when executed by the processor, cause the computer system to perform the method of any one of claims 1 to 13. Computer system.

Images