Method and system for annotation of sensor data

Abstract

The present invention relates to a computer-implemented method for annotation of driving scenario sensor data, the method comprising receiving raw sensor data including a plurality of successive LiDAR point clouds and / or a plurality of successive camera images; identifying objects in each image and / or each point cloud of the camera data using one or more neural networks; correlating each object in the successive images and / or successive point clouds; removing false positive results based on validity criteria; and exporting the annotated sensor data of the driving scenario.

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method, a computer-readable data carrier and a computer system for the automatic annotation of sensor data of driving scenarios. [Background technology]

[0002] Autonomous driving promises unprecedented comfort and safety in everyday traffic. However, despite significant investments from various companies, existing approaches are only usable under limited conditions or can only achieve partially autonomous behavior in practice. This is due to a lack of a sufficient number and variety of driving scenarios. Annotated sensor data from driving scenarios is required for training autonomous driving functions. A common approach for data annotation, also known as data enrichment, involves the manual annotation of each image using a large number of human resources. Therefore, traditional data enrichment methods are very time-consuming, error-prone, and therefore prohibitively expensive.

[0003] Semi-automatic approaches involving keyframe annotation and interpolation / extrapolation offer some (limited) improvement. Such an approach, shown schematically in Figure 2, involves selecting multiple manually annotated images as keyframes using propagation / extrapolation. After manually processing the keyframes, annotations for frames located between them are generated by interpolation. Objects on the keyframes are then manually identified by a human, who then connects related objects using extrapolation. The interpolation process then uses this information (object identification and connection) to generate annotations for the same objects on all frames between the keyframes. In theory, increasing the spacing between keyframes can improve the efficiency of the mechanism, since more annotations are automatically generated. However, increasing the distance between keyframes significantly increases the amount of manual correction required. For example, objects that are only briefly visible in non-keyframes must be occluded by manual intervention. In this case, the automated approach reaches its limits very quickly.

[0004] Therefore, traditional annotation strategies can only undertake small to medium-sized data enrichment projects, and other higher-level functions such as autonomous driving feature validation, data selection, or scenario library creation are not feasible due to the huge manual effort and associated costs. Summary of the Invention [Problem to be solved by the invention]

[0005] It is therefore an object of the present invention to provide an improved method for annotation of driving scenario sensor data, and particularly desirable is an automated annotation method that minimizes the need for human intervention. [Means for solving the problem]

[0006] The above problem is solved by a method for annotation of driving scenario sensor data according to claim 1, a computer-readable data carrier according to claim 11 and a computer system according to claim 12. Advantageous developments are the subject of the respective dependent claims.

[0007] That is, a computer-implemented method for annotation of driving scenario sensor data, comprising: receiving raw sensor data, the raw sensor data comprising a plurality of successive LiDAR point clouds and / or a plurality of successive camera images; - identifying objects in each image and / or each point cloud of camera images using one or more neural networks, wherein the identified objects are preferably assigned an object class, an object position, an object size and / or an object extent, in particular the coordinates of a bounding box containing the object; Correlating each object in successive images and / or successive point clouds; - removing false positive results based on validity criteria; exporting the annotated sensor data of the driving scenario; A method is provided, comprising:

[0008] Advantageously, neural networks for object identification can be optimized for high recall, i.e., for identifying as high a percentage of actually present objects as possible, since subsequent removal of such false positive results based on validity criteria effectively minimizes erroneous identifications without manual intervention.

[0009] The present invention is based on the observation that semantic information derived from the time correlation of each object in the individual images of a capture can be effectively utilized using a series of steps / techniques, which first advantageously involve identifying objects that are tracked or correlated with each other through the series of images. The techniques used further include removing false positives where objects are incorrectly identified, filling voids, optimizing object size and object position by regression, and smoothing trajectories. Some steps / techniques, such as regression of object size and / or object position, may be optional.

[0010] The present invention introduces a paradigm shift where annotation is done automatically upfront and only quality control is required to complete the project successfully. High quality annotations can be achieved by automatically applying automatic annotation methods to raw data. Much faster (with better project execution times), - Lower cost (less manual effort), -Consistent quality, In contrast, prior art annotations are based on workforces in low-cost countries with large numbers of people with little experience in creating annotations. This typically leads to significant quality issues and extensive post-processing cycles, which further delay and increase the cost of the project.

[0011] Our method achieves a new degree of quality, consistency, and automation, i.e., a much higher throughput rate than previous annotation methods. Only quality assurance needs to be covered by human labor, resulting in significant cost savings.

[0012] This makes it much more feasible to perform projects on a scale that was previously infeasible due to the cost of manual annotation, as well as higher-level functions like data selection, because annotation is now a matter of computational time, not manual costs. Thus, all raw data can be annotated upfront, and scenes of interest can then be selected based on the annotations. The manual costs of quality assurance can then be applied in a subsequent step to only the data of interest.

[0013] In a preferred embodiment of the invention, the sensor-raw data comprises a LiDAR point cloud that is divided into at least two regions, where a neural network of a first architecture is used in the first region and a neural network of a second architecture that is different from the first architecture is used in the second region. Automatic selection allows optimal adaptation of the neural network used to meet requirements.

[0014] In a particularly preferred embodiment of the present invention, the first region includes the closer surroundings of the measurement vehicle where the raw sensor data was recorded, while the second region has the smallest distance to the measurement vehicle. Preferably, a center-point-based architecture is used for the neural network for object identification in the first region, while a PointRCNN-based architecture is used in the second region. For relatively distant objects, which often have only a small number of measurement points, a neural network with the PointRCNN architecture allows for the generation of bounding boxes with appropriate dimensions. For nearby regions, a neural network with the center-point architecture allows for good discrimination between various object classes. Therefore, the cost of manual correction is minimized.

[0015] In a preferred embodiment of the present invention, the method further includes removing duplicates before correlating the objects, and the removal of duplicates is preferably based on the overlap criteria and / or confidence level of the neural network. If two bounding boxes have a significant overlap, this can be determined by comparing with a threshold value of about 0.98 and can be assumed to relate to the same object. Additionally, only objects with low confidence levels can be discarded. The confidence level can be calculated using a specially trained network or can be approximated by the data of the network used for object identification.

[0016] In a particularly preferred embodiment of the invention, duplicates of objects identified in an image and / or point cloud are removed by checking whether two overlapping objects, i.e., a first object identified with a first confidence level and a second object identified with a second confidence level lower than the first confidence level, have an overlap, in particular an IoU (Intersection-over-Union), above a set threshold, and if so, discarding the second object as a duplicate. In principle, it is also possible to arbitrarily select the duplicates to be discarded.

[0017] In a preferred embodiment of the present invention, correlating each object involves combining objects in consecutive frames, i.e., images and / or point clouds, such that an object in a first frame is correlated with an object in a second frame if the objects belong to the same object class and the overlap, particularly the IoU (Intersection-over-Union), exceeds a set threshold. Because LiDAR sensors, and especially cameras, perform multiple measurements per second, the vehicle's surroundings typically undergo gradual changes. By considering the temporal correlation between consecutive frames, further information can be obtained and / or the validity of object identification can be verified.

[0018] In a particularly preferred embodiment of the invention, correlating each object involves predicting the object's position on subsequent images and / or on subsequent point clouds using Gaussian process reduction or a Kalman filter and / or tracking the object in successive images using a factor graph, i.e., a bipartite graph that factorizes a probability distribution. The mentioned methods are particularly advantageous for calculating correlations.

[0019] Particularly preferably, after each object is correlated, missed objects are corrected, where if three or more consecutive frames are observed and an object in a first frame is correlated with an object in a third frame but not identified in an intermediate frame, the object is inserted into the second frame. Alternatively or additionally, it is conceivable to observe a larger number of consecutive frames, thereby filling such gaps that include multiple frames. Missed objects may arise, in particular, due to occlusion or partial obscuration. An additional criterion can be provided such that gaps consisting of two or more frames are filled only if another object is identified at or around the relevant location. In particular, based on the observation of the bounding box of the other object, it can be determined whether the other object is large enough to cause occlusion.

[0020] In a preferred embodiment of the present invention, the validity criteria for eliminating false positive results are based on the height above ground, duration, and / or the confidence level of the neural network. An object can be discarded as invalid if more than a set amount is identified below the ground surface. Furthermore, an object can be discarded as invalid if it is detected above the ground by more than a set threshold. The validity criteria and / or threshold can be selected depending on the object class. Furthermore, an object can be discarded as invalid if it has only a very short duration, i.e., in particular if it is identified only in one frame but not in adjacent frames. When considering duration, possible obscuration can be advantageously taken into account. Alternatively or complementary, the discarding of invalid objects can also depend on the confidence level of the neural network, so that only objects with a low confidence level are discarded. The confidence level can be calculated using a specially trained network or can be approximated by data from the network used for object identification.

[0021] In a preferred embodiment, the method further comprises optimizing object size and / or object location in each image and / or each point cloud of the camera data by regression. Since object identification has already been performed, the evaluation can be optimized specifically for accurate bounding box computation.

[0022] In a preferred embodiment, the method further comprises unifying the sizes of correlated objects in successive images and / or successive point clouds, and preferably unifying the sizes of correlated objects is performed or not depending on the object class. For example, vehicles such as cars or motorcycles must have the same size. The unified size makes the objects appear more natural.

[0023] Advantageously, the sensor-raw data includes the LiDAR point cloud and simultaneously captured camera data, and the correlation of objects is performed taking into account the relative spatial orientation of the LiDAR sensor and the camera. In this way, the spatial information of the LiDAR sensor and the higher resolution of the camera can be optimally utilized.

[0024] In a preferred embodiment, at least one neural network for identifying attributes is applied to the identified objects. Preferably, at least one attribute, in particular the vehicle's turn signal status, is calculated based on the camera image and assigned to the object in the LiDAR point cloud. The at least one attribute and the one or more neural networks used for identification can be selected depending on the object class. When both the point cloud and the camera image are present, sensor fusion can be advantageously performed, in which the camera image is used for attribute identification and the attribute is assigned to the fused object.

[0025] The invention further relates to a computer-readable data carrier comprising instructions which, when executed by a processor of a computer system, cause the computer system to carry out the method according to the invention.

[0026] The present invention further relates to a computer system comprising a processor, a man-machine interface and a non-volatile memory, the non-volatile memory containing instructions which, when executed by the processor, cause the computer system to carry out the method according to the present invention.

[0027] The processor may be a general-purpose microprocessor, such as those typically used as central processing units in workstation computers, or may include one or more processing elements adapted to perform specific calculations, such as a graphics processing unit. In alternative embodiments of the invention, the processor may be replaced or supplemented by a programmable logic module, such as a field programmable gate array, configured to perform a predetermined number of operations and / or include an IP core microprocessor.

[0028] The present invention will now be described in detail with reference to the drawings, in which like elements are designated by the same reference numerals, and in which the illustrated embodiments are highly schematic, i.e., the spacing and lateral and longitudinal dimensions are not to scale and, unless otherwise specified, the dimensions have no determinable geometric relationship to one another. [Brief explanation of the drawings]

[0029] [Figure 1] FIG. 1 illustrates an exemplary circuit diagram of a computer system. [Figure 2] FIG. 1 is a schematic diagram showing multiple images semi-automatically annotated using keyframes. [Figure 3] 1 is a schematic flow chart illustrating an embodiment of the method according to the invention for annotation of driving scenario sensor data; [Figure 4] FIG. 1 is a schematic diagram showing a number of images on which automatic object identification has been performed. [Figure 5] FIG. 1 is a schematic diagram showing a plurality of images for which temporal correlations have been calculated; [Figure 6] FIG. 1 is a schematic diagram showing a series of images containing trajectories from which subsequent evaluation can be performed. [Figure 7] FIG. 1 illustrates an embodiment of an automated system for carrying out the method according to the present invention. [Figure 8]FIG. 1 illustrates one embodiment of a method for automatically training an attribute detector. [Figure 9] FIG. 1 illustrates an embodiment of an operating surface for quality control. DETAILED DESCRIPTION OF THE INVENTION

[0030] FIG. 1 illustrates an exemplary embodiment of a computer system.

[0031] The illustrated embodiment includes a host computer PC equipped with a monitor DIS and input devices such as a keyboard KEY and a mouse MOU.

[0032] The host computer PC includes at least one processor CPU having one or more cores, a working memory RAM with selectable access, and a number of devices connected to a local bus (e.g., PCI Express) exchanging data with the CPU via a bus controller BC. These devices include, for example, a graphics processor GPU for controlling a display, a controller USB for connecting peripherals, a non-volatile memory such as a hard disk or solid-state disk, and a network interface NC. The non-volatile memory may contain instructions that, when executed by one or more cores of the processor CPU, cause the computer system to carry out the method according to the invention.

[0033] One embodiment of the present invention is represented by a cloud in the figure, where a host computer may include one or more servers, each including one or more computing elements such as a processor or FPGA, and connected via a network to a client including a display and input device. Thus, the method for generating a simulation scenario may be partially or completely performed on a remote server, for example, in a cloud computing setup. Instead of a PC client, the graphical user interface of the simulation environment may be displayed on a portable computing device, in particular a tablet or smartphone.

[0034] FIG. 2 shows a schematic of a series of images that are semi-automatically annotated using keyframes.

[0035] The first, third, and fifth images are keyframes (indicated by bold frames), while the second and fourth images are treated as intermediate images (indicated by dashed frames). In the first step (1. Annotate), the desired annotations are made manually to the first keyframe, e.g., by drawing bounding boxes around identified objects, and additional attributes, such as object class, are added. In the second step (2. Extrapolate), the expected positions of the objects annotated in the first keyframe in the second keyframe are calculated and manually corrected. In some cases, newly added objects are annotated completely manually. In the third step (3. Extrapolate), the expected positions of the already annotated objects in the third keyframe are calculated and manually corrected. Once all keyframes have been annotated, in the fourth step (4. Interpolate), the object positions in the intermediate frames are calculated by interpolation, and attributes are transferred.

[0036] Increasing the automation rate by increasing the keyframe interval is limited because the extrapolation, and possibly even interpolation, of object positions is no longer accurate enough, and the cost of manual correction increases very quickly.

[0037] The introduction of temporal correlation, which automatically correlates annotations of the same object with each other over time, combined with neural network-based object identification, can effectively overcome the limitations of current semi-automation. Temporal correlation is based on the fact that successive images only change gradually over time. Logical connections between objects can be automatically identified and exploited to extract higher-level semantic information, ultimately enabling an object identification-based approach that leads to a significantly higher degree of automation.

[0038] In FIG. 3 a schematic flow chart of an embodiment of the method according to the invention for annotation of driving scenario sensor data is shown.

[0039] In a first step S1 (identifying objects), a neural network for object detection is used to identify preferably as many objects as possible in each image or each point cloud.

[0040] A series of images in which the neural network has identified a series of objects in a first round is shown schematically in Figure 4. For example, the image may be of a car as shown, but essentially various object types or classes, i.e., vehicles as well as pedestrians, animals and obstacles or any respective static or dynamic object of interest, are identified in each image of a clip or sequence of images and / or in the LiDAR point cloud.

[0041] In a preferred embodiment, different architectures of neural networks are used to identify objects depending on object class and / or region of sensor data. In particular, different architectures of neural networks can be used in different regions to identify objects in LiDAR point clouds, where the region is preferably divided into near and far regions around the measurement vehicle.

[0042] Here, a distant field architecture based on pointRCNN, as described by Shaoshuai Shi et al., "PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud", arXiv:1812.04244v2, 16.05.2019, is preferably used, which is particularly good at proposing meaningful box sizes and is therefore suitable for annotating distant objects, which may sometimes consist of only a few measurement points.

[0043] In the proximity region, a center point-based neural network architecture is preferably used, as described in Tianwei Yin et al., "Center-based 3D Object Detection and Tracking", arXiv:2006.11275v1, 19.06.2020, which is well suited to distinguishing between different object classes and therefore provides better results in the proximity region where there are many measurement points.

[0044] According to alternative embodiments of the present invention, different network architectures can be combined for better coverage.

[0045] In the second step S2 (discard duplicates), duplicates are removed, which are cases where the object detector used in the frame (image or point cloud) under consideration has generated multiple bounding boxes around one object. This occurs because neural networks are usually optimized for as high a coverage as possible to avoid missing objects, resulting in many duplicates. Removal or discarding of duplicates can be based on filtering rules that can include one or more criteria.

[0046] Here, the criterion can consider the overlap between two bounding boxes, in particular the Intersection-over-Union (IoU), also known as the Jaccard coefficient. If the overlap between two boxes exceeds a set threshold, in particular 0.98 for IoU, they are considered duplicates. The set threshold can also be selected depending on the identified object class. It can also be assumed that, in order for two boxes to be considered duplicates, in addition to the set overlap, the same object class must be present. Furthermore, the confidence level of the network used for object identification can be used as a criterion for the filtering rules and / or can be considered in the criteria combined with the object class. Furthermore, if duplicates are identified, the selection of boxes to be eliminated can be based on the assigned confidence level, and can be configured to eliminate boxes with lower confidence levels in particular.

[0047] In the third step S3 (temporally correlating objects), the remaining objects are observed along their respective time axes to confirm temporal correlation. This is shown diagrammatically in Figure 5, which shows three consecutive frames in which the vehicle is indicated by an arrow as being the same across frames. By determining which objects correlate with each other, the objects can be linked into chains that can then be further evaluated and used to fill in any identification gaps in intermediate frames.

[0048] Object correlation can be performed as one-way or two-way binding, and the first round of identification gaps can be bridged based on such binding, thereby overcoming obscured, lost, and overlooked objects.

[0049] In a first embodiment of the invention, one-way combining is performed by estimating the object's position in subsequent images. During object identification, if an object is found within the estimated range that satisfies certain validity tests (class type, Ackermann model for vehicle dynamics, ...), these two object instances are marked as belonging to the same object.

[0050] Gaussian process reduction or Kalman filtering can be used as a strategy for estimating location. Gaussian process reduction is described in Rohit Tripathy et al., “Gaussian processes with built-in dimensionality reduction: Applications in high-dimensional uncertainty propagation,” arXiv:1602.04550v1, February 15, 2016. Kalman filtering, also known as linear-squared estimation, is used in statistics and control theory. Kalman filtering algorithms use a series of measurements observed over time, including statistical noise and other imprecisions, to generate estimates of unknown variables that are typically more accurate than estimates based on a single measurement. Furthermore, particle filters or sequential Monte Carlo (SMC) methods can be used for location estimation, i.e., a series of Monte Carlo algorithms used to solve filtering problems in signal processing and in Bayesian statistical estimation. The longer the chain of already connected objects, the more information can be used for accurate location estimation.

[0051] In a second embodiment of the present invention, a factor graph (a bipartite graph representing the factorization of a function) can alternatively be used for object tracking. Here, each object in an image is connected to each object in a subsequent image. Each connection is weighted according to a predetermined criterion, and then a complete connection across the two frames and across all objects is derived. The weighting rules can be formulated based on heuristics or behavioral models.

[0052] As soon as correlation of objects is carried out in successive frames, additional semantic information can be derived based on this combination, such as trajectory (i.e. the path taken by a mass point in its movement through space as a function of time), velocity (as a function of time), location, occlusion (objects that are hidden in time by other objects) and discontinuation (at the edges of the area detected by the sensor), as well as information on identification errors (objects that are present on the preceding and subsequent images and therefore may result in intermediate identification failures) and can be used as input for subsequent automation steps.

[0053] In the fourth step S4 (removing false positive results), incorrectly identified objects are removed. Multiple rules or criteria can be used and / or combined to remove false positive objects.

[0054] For example, ground height can be taken into account, in which case objects identified as partially below the road surface and / or objects that appear to be floating above the road surface are discarded as implausible. Alternatively or complementary, vehicle movement or trajectory observations can be performed, in which one or more quantities, such as acceleration or change in direction, can be calculated and compared to set limits. That is, accelerations outside the normal range for a given vehicle type or outside the physical possibilities limited by the tire's grip on the road surface are considered implausible. Duration can also be considered, in which case objects that are present only within a single frame or for a few seconds and do not appear in adjacent frames are deemed implausible. Here, it is advantageous to also take into account possible obscuration caused by other objects to avoid errors in the duration calculation. Furthermore, complementary considerations can be made to discard as implausible only objects that are identified by the neural network with low confidence.

[0055] The identification of false positive results is strongly related to semantic information derived from temporal correlations. In the real world, an object should not be present for a period of, say, a fraction of a second. By discarding invalid objects, the annotation project can achieve contractually agreed accuracy and re-identification values.

[0056] In a fifth step S5 (optimizing object size by regression), each object is regressed with its full position and size via a neural network. According to one embodiment of the present invention, neighboring images can also be taken into account when regressing the box size. Alternatively or complementary (as shown below as step S7), it is also conceivable to adjust the box sizes of identified or correlated objects to one another in a further post-processing step.

[0057] In the sixth step S6 (filling gaps), gaps are filled by interpolation based on the temporal context. Additional semantic information can be derived from the correlation of objects in consecutive frames, allowing consistent trajectories for traffic users to be obtained even in cases where foreground objects temporarily obscure other objects and / or in cases of clipping, where an object falls outside the area detected by the sensor, or in cases of identification errors, where an object is present in both a previous and a subsequent image and therefore intermediate identification may fail. Further semantic information can also be derived from the temporal correlation and used as input for subsequent automated or post-processing steps. Figure 6 shows a schematic diagram of a series of frames in which correlated objects are selected for further evaluation.

[0058] In the seventh step S7 (size unification), the size is unified across a chain of consecutive images and / or a chain of LiDAR point clouds. This is particularly important for achieving realistic sizes, which make objects appear better and feel more natural to a human observer. The unification of sizes of correlated objects in consecutive images and / or point clouds is preferably performed or not depending on the object class. Size unification can be performed especially for vehicles, since automobiles usually maintain their size across multiple frames. It is advantageous not to unify sizes for pedestrian object classes, since pedestrians may have different assumed sizes depending on their current body pose.

[0059] In the eighth step S8 (smoothing trajectories based on a physical model), the trajectories of dynamic objects or traffic users in the scene are smoothed according to a physical model based on them. For example, a conventional lane model can be used to describe the driving behavior of a car.

[0060] In a ninth step S9 (Calculate Attributes), an identification of attributes is performed, which may include various further steps.

[0061] While the relevant attributes may vary depending on the label specifications in effect for the project at the time, various object classes often have different attributes. For example, a car may have turn signals to indicate a planned change in direction and brake lights to indicate a braking maneuver. In the ninth round or step, project attribute identification can preferably be based on sensor fusion. Depending on the need, values ​​of turn signals, brake lights, emergency lights, or other descriptive characteristics may be of interest. To be able to correlate object instances in the 2D camera shots with object instances of the same object in the 3D LiDAR scan, common information in both datasets must be identified. The two datasets are then fused together using these common data points. To identify attributes, various 2D-based object identification networks can be run on the merged dataset. Specifically, attributes on the camera images are calculated and can be assigned to objects in the LiDAR point cloud.

[0062] The annotated sensor data can be used for various purposes, such as for example, driving scenario extraction or training of perception networks, for which annotated or augmented data is required as ground truth (data enrichment). Depending on the automation goal, both the order and the individual configuration of the above-mentioned steps can be adapted, whereby some of the above-mentioned steps can be optional, i.e., can be performed or omitted, and additional steps can be added as needed for further data enrichment.

[0063] For example, to extract a scenario, 1. Step of identifying an object (Step S1); 2. A step of tracking based on time correlation (step S3); 3. Identifying / excluding false positive results (step S4); 4. Regression step (step S5); 5. Step of unifying the sizes (Step S7); 6. Smoothing the trajectory (step S8); can be executed.

[0064] For example, for data selection, 1. identifying an object; 2. Tracking based on time correlation; 3. Identifying / excluding false positives; can be executed.

[0065] Alternatively, for example, for data augmentation, 1. Step of identifying an object (Step S1); 2. A step of tracking based on time correlation (step S3); 3. Identifying / excluding false positive results (step S4); 4. Position and size regression (see step S5); 5. Step of unifying the sizes (Step S7); 6. Position-only regression (see step S5); 7. Identifying attributes, e.g., turn signals, brake lights, and other descriptive characteristics (step S9); can be executed.

[0066] Preferably, the algorithms for performing the various steps are constructed so that only one common interface or API is required, making them interchangeable with each other and in a predetermined order, allowing the steps to be combined in different ways to achieve various desired results.

[0067] One embodiment of an automated system for carrying out the method of the present invention is shown in Figure 7. The automated system implements various complementary steps in dedicated components and is well suited for execution in a cloud computing environment.

[0068] In a first step of data preprocessing, raw sensor data, which may be unclassified, is received. The raw sensor data may include data from environmental sensors, such as a LiDAR point cloud and / or images from one or more cameras, as well as data from one or more vehicle sensors, such as a GPS receiver and / or wheel rotation speed, acceleration, or yaw rate sensors. The data may be normalized, e.g., split into individual frames and / or grouped into more packets to enable uniform processing. Integration of data recorded simultaneously by different sensors may also be performed.

[0069] In a second step of the automation engine, the packets of frames are processed in at least one automation component. The automation component is the central module of the annotation system workflow. The automation component may in particular be a neural network or another machine learning-based technique for supervised, semi-supervised or unsupervised learning from a dataset. The automation component may implement individual steps or individual sub-steps of the method according to the invention. In some steps, for example in the step of identifying attributes (S9), multiple automation components may be used, such as different neural networks for identifying the respective attributes.

[0070] In a third step of quality control, a random sample of frames is preferably selected and inspected by a human. In this case, the human quality inspector can be shown, for example, camera images and / or LiDAR point clouds with corresponding annotations (e.g., bounding boxes) and can be queried whether the bounding boxes are correct. Alternatively, if the neural network misses an object, the quality inspector can be presented with a user interface for adapting and / or adding boundary frames. The automated system receives the quality inspector's input. One possible embodiment of the user interface is shown in Figure 9.

[0071] If the quality check is successful, the annotated data can be exported. In a fourth step, customer inspection, the customer can inspect a random sample of the exported frames to ensure that the annotations correspond to the customer's specifications and desired annotation quality. If the customer rejects the annotations, the packets of annotated data are manually corrected in a correction step.

[0072] The fifth step of the correction involves manual annotation of the claimed packets. After a successful new quality control and customer inspection, the annotated frames can be used as training / validation or test datasets. These datasets are symbolically represented by a single cylinder and can be used for new or complementary training.

[0073] In the sixth step, "Flywheel," one or more neural networks or automation components are newly trained based on the training dataset. After successful testing with the corresponding test dataset, the improved neural networks can be extracted or used in an automation engine.

[0074] FIG. 8 shows an exemplary embodiment of a "flywheel" method for automatically training an attribute detector.

[0075] Flywheel includes techniques for efficiently storing training sets for each automated component, monitoring training set changes, and automatically triggering new training as soon as a predetermined or automatically calculated threshold for training set change is identified. Additionally, Flywheel includes techniques for automatically rolling out newly trained models across automated components.

[0076] Which attributes must be identified are described in the label specification of each annotation project. For an object of the passenger car class, for example, the status of the turn signals, the movement status and any existing obscuration are called attributes. Basically, an object can have several attributes listed for each object class.

[0077] Preferably, for each attribute, a neural network pre-trained on an existing dataset is used as an attribute detector. With each new dataset examined, at least one attribute detector is newly trained, which allows the attribute detectors to have improved performance over the course of an annotation project and reduces manual costs.

[0078] FIG. 9 shows one embodiment of a user interface for quality control.

[0079] On the left side, an excerpt from a LiDAR point cloud is shown, along with its bounding box and a circular indication of the ground. In the center, an excerpt from a camera image taken at the same time is shown, with the bounding box also drawn on the camera image. On the right side, a selection list is shown, allowing the user to select or enter various attributes of the object to be included in the bounding box. For example, here, the vehicle is stationary and the left / right blinkers are off. The control surface can also include a button (not shown). This button opens a further window for reporting quality issues. For example, here, the bounding box dimensions do not match because the corners of the vehicle are located outside the box. Such a control surface simplifies quality control and reduces the time required for this.

[0080] Our method exploits the temporal correlation between successive frames to compute plausible object identification and consistent size, thereby enabling annotation of even large amounts of data.

Claims

1. 1. A computer-implemented method for annotation of driving scenario sensor data, the method comprising: receiving sensor-raw data including at least one of a plurality of successive point clouds from a LiDAR sensor and a plurality of successive images taken by one or more cameras; using one or more neural networks to identify objects in each image and / or each point cloud; Correlating each object in successive images and / or successive point clouds; discarding false positive results based on a validity criterion based on the object's height above ground, the object's age and / or the neural network's confidence level; exporting the annotated sensor data of the driving scenario; A method comprising:

2. the sensor-raw data includes the point cloud divided into at least two regions, and a neural network of a first architecture is used in a first region and a neural network of a second architecture different from the first architecture is used in a second region to identify objects; The method of claim 1.

3. the first region includes a perimeter of the measurement vehicle, while the second region has a minimum distance to the measurement vehicle; Preferably, for the neural network for object identification, a CenterPoint-based architecture is used in the first domain and a PointRCNN-based architecture is used in the second domain. The method of claim 2.

4. the method including removing duplicates before correlating each of the objects; The step of removing duplicates is preferably based on an overlap criterion and / or a confidence level of the neural network.

4. The method according to any one of claims 1 to 3.

5. Removal of duplicates of identified objects in the images and / or point clouds is performed; two overlapping objects, i.e. a first object identified with a first confidence level and a second object identified with a second confidence level lower than the first confidence level, are checked to see if the overlap, in particular the Intersection-over-Union (IoU), exceeds a set threshold, and if so, the second object is discarded as a duplicate; The method of claim 4.

6. the step of correlating each object comprises combining objects in successive frames, i.e. in the image and / or in the point cloud, such that an object in a first frame is correlated with an object in a second frame if the objects belong to the same object class and the overlap, in particular the Intersection-over-Union (IoU), is above a set threshold; 6. The method according to any one of claims 1 to 5.

7. The step of correlating each object comprises: predicting the location of objects on subsequent images and / or subsequent point clouds using Gaussian process reduction or a Kalman filter; and / or Tracking objects in successive images using factor graphs, i.e., bipartite graphs that factorize probability distributions; Including, 7. The method according to any one of claims 1 to 6.

8. The method further includes the step of reviving and inserting the overlooked object; Three or more consecutive frames are observed, and if an object in a first frame is correlated with an object in a third frame but is not identified in an intervening frame, the object is inserted into a second frame.

8. The method according to claim 6 or 7.

9. the method further comprising optimizing by regression object size and / or object position in each image and / or each point cloud of camera data; 9. The method according to any one of claims 1 to 8.

10. The method further comprises standardizing the size of correlated objects in successive images and / or successive point clouds; Preferably, the step of unifying the sizes of the correlated objects is performed or canceled depending on the object class.

10. The method according to any one of claims 1 to 9.

11. the sensor-raw data includes the point cloud and simultaneously captured camera data, and the step of correlating the objects is performed taking into account the relative spatial positional relationship between the LiDAR sensor and the camera; 11. The method according to any one of claims 1 to 10.

12. applying at least one neural network to identify attributes of the identified objects, preferably at least one attribute, in particular the state of a vehicle's turn signal, to be calculated based on camera images and assigned to the objects in the point cloud; 12. The method according to any one of claims 1 to 11.

13. A computer readable data carrier comprising instructions for carrying out the method of any one of claims 1 to 12 when executed by a processor of a computer system.

14. 13. A computer system comprising a processor, a man-machine interface, and a non-volatile memory, the non-volatile memory containing instructions which, when executed by the processor, cause the computer system to perform the method of any one of claims 1 to 12.

Images