Method and device for quality assurance of semantic annotations for images

Abstract

Procedure (10), characterized by the following features: - Providing at least one crowd-generated annotation (20) for at least one object depicted in an image, comprising at least one traffic light (21), one vehicle (22), one truck (24) or one traffic sign (25), - Providing at least one reference annotation (30) for the object, wherein the reference annotation (30) includes an object such as a red traffic light (31), a silver vehicle (32), a white vehicle (33), a black SUV (34) or an aquaplaning warning sign (35), - Performing a comparison of the annotations (11) between the crowd annotation (20) and the reference annotation (30) using an IoU (12), - Filtering (13) crowd annotations (20) whose IoU (12) is below a specified threshold as bad (14) annotations.

Description

[0001] The present invention relates to a method for quality assurance of semantic annotations for images. The present invention further relates to a corresponding device, a corresponding computer program, and a corresponding storage medium. State of the art

[0002] In the field of autonomous driving system development, the need for extensive, precisely annotated datasets is very high. Such datasets are required to train algorithms for the recognition and interpretation of driving scenes. Current approaches often utilize professional annotators or specialized teams, which, however, is costly and time-consuming. One conceivable approach is the use of so-called gaming crowds, i.e., a large number of online game players who generate annotations during playful interactions. However, the problem here is that the quality of these annotations created by laypeople can vary greatly. Factors such as a lack of expertise, a lack of motivation for accuracy, or insufficient understanding of the annotation tasks often lead to the generated annotations not meeting the required standards.Despite the great potential of generating data on gaming crowds on a broad and rapid basis, the inadequate quality of the annotations remains a barrier to the reliable use of such data in autonomous driving. Disclosure of the invention

[0003] One problem is that when using gaming crowds to create annotations for driving scenes, the quality of the annotations is often inadequate and inconsistent. This makes it difficult to use such data to train image and object recognition algorithms for autonomous vehicles.

[0004] The described problem is solved according to the independent claims by providing a method for quality assurance of annotations created by gaming crowds, in which an intersection-over-union metric is used to compare crowd annotations with reference annotations. The reference annotations are created by experts or a verified "reference crowd." The method comprises the following steps: - Providing at least one crowd annotation generated by a gaming crowd for objects in images, including, for example, traffic lights, vehicles, trucks, or traffic signs. - Providing reference annotations where the objects have been annotated precisely and according to certain standards (e.g. red traffic light, silver vehicle, white vehicle, black SUV, aquaplaning warning sign). - Performing a comparison between crowd annotation and reference annotation using IoU to quantitatively determine the degree of agreement for both the semantic annotation and the bounding box. - Filtering out annotations whose IoU falls below a certain threshold as insufficient. These can then be manually edited or re-annotated by a gaming crowd. - Iteratively repeat the process to gradually improve the quality of the annotations generated by the gaming crowd. Gamification elements, training measures, or reward mechanisms can be used to increase the motivation and diligence of the annotators.

[0005] This approach has the advantage of achieving high and consistent data quality despite the use of non-professional annotators (gaming crowds). IoU-based quality comparison and the subsequent filtering of erroneous annotations can create a data pool that is sufficiently precise and comprehensive to reliably train and validate autonomous driving algorithms. Furthermore, iterative improvement processes can generate continuously optimized datasets, ensuring increasing robustness and safety of autonomous systems.

[0006] Further advantageous embodiments of the invention are specified in the dependent patent claims. For example, it is possible to integrate this quality assurance process directly into the development cycle of advanced driver assistance systems (ADAS) and fully autonomous driving systems. The precisely classified data supports sensor data analysis and scene interpretation in real time, allowing the vehicle to reliably detect pedestrians, vehicles, traffic lights, traffic signs, and road conditions and make appropriate decisions. Uncertainty quantification methods, the detection of driver interventions, and their transmission to a cloud platform can also be used to provide annotators with new data and downgrade vehicle functions if results are inadequate in certain areas.

[0007] The combination of precise training and validation data, iterative quality improvement, and direct integration into vehicle development contributes significantly to increasing road safety, optimizing autonomous driving functions, and accelerating the adoption of autonomous vehicles in everyday life. Short description of the drawings Fig. Figure 1 schematically shows an application of the method according to the invention. On the left, a crowd annotation (20) created by a gaming crowd is shown, in which various objects such as traffic lights (21), vehicles (22), a truck (24), and a traffic sign (25) were annotated. On the right, a reference annotation (30) is shown, which contains high-quality annotations of the same scene with precise marking of a red traffic light (31), a silver vehicle (32), a white vehicle (33), a black SUV (34), and an aquaplaning warning sign (35).

[0008] A comparison of the same scene (11) is performed between these two representations using an IoU metric (12). A filter unit (13) then selects poor annotations (14) that fall below a given IoU threshold and separates them from good annotations (15) that meet the quality requirements. Embodiments of the invention

[0009] An embodiment will now be explained with reference to the drawings.

[0010] In an exemplary embodiment, images captured during the development of an autonomous vehicle are annotated with the help of a gaming crowd. These annotators see, for example, images of a street intersection with traffic lights, multiple vehicles, and traffic signs. Motivated by an online game or an interactive platform, the players assign bounding boxes around relevant objects and classify them. The resulting crowd annotations are then compared with precise reference annotations created by experts or a specially qualified reference crowd. The IoU metric is used, which quantifies the degree of overlap between a crowd annotation and the corresponding reference annotation. If the IoU is below a defined threshold, the annotation is marked as insufficient.These inadequate annotations can be sent back to the gaming crowd for correction or manually edited. This gradually improves the quality of the annotations. Gamification mechanisms such as point systems, rankings, or rewards can be used to increase annotator motivation. It's also conceivable to train annotators in advance or provide them with context-sensitive hints to help them capture and name objects more precisely.

[0011] The method can be directly integrated into the development process of ADAS or fully autonomous driving systems. The data sets verified by IoU allow the algorithms to become familiar with a wide variety of driving situations, thereby increasing their robustness and reliability. Rare or critical scenarios (such as unusual road conditions, unusual driving behavior of other road users, or extreme weather conditions) can be better covered thanks to the large amount of annotated data. Furthermore, in an advantageous implementation, uncertainty quantification can be used on the vehicle side. As soon as uncertainties or deviations are detected during field operation, the corresponding data is transmitted to a computing unit in the cloud for re-annotation or refinement by the gaming crowd.If the results are unsatisfactory, the vehicle can deliberately reduce its driving functions in the relevant areas (degradation approach). This also allows a distinction to be made between safety-relevant driver interventions and comfort-based adjustments, allowing this information to be incorporated into the further development of driving functions.

[0012] Overall, the present invention thus offers an effective means of deriving high-quality, consistent, and robust annotations for autonomous driving systems from the rapid and extensive data collection by gaming crowds. Iterative refinement and flexible application enable continuous adaptation to new challenges and traffic situations. List of reference symbols 10 procedures 11 Comparison of annotations 12 Intersection-over-Union Metric (IoU) 13 filters 14 bad 15 good 20 Crowd Annotation 21 traffic lights 22 vehicles 24 trucks 25 traffic signs 30 Reference annotation 31 red traffic lights 32 silver vehicle 33 white vehicle 34 black SUVs 35 Aquaplaning warning sign

Claims

[1] Procedure (10), characterized by following features: - Providing at least one crowd annotation (20) generated by a gaming crowd for at least one object depicted in an image, comprising at least one traffic light system (21), a vehicle (22), a truck (24) or a traffic sign (25), - providing at least one reference annotation (30) for the object, wherein the reference annotation (30) comprises an object such as a red traffic light (31), a silver vehicle (32), a white vehicle (33), a black SUV (34) or an aquaplaning warning sign (35), - performing a comparison of the annotations (11) between the crowd annotation (20) and the reference annotation (30) using an IoU (12), - Filtering (13) crowd annotations (20) whose IoU (12) is below a given threshold as bad (14) annotations. [2] Method (10) according to claim 1, characterized byfollowing feature: - iteratively improving the filtered (13) bad (14) crowd annotations (20) by re-annotating them by the gaming crowd until the specified threshold for the loU (12) is exceeded and thus good (15) annotations are obtained. [3] Device, characterized by following feature: - the device is designed to carry out a method (10) according to one of claims 1 or 2. [4] Computer program which is designed to carry out all steps of a method (10) according to one of claims 1 or 2. [5] A machine-readable storage medium having a computer program according to claim 4 stored thereon.

Images