Method and apparatus for verifying annotations of objects

Abstract

A method of verifying annotations of objects is provided. Spatial data points acquired by a sensor and annotation data are received. The annotation data is associated with the acquired spatial data points and comprises an identification of each respective object. Via a processing unit, the annotations of the objects are verified by performing the following steps: determining a target range of at least one property of the objects, determining a respective value of the at least one property of each respective object from the acquired spatial data points and / or from the annotation data, and for each object, identifying the object as a false object if the respective value of the at least one property is outside the target range of the at least one property. The false object is selected for review of the annotations regarding the false object.

Description

Technical Field

[0001] This disclosure relates to a method and apparatus for verifying annotations of objects identified based on multiple spatial data points, wherein the annotations include the object’s corresponding classification and spatial location. Background Technology

[0002] Reliable perception of the vehicle's environment is essential for driver assistance systems and autonomous driving. To perceive the environment, lidar sensors are now crucial components, for example, for identifying objects within the vehicle environment. Lidar sensors typically provide what is known as a point cloud, which comprises three-dimensional spatial data points in a coordinate system associated with the vehicle or lidar sensor.

[0003] Besides directly analyzing the point clouds provided by lidar sensors, the data points from these point clouds often form the basis for many machine learning algorithms, for example, by providing real data for the learning process of neural networks. Data points from lidar sensors can also provide real data for other sensors, such as those in radar systems.

[0004] For example, in order to generate realistic data for machine learning, data points from LiDAR sensors must be annotated. That is, objects are identified within the LiDAR point cloud, either manually or through some pattern recognition system, and the identified objects are classified as, for example, passenger vehicles, pedestrians, etc., and the spatial location of the respective objects is described, for example, by surrounding the identified objects with a cube-like geometry.

[0005] To provide a reliable data foundation, such as for machine learning, annotated objects need to be audited. This means checking, for example, whether objects are correctly classified, whether the cubes surrounding the objects are correctly positioned, whether each object is tightly and completely covered by the cubes, and whether the cube orientations are correct (roll, pitch, and yaw angles relative to the coordinate system of the LiDAR sensor or vehicle). Typically, this auditing of annotated objects derived from LiDAR point clouds is performed manually. This manual auditing is a time-consuming task because the correctness of the cubes surrounding the objects must be verified from various perspectives in 3D space. Therefore, verifying objects derived from LiDAR point clouds usually takes significantly more time than, for example, auditing 2D bounding boxes.

[0006] Furthermore, the tedious process of manually reviewing objects or cubes associated with 3D LiDAR point clouds can be error-prone. For example, incorrectly annotated cubes may be easily overlooked, resulting in a lack of high-quality review because the reviewer's attention and motivation may be diminished during manual review.

[0007] Therefore, there is a need for a method and device that can validate object annotations and automatically identify objects that are incorrectly annotated. Summary of the Invention

[0008] This disclosure provides a computer-implemented method, a computer system, and a non-transitory computer-readable medium.

[0009] In one aspect, this disclosure relates to a computer-implemented method for verifying annotations of objects. According to the method, a plurality of spatial data points acquired by a sensor, the spatial data points being related to the external environment of the sensor, are received, and annotation data of objects is received, the annotation data being associated with the acquired spatial data points and including an identifier for each corresponding object. Annotations of objects are verified by a processing unit by performing the following steps: determining a target range for at least one attribute of the object; determining a corresponding value for the at least one attribute of each corresponding object from the acquired spatial data points and / or from the annotation data; and for each object, if the corresponding value of the at least one attribute is outside the target range of the at least one attribute, the object is identified as an erroneous object, and the erroneous object is selected for review regarding erroneous annotations.

[0010] The sensor can be, for example, a lidar system, and the spatial data points can be defined in a three-dimensional coordinate system whose origin is located at the sensor (e.g., at the lidar system). That is, the plurality of spatial data points can also be regarded as lidar point clouds.

[0011] The sensor can be installed in the vehicle. A three-dimensional coordinate system is defined for the spatial data points; therefore, it can be replaced by the vehicle coordinate system.

[0012] The annotation object includes providing an identifier or specification for the corresponding object. This identifier may include the object's classification, that is, associating the object with one of several predefined object categories, such as whether the object is a passenger car, truck, pedestrian, etc. Furthermore, the identifier may include the object's geometric information, such as information about the object's dimensions and spatial orientation.

[0013] Before this method can perform a verification step on the annotations of an object, it needs to receive spatial data points, such as LiDAR point clouds, and annotation data that provides the annotations to be verified or evaluated on the object. In other words, spatial data points and annotation data are prerequisites for executing this method.

[0014] The target range for at least one attribute of an object can be predefined based on the expected value of that attribute, such as the length or width of a car. Alternatively, as described below, the target range can be determined based on a respective probability distribution derived from a selected portion of data points and sample objects associated with that selected portion.

[0015] The objects selected for review regarding erroneous annotations are presented to the user as a list of objects to be reviewed. Therefore, this method automatically prioritizes objects located in the sensor environment and annotated in the first step (e.g., manually or via some pattern recognition system) (i.e., before executing this method) for the second step of annotation review. This method performs annotation verification on all objects by indicating, for example, a list of selected objects. Therefore, the reliability of the final annotations is improved, as it can be expected that even the most serious erroneous annotations on the objects will be detected.

[0016] On the other hand, only a portion or subset of the total number of objects (i.e., the selected objects) must have their annotations reviewed for errors. This significantly reduces the effort required for the entire annotation process compared to manual review of annotations for all identified objects.

[0017] According to one embodiment, the identifier of a corresponding object may include a category and a predefined geometry, wherein the predefined geometry is associated with a subset of spatial data points acquired for each corresponding object. In other words, since the predefined geometry is associated with those spatial data points believed to belong to the corresponding object, the identifier may include the category of the corresponding object and its assumed spatial location. This may include objects being tightly surrounded by a geometry, but completely covered, such that data points assumed to belong to each object are located inside the geometry.

[0018] The predefined geometry can be a cube. A cube can tightly enclose a subset of spatial data points belonging to each object, such that the subset is entirely within the cube. The size and orientation of each cube can be easily determined, which reduces the effort required to implement this method. Alternatively, the predefined geometry can be, for example, a cylinder or a sphere.

[0019] The target range can be determined by performing the following steps: selecting a portion of spatial data points that includes a corresponding subset of spatial data points for each of a plurality of sample objects; determining a corresponding value for at least one attribute of each sample object based on the assigned geometry and / or the corresponding subset of spatial data points; estimating at least one probability distribution of the attribute based on the statistical distribution of the attribute values ​​of the sample objects; and deriving the target range of the at least one attribute of the corresponding object from the at least one probability distribution.

[0020] The portion of spatial data points that can be selected and used to identify sample objects must be determined in such a way that a sufficient statistical basis can be used to estimate at least one probability distribution of the attribute. That is, the number of sample objects, and therefore the number of values ​​for at least one attribute, must be large enough to meet the statistical criteria for reliably deriving the probability distribution based on the values ​​of at least one attribute. For example, the portion of spatial data points can be selected by acquiring all data points from a continuous region within the instrument field of view of a lidar system, where the size of that region can be adjusted until a sufficient number of sample objects are included.

[0021] Therefore, this method can automatically determine the target range for at least one attribute. Since a portion of the data points is considered only for estimating the probability distribution and for deriving the target range, this embodiment may require less computational effort, although the target range is suitable for the statistical distribution of the object with respect to at least one attribute.

[0022] According to an embodiment, at least one attribute of an object (including a sample object) may include parameters of the object's spatial location. For example, these parameters may include the length, width, and height of a predefined geometry, as well as yaw, pitch, and roll angles, assigned to spatial data points of each object in the sensor coordinate system. Typically, parameters may include the spatial dimensions and orientation of the various geometries associated with the object.

[0023] If more than one attribute is identified for an object, an attribute or signature vector can be defined for each object. Each component of this vector can represent the value of the corresponding attribute.

[0024] By using parameters of an object's spatial location as the basis for estimating its probability distribution, object annotation verification can be performed directly, as these parameters can be easily derived from the geometry of the corresponding object, which is assigned to a subset of spatial data points. Therefore, one or more attributes of an object can be determined directly with minimal computational effort.

[0025] The at least one attribute can also be derived from the spatial distribution of the data points of the corresponding subset relative to the assigned geometry. Furthermore, the at least one attribute may include at least one statistical attribute of the data points of the corresponding subset.

[0026] Typically, this method is flexible in terms of the number and types of attributes considered for validation annotations. According to one embodiment, at least one parameter relating to the object's size and / or spatial orientation can be selected, such as vehicle length and / or vehicle width. However, the reliability of the validation annotations can be enhanced by considering other parameters relating to spatial location. For example, the six parameters mentioned above relating to the spatial location of the corresponding object—length, width, and height—as well as yaw, pitch, and roll angles, can be considered. As a supplement or replacement to parameters relating to the object's spatial location or assigned geometry, the relationship between a subset of spatial data points and the corresponding geometry can be considered.

[0027] For example, spatial data points belonging to a corresponding subset can be transformed into a coordinate system defined relative to the corresponding geometry. As an example of the statistical properties of spatial data points belonging to a corresponding subset, the average location or centroid of the data points and their second moment (covariance matrix) can be determined. In this way, it is possible to analyze, for example, whether the corresponding geometry closely matches the object to be annotated, by checking whether the majority of data points belonging to that subset "cluster" near the surface of the geometry. Conversely, if the average of the data points is close to the center of the cube, the corresponding object is most likely to be misannotated. Furthermore, misclassification of objects can be identified based on outliers in the elements of the covariance matrix.

[0028] According to another embodiment, the probability value of the at least one attribute can be determined based on at least one probability distribution, and if the probability value is less than a predetermined threshold, the corresponding value of the at least one attribute may be outside the target range. In other words, if an object's determined attribute value is assigned a low probability value, the object can be selected for review, and therefore the probability that the object's annotation is correct is quite low. Conversely, for objects not selected for review, the probability that their annotations are incorrect is quite low. Therefore, in this embodiment, annotation verification can be performed with high quality.

[0029] If more than one attribute is considered for object selection, each attribute can be assigned at least one probability distribution. In this case, if at least one of the assigned probability values ​​is less than a predetermined threshold, the object can be selected for review.

[0030] According to another embodiment, a predefined percentage share of the total number of objects can be used for auditing erroneous annotations, and the corresponding object with the lowest probability value can be iteratively selected for this audit until the number of selected objects equals the predefined percentage share of the total number of objects. In other words, objects with "least likely" annotations can be iteratively selected for auditing, where once a corresponding object is selected, it is not considered for the next iteration. The percentage share can be a fixed number such that, for example, 20% of the "least likely" annotations can be identified.

[0031] Therefore, in this embodiment, the effort and / or time spent selecting objects for review may be limited, and the time spent performing the method may generally be limited in this way. However, in this embodiment, it may not be possible to ensure that the most serious erroneous comments are identified.

[0032] Multiple spatial data points can be based on a LiDAR scan sequence over a predetermined time period. Furthermore, the at least one attribute can include the corresponding velocity of the object relative to the sensor, and this corresponding velocity can be determined based on the LiDAR scan sequence. In this embodiment, when determining the at least one attribute, object movement is also considered, i.e., it is replaced by or added to static attributes such as the size and orientation of the corresponding object. Considering object movement can improve the reliability of identifying and selecting incorrectly labeled objects. For example, pedestrian speeds are typically lower than passenger car speeds. Therefore, if the speed of an incorrectly labeled pedestrian is too high, the incorrectly labeled pedestrian can be easily identified, which can be reflected by this method with a low probability value of the incorrectly labeled pedestrian's speed.

[0033] According to another embodiment, the classification of objects may include associating each object with one of a plurality of object categories, and estimating at least one probability distribution of at least one attribute may include estimating a separate probability distribution for each of the plurality of object categories. Different object categories may include, for example, pedestrians, passenger vehicles, trucks, etc. For each such object category, their attributes may be expected to "cluster" within a specific range for the corresponding attribute. For example, the height value of a pedestrian may be expected to be in the range of, for example, 1.6m to 2.0m. Similarly, the length and width of a passenger vehicle may "cluster" within a certain range. Because of the separate estimation of the corresponding probability distribution for each object category, the estimation of the corresponding probability distribution can be simplified, which may also improve the reliability of the estimated probability distribution.

[0034] Each probability distribution can be based on a Gaussian mixture model. A Gaussian mixture model is a probabilistic model that assumes each data point is generated from a mixture of a finite number of Gaussian distributions with unknown parameters (i.e., the centers or means and standard deviations). Typically, it may not be desirable to adequately describe a probability distribution by a single Gaussian distribution that includes only one mean (e.g., for object categories like passenger cars). Instead, a multimodal distribution can be expected. For example, the statistical distribution of passenger car length can have more than one maximum because there are often different types of passenger cars (cars, sedans, vans, etc.) with similar lengths within different ranges. Therefore, a Gaussian mixture model can improve the reliability of the probability distribution.

[0035] A Gaussian mixture model can include multiple Gaussian distributions, where the center of each Gaussian distribution can be determined based on the median of the determined values ​​of at least one attribute of the corresponding object category, and the standard deviation can be determined based on the median of the absolute deviations of the determined values ​​of at least one attribute of the corresponding object category. Each median can be a robust estimate of the center of the respective Gaussian distribution. Since the determined values ​​of at least one attribute can be expected to have outliers, using the median prevents such outliers from interfering with the estimation of the individual probability distributions. A similarly robust estimate of the standard deviation can be the median of the absolute deviations (MAD).

[0036] According to another embodiment, potential annotation errors can be indicated for objects selected for review. For example, the value of at least one attribute can be indicated or evaluated (e.g., indicated or evaluated as too large or too small), which is expected to lead to erroneous annotations. In this embodiment, the method not only provides, for example, a list of objects to be reviewed with erroneous annotations, but also provides clear indications of potentially erroneous annotations. When reviewing selected objects manually or automatically, the cause of the erroneous annotations can be easily found by analyzing the indicated potential errors associated with the corresponding attributes. Therefore, the time required to review selected objects can be reduced by indicating hypothetical annotation errors, and the reliability of the review and final annotations can be improved.

[0037] As used herein, the terms processing apparatus, processing unit, and module may refer to: being part of, or including, an application-specific integrated circuit (ASIC); electronic circuitry; combinational logic circuitry; field-programmable gate arrays (FPGAs); a processor (shared, dedicated, or grouped) that executes code; other suitable components that provide the aforementioned functionality; or combinations of some or all of the foregoing, such as in a system-on-a-chip. The term module may include memory (shared, dedicated, or grouped) that stores code executed by the processor.

[0038] According to an embodiment, the sensor may include a lidar system. Because a lidar system is capable of providing a dense point cloud, this embodiment ensures that the number of available spatial data points is large enough to provide a sufficient data foundation for reliably performing object annotation verification.

[0039] In another aspect, this disclosure relates to a computer system configured to perform some or all of the steps of the computer-implemented methods described herein.

[0040] A computer system may include a processing unit, at least one memory unit, and at least one non-transitory data memory. The non-transitory data memory and / or memory unit may include computer programs for instructing the computer to perform some or all of the steps or aspects of the computer-implemented methods described herein.

[0041] In another aspect, this disclosure relates to a non-transitory computer-readable medium comprising instructions for performing several or all of the steps or aspects of the computer-implemented methods described herein. The computer-readable medium may be configured as: an optical medium, such as an optical disc (CD) or digital versatile disc (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid-state drive (SSD); a read-only memory (ROM); flash memory; and so on. Furthermore, the computer-readable medium may be configured as a data storage device accessible via a data connection such as an internet connection. The computer-readable medium may, for example, be an online database or cloud storage.

[0042] This disclosure also relates to a computer program for instructing a computer to perform some or all of the steps or aspects of the computer-implemented methods described herein. Attached Figure Description

[0043] This document describes exemplary embodiments and functions of the present disclosure in conjunction with the following schematically illustrated figures:

[0044] Figure 1 The apparatus and method steps for verifying objects according to this disclosure are described.

[0045] Figure 2 An example is depicted where an object is annotated by assigning cubes to data points.

[0046] Figure 3 Examples of statistical distributions and probability distributions used in the devices and methods according to this disclosure are depicted, and

[0047] Figure 4 Examples of output information provided by the devices and methods according to this disclosure are described. Detailed Implementation

[0048] Figure 1 A schematic overview of a computer system 11 is depicted, which is communicatively connected to a sensor 13 in the form of a lidar system, and the computer system includes a processing unit 15 configured to analyze data provided by the lidar system 13. The instrument field of view of the lidar system 13 is schematically represented by line 14.

[0049] A lidar system 13 is installed in a host vehicle (not shown) to provide the lidar system 13 with the ability to monitor the host vehicle's environment. The host vehicle's environment typically includes a number of objects 17. Passenger cars and pedestrians are shown as examples of objects 17. For driver assistance systems and / or autonomous driving of the host vehicle, reliable perception of the host vehicle's environment is important, i.e., particularly reliable perception of the position and movement of the respective objects 17.

[0050] As is known in the art, the lidar system 13 provides a plurality of spatial data points 19 covering the corresponding object 17. The spatial data points 19 are, for example, three-dimensional data points defined in a coordinate system with the origin located at the lidar system 13.

[0051] Spatial data points 19 are used to annotate objects 17 from the environment of the main vehicle. The annotations include identification of the corresponding object 17 regarding object category (i.e., classification of the corresponding object 17) and geometric information (i.e., the size and spatial orientation of the corresponding object 17).

[0052] exist Figure 2 An example of this annotation is shown in the image. Figure 2 A portion of the data points 19 acquired by the lidar system 13 in front of the main vehicle is depicted. Due to the high density of the spatial data points 19 (especially close to the object 17), the spatial data points 19 provided by the lidar system 13 can also be considered as lidar point clouds.

[0053] like Figure 2 As shown, spatial data point 19 includes a subset 31 of data points belonging to a specific object 17, namely... Figure 2 The example is a passenger vehicle. To annotate object 17 based on a lidar point cloud or multiple spatial data points 19, a cube 33 is assigned to a subset 31 of the spatial data points 19. Specifically, the subset 31 of spatial data points 19 is tightly enclosed by the cube 33, such that almost all data points 19 belonging to subset 31 are located within the cube 33. However, the cube 33 must tightly enclose subset 31 and therefore object 17, minimizing the distance between the surface of the cube 33 and the adjacent data points 19 of subset 31. Furthermore, the annotation of object 17 includes classifying the object by identifying its category. Figure 2 In the example, object 17 was identified or classified as a passenger vehicle.

[0054] In summary, the annotation objects include “enclosing” object 17 by cube 33, that is, assigning cube 33 to a subset 31 of spatial data points 19 belonging to the corresponding object 17 by defining the size and orientation of cube 33, and classifying object 17, that is, classifying the corresponding object 17 with multiple predefined object categories 41 (see Figure 4 The dimensions and orientation of cube 33 include its length, width, and height, as well as its roll angle, pitch angle, and yaw angle. In this embodiment, the annotation of object 17 based on lidar point clouds or spatial data points 19 is performed manually. Alternatively, an automatic pattern recognition system can be used for the annotation of object 17.

[0055] Whether object 17 is annotated manually or automatically, the annotation of object 17 may be incorrect, especially when a large number of objects 17 must be annotated. For example, cube 33 may not properly match a subset 31 of spatial data points 19 belonging to the corresponding object 17. Specifically, the size of cube 33 may be chosen during annotation such that cube 33 only surrounds a portion of the data points 19, or conversely, the size of cube 33 may be too large to tightly surround object 17. In addition, the orientation of cube 33 may also not properly match the actual object 17. Furthermore, object 17 may also be misclassified.

[0056] Therefore, before the annotated object 17 can be used by other systems in the vehicle (e.g., as real data that can be used by driver assistance systems or machine learning algorithms in autonomous driving), the annotated object 17 typically needs to be reviewed. If the annotations of all objects 17 identified based on LiDAR point clouds had to be manually reviewed, this would be a very time-consuming and tedious task. Furthermore, such a manual review process is prone to errors.

[0057] Therefore, this disclosure provides a method and computer system 11 for automatically verifying annotations of object 17. First, a portion of spatial data points 19 provided by the lidar system 13 is selected. The selected portion of data points 19 is associated with a sample object via a corresponding subset 31 belonging to the selected portion of data points 19. The sample object is associated with corresponding annotation data, which is described in detail below. The selected portion of data points 19 and the corresponding sample object including its annotation data are... Figure 1 The value is represented by 20 and received by the processing unit 15.

[0058] The annotation of the sample objects performed before the processing unit 15 executes the method includes identifying and classifying the sample objects within a selected portion of the spatial data points 19, and assigning the corresponding cube 33 (see...) Figure 2A subset 31 of spatial data points 19 belonging to the corresponding object 17 is assigned. Therefore, the annotation data for the sample objects includes classification and spatial information about the cube 33 surrounding each sample object, i.e., information about size and spatial location. Annotation is performed on a portion of the spatial data points 19 without reviewing the annotation results. Subsequently, the computer system 11 automatically determines the value of at least one attribute for each sample object based on the corresponding subset 31 of the spatial data points 19 and / or based on the cube 33 assigned to the corresponding sample object using this method.

[0059] The at least one attribute includes the dimensions of the corresponding object 17 or cube 33, namely length, width, and height, and / or one of the angles describing the orientation of the object 17 or cube 33, namely roll angle, pitch angle, and / or yaw angle. Furthermore, if a series of lidar scans can be obtained to monitor the timely development of the motion of the corresponding object 17, the velocity of the object 17 can be considered as one of the attributes of the corresponding object 17.

[0060] If we consider n attributes to validate the annotation, the values ​​of these attributes are represented by an n-dimensional vector, also known as the "signature" vector of the sample objects. According to this method, for each attribute, i.e., for each component of the n-dimensional signature vector, a statistical distribution of all sample objects belonging to a certain object category is generated.35 (see...) Figure 3 ).

[0061] According to the method of this disclosure, the number and types of attributes of the object 17 considered for verification annotation are flexible with respect to the computer system 11. Since the lidar system 13 is provided as a sensor for the computer system 11, additional attributes besides the size, orientation, and velocity of the object 17 can also be considered. These additional attributes relate to a subset 31 of spatial data points 19 and a corresponding cube 33 for enclosing the subset 31 (see...). Figure 2 ) relationship.

[0062] For these additional attributes, the spatial data points 19 belonging to each subset 31 are transformed into a coordinate system defined relative to the respective cubes 33 belonging to that subset 31. For example, a corner of cube 33 is used as the origin of this coordinate system. For each cube 33, statistical properties of the spatial data points 19 belonging to the corresponding subset 31 can be derived, such as the mean position or centroid of the data points 19 and their second moments (covariance matrix). In this way, the spatial distribution of the data points 19 within cube 33 can be described and analyzed. Using these additional attributes, it can be analyzed, for example, whether the corresponding cube 33 closely matches the object 17 to be annotated, by checking whether the majority of data points 19 belonging to subset 31 are "clustered" near the surface of cube 33. Conversely, if the mean of the data points 19 is close to the center of the cube, the corresponding object 17 is likely to be mis-annotated. Furthermore, misclassification of object 17 can be identified based on outliers in the elements of the covariance matrix.

[0063] for Figure 2 As illustrated in the example, a human observer would likely be able to identify the outline of a car from a subset 31 of data points 19. Similarly, the method according to this disclosure can evaluate the classification of object 17 based on the statistical properties of data points 19 within subset 31, if these data points 19 are described in a coordinate system defined relative to cube 33. For example, trees and cars can be distinguished based on the corresponding statistical properties of data points within the respective cubes, such as based on the mean and / or based on the elements of the covariance matrix.

[0064] exist Figure 3 A and Figure 3 B shows two examples of statistical distributions 35 related to the attributes of object 17. For the object category "passenger vehicle", two attributes have been selected: the length of the vehicle in meters (m). Figure 3 A) and the width of the car in meters ( Figure 3 B) To verify the annotation. The relative density (proportional to the relative frequency) on the sample object is depicted as bars of the corresponding statistical distribution 35. That is, for this example, the corresponding signature vector has two components, namely the car length and the car width. For these two attributes, their respective probability distributions 37 are derived based on the statistical distribution 35. In detail, a Gaussian mixture model is applied to describe the probability distribution. The Gaussian mixture model consists of a finite number of Gaussian distributions with unknown parameters determined based on the statistical distribution 35.

[0065] Gaussian mixture models can provide multimodal probability distributions. Figure 3B illustrates a prominent example of this multimodal probability distribution. Specifically, the statistical distribution 35 of the car width clearly has at least two maxima, one at approximately 1.8 m and the other at approximately 2.0 m. Therefore, using a single Gaussian distribution to approximate the statistical distribution 35 would oversimplify the actual distributions of the individual attributes and potentially deliver erroneous results. Thus, a Gaussian mixture model incorporating more than one Gaussian distribution provides a better approximation of the actual distributions 35 of the car length and width.

[0066] To determine the cluster centers of statistical distributions 35 that will be used as the respective means or centers of one of the Gaussian distributions within the Gaussian mixture model, the k-means algorithm, known in the art, is employed. Specifically, by applying the median, i.e., the k-median algorithm, to each statistical distribution 35, an appropriate number of centers and their locations, along with their corresponding covariance matrices, are determined. Using the median provides robust estimates of the centers and standard deviations of the individual Gaussian distributions within the Gaussian mixture model, since there is no review of the annotations of the sample objects, and therefore it is expected that they will include outliers within the distributions 35 of their properties.

[0067] via processing unit 15 (see Figure 1 Estimate each attribute (e.g., such as...) Figure 3 A and Figure 3 After the corresponding probability distribution 37 of the car length and car width shown in Figure B, all available spatial data points 19 (i.e., the entire lidar point cloud) provided by the lidar system 13, along with the corresponding annotation data, are received by the processing unit 15. The overall data points 19 and the annotation data for all objects 17 are then processed. Figure 1 The annotation of all data points 19 is performed in the same manner as the annotation of the selected portions of data points 19 associated with the sample objects described above. That is, objects 17 are identified and classified for all spatial data points 19, and cubes 33 are assigned to the corresponding subsets 31 of spatial data points 19 for each identified object 17.

[0068] After determining the probability distribution 37, the processing unit 15 determines the value of each relevant attribute for each object. Figure 3 The example shown is for the corresponding object category 41 (see Figure 4 The length and width of all objects 17 within the range are determined. Based on the probability distribution 37 of the corresponding attributes (e.g., car length and car width), the corresponding target range 39 is derived (see...). Figure 3 A and Figure 3 B) or the range of acceptable values ​​39. In detail, the range of acceptable values ​​39 includes the values ​​of the corresponding attributes whose corresponding probability values ​​provided by the corresponding probability distribution 37 are greater than a predetermined threshold.

[0069] The probability distribution 37, previously determined for the corresponding attributes based on the statistical distribution 35 of the sample objects' attributes, is used to determine the probability value of the corresponding attribute for each object (i.e., other objects 17 different from the sample objects). For example, if a certain length and a certain width of a car are determined, the corresponding probability values ​​for that length and width can be directly taken from... Figure 3 A and Figure 3 The corresponding probability distribution shown in B is 37.

[0070] Based on the corresponding attribute value (e.g., as shown in the image) Figure 3 The probability value determined by the values ​​of car length and car width shown is used to select object 25 for review of the erroneous annotation (see...). Figure 1 and Figure 4 That is, if the probability value assigned to one of the attributes is less than, for example, a predetermined threshold, then the corresponding object 25 is assumed to have an incorrect annotation and is therefore selected for review. In a more sophisticated way, the Local Outlier Factor (LOF), a non-parametric, density-based algorithm, can be used to find outliers. For example... Figure 3 As shown, a local outlier factor has been applied to distribution 35. As a result, objects 17 with a car length or width within the corresponding acceptable value range 39 are assumed to be correctly annotated because their car length and width, represented by small bars, fall within the corresponding range 39. Conversely, all objects whose car length and width are within range 40, i.e., outside the acceptable value range 39, are identified as outliers by the local outlier factor. The corresponding attribute (car length and / or car width) has a low probability value specified by the corresponding probability distribution 37.

[0071] This method provides output information 21 (see Figure 1 The output information 21 includes a list 23 of selected objects 25 that are assumed to require review of error comments. Finally, the review of the selected objects 25 is performed at 27.

[0072] Figure 4 A list 23 of objects 25 selected for review of error comments is described in more detail. This is handled by processing unit 15 (see...). Figure 1 List 23 is provided as output information 21.

[0073] For each selected object 25, list 23 includes the object category 41, i.e., whether object 25 is a car, pedestrian, truck, etc. List 23 also provides a unique identifier 43 and spatial location information 45 for each object 25. In this example, the specific number of frames in which object 25 is to be found is provided.

[0074] In addition, information regarding potential annotation errors 47 is provided. Specifically, it indicates which attribute of the corresponding object 25 is considered incorrectly annotated. In this example, the relevant attributes of the selected object 25 are the width, height, and speed of the car, pedestrian, and truck, respectively. Because the width of the car is considered too small, the height of the pedestrian is considered too high, and the speed of the truck is considered too high, the object category 41 of the selected object 25 is also assumed to be problematic and must be reviewed in the final audit 27 (see [link to audit]). Figure 1 The review will be conducted in [the relevant area]. Although... Figure 3 The corresponding probability distribution 37 is explicitly described only for the length and width of the car; however, it should be understood that similar probability distributions 37 can also be derived for speed and other attributes of the sample object. For speed, for example, a series of lidar scans can be used to determine the changes in the spatial position of data point 19.

[0075] In the final review 27, only the selected objects 25 provided by list 23 need to be checked regarding their annotations, rather than the total number of objects 17 identified separately in the environments of the LiDAR system 13 and the main vehicle. Therefore, the time spent reviewing annotations is significantly reduced compared to a manual review of all objects 17. Furthermore, serious annotation errors can be more easily identified and corrected. Such serious annotation errors are also less likely to be overlooked compared to a manual review of all objects 17. Therefore, the quality of the annotations for the identified objects 17 is improved. In addition, the potential annotation errors 47 for each selected object 25 clearly indicate which attributes of the object 25 are considered incorrect due to the annotations. This also helps in identifying and correcting the corresponding annotation errors. The indication of potential annotation errors 47 further reduces the time spent correcting these errors, and thus additionally improves the quality of the final annotation results.

[0076] List of reference numerals

[0077] 11 Computer Systems

[0078] 13 Sensors and LiDAR Systems

[0079] 14 lines

[0080] 15 processing units

[0081] 17 Objects

[0082] 19 spatial data points

[0083] 20 Selected data points and sample objects

[0084] 21 Output Information

[0085] 22 Data points and annotation data for all objects

[0086] 23. List of selected objects to be reviewed

[0087] 25. Incorrect or selected object

[0088] 27 Final Review

[0089] 31 Data Point Subset

[0090] 33 cubes

[0091] 35. Statistical distribution of object properties

[0092] 37 Probability Distribution

[0093] 39. Target range or acceptable range

[0094] 40. Range of outliers

[0095] 41 Object Categories

[0096] 43 Unique object identifier

[0097] 45 Spatial Location Information

[0098] 47 Potential commenting errors

Claims

1. A computer-implemented method for verifying annotations on an object (17), the method comprising: Receive multiple spatial data points (19) acquired by sensor (13), wherein the spatial data points (19) are related to the external environment of sensor (13). Receive annotation data for objects (17) associated with the acquired spatial data points (19), the annotation data including an identifier for each corresponding object (17), and The annotation of the object (17) is verified by the processing unit (15) by performing the following steps: Determine the target range (39) of at least one attribute of the object (17), wherein the at least one attribute includes a parameter of the spatial position of the object (17) and / or the corresponding velocity of the object (17) relative to the sensor (13). The corresponding value of at least one attribute of each corresponding object (17) is determined from the acquired spatial data points (19) and / or from the annotation data, and For each object (17), if the corresponding value of the at least one attribute is outside the target range (39) of the at least one attribute, then the object (17) is identified as an error object (25), and the error object (25) is selected for review (27) of the error annotation. The target range (39) is determined by performing the following steps: Select a portion of the spatial data points (19) that includes a corresponding subset (31) of the spatial data points (19) for each of the multiple sample objects. The corresponding value of at least one attribute of each sample object is determined based on the predefined geometry (33) and / or the corresponding subset (31) of the spatial data points (19) assigned to the sample object. Estimate at least one probability distribution of the attribute based on the statistical distribution of the attribute values ​​of the sample object (37), and The target range (39) of the at least one attribute of the corresponding object (17) is derived from the at least one probability distribution (37).

2. The method according to claim 1, wherein, The identification of the corresponding object (17) includes classification and predefined geometry (33), which is associated with a subset (31) of spatial data points (19) acquired for each corresponding object (17).

3. The method according to claim 2, wherein, The predefined geometry (33) is a cube.

4. The method according to claim 1, wherein, A probability value is determined for the value of the at least one attribute based on at least one probability distribution (37), and If the probability value is less than a predetermined threshold, the corresponding value of the at least one attribute is outside the target range (39).

5. The method according to any one of claims 1 to 4, wherein, The percentage share of the total number of predefined objects (17) for the audit (27) regarding the error comments, and For the audit (27), the corresponding object (25) with the lowest probability value is selected iteratively until the number of selected objects (25) is equal to a predefined percentage share of the total number of objects (17).

6. The method according to claim 1, wherein, The classification of the corresponding object (17) includes associating the corresponding object with one of a plurality of object categories (41), and Estimating the at least one probability distribution (37) of the at least one attribute includes estimating the individual probability distribution (37) of each of the plurality of object categories (41).

7. The method according to claim 6, wherein, Each probability distribution (37) is based on a Gaussian mixture model.

8. The method according to claim 7, wherein, The Gaussian mixture model includes multiple Gaussian distributions, and For each Gaussian distribution: The center is determined based on the median value of at least one attribute of the corresponding object category (41), and The standard deviation is determined based on the median of the absolute deviations of the determined values ​​of at least one attribute of the corresponding object category (41).

9. The method according to claim 2, wherein, The at least one attribute is derived from the spatial distribution of the data points (19) of the corresponding subset (31) relative to the assigned geometry (33).

10. The method according to claim 2, wherein, The at least one attribute includes at least one statistical attribute of the data point (19) of the corresponding subset (31).

11. The method according to claim 1, wherein, The plurality of spatial data points (19) are based on a lidar scanning sequence over a predetermined time period, wherein the corresponding velocity is determined based on the lidar scanning sequence.

12. The method according to claim 1, wherein, For each object (25) selected to undergo the audit (27) regarding erroneous annotations, a potential annotation error is indicated (47).

13. A computer system (11) configured to perform a computer-implemented method according to any one of claims 1 to 12.

14. A non-transitory computer-readable medium comprising instructions for performing a computer-implemented method according to any one of claims 1 to 12.

Images