Automatic Traffic-Aware Semantic Annotation of Dynamic Objects

Abstract

1. A computer-implemented method for annotating sensor data recorded from around a vehicle at multiple points in time, comprising: determining objects within the sensor data; determining a current object area covered by the object and determining a future vehicle trajectory of the vehicle; and annotating the object as relevant if the object area is determined to obstruct the future vehicle trajectory. method.

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method and a sensor data annotation device for annotating sensor data recorded from the surroundings of a vehicle at multiple points in time.

[0002] The invention also relates to a computer readable recording medium storing program code, the program code comprising instructions for carrying out such a method. [Background technology]

[0003] In automated and autonomous driving, road safety, meaning avoiding collisions with other objects (e.g., vehicles or vulnerable road users), is crucial. Traffic situations in urban environments are highly complex and involve a large number of traffic participants that need to be taken into account by automated vehicles. Therefore, a sufficiently large training dataset is crucial for machine learning algorithms for automated and autonomous driving.

[0004] In the prior art, various annotation methods are known for building training datasets for supervised learning of autonomous driving. However, existing methods (e.g., those involving manual labeling of acquired sensor data) are too time-consuming and insufficient to provide the large amount of (labeled) training data required to train modern algorithms.

[0005] Improved methods are needed not only for acquiring training data, but also for labeling previously acquired training data. Summary of the Invention

[0006] To address one or more of the above-mentioned problems of the prior art, it is an object of the present invention to provide a computer-implemented method for annotating sensor data and for efficiently building a training dataset with annotated data in autonomous driving.

[0007] A first aspect of the invention is a computer-implemented method for annotating sensor data recorded from around a vehicle at multiple points in time, comprising: determining objects in the sensor data; determining a current object area covered by the object and determining a future vehicle trajectory of the vehicle; and annotating the object as relevant if the object area is determined to obstruct a future vehicle trajectory; A method is provided.

[0008] The method of the first aspect allows for annotating multiple objects in sensor data according to their relationships. By distinguishing between related and unrelated objects, a training dataset can be created with relationship-oriented annotations. The annotated training dataset can then be used to optimize supervised training for learning-based methods, for example, in automated driving (AD). Preferably, the method of the first aspect is performed after the sensor data has been stored, for example, by a fleet of sensor-equipped vehicles. The fleet is driven by human drivers, but the sensor data thus obtained, together with the annotations defined by the presented method, may be used as input for a training method for (semi-)automated driving.

[0009] The sensor data may include camera data of the vehicle containing a recording of the vehicle's surroundings. The sensor data may also include any data obtained from the vehicle's sensors to record the vehicle's surroundings and / or the state and / or movement of the vehicle. The sensor data may include a pre-collected database that allows offline computation of the annotation method.

[0010] The objects may be static or dynamic. In particular, dynamic objects may include other moving vehicles or other moving traffic participants such as bicyclists, motorcyclists, or pedestrians in the vehicle's surroundings. Static objects may include non-moving vehicles or other stationary traffic participants.

[0011] The term future vehicle trajectory herein refers to the trajectory of the vehicle at a time later than the current time. The current time may be one of multiple time points. Since the sensor data is preferably pre-recorded data, the "time point" may refer to the time point recorded, i.e., the time point when the record was made. In other words, the method embodiment of the first aspect may also be thought of as examining the sensor data recorded at a single point in time.

[0012] Objects, e.g., those important for collision avoidance, can be identified by annotating objects determined to be relevant in the sensor data, thereby creating a training dataset with data annotated according to relevance.

[0013] In a first implementation of the method according to the first aspect, an object area is determined to obstruct a future vehicle trajectory if the object area falls within a predetermined distance from the future vehicle trajectory. The inclusion of the predetermined distance allows for a safety buffer. This can be advantageous in situations where the distance between vehicles is small and a potential collision between the vehicle and an object is predicted. Annotating an object as relevant ultimately implies relevance for road safety. The predetermined distance may be predetermined based on various factors extracted from the recorded data, such as the vehicle speed, the object speed, the object distance, the object type, environmental factors, or other considerations. For example, a faster vehicle speed may correspond to a greater distance. This has the advantage that other objects are more easily recognized as relevant when the vehicle needs to maintain a greater distance from them at a higher speed.

[0014] In a further implementation of the method according to the first aspect, the sensor data comprises camera data. Preferably, the sensor data may be generated using LIDAR, RGBD, a stereo camera, or a fusion of these sensors to record dynamic objects around the vehicle.

[0015] Preferably, the method further comprises determining the trajectory of the vehicle based on collecting data on the vehicle's position and / or orientation over time using dedicated sensors, in particular GPS sensors. Optionally, determining the vehicle's trajectory may be based on collecting sensor data from the vehicle's surroundings and / or from sensors integrated into the vehicle, such as an IMU device and / or wheel tick sensors. The sensor data are post-processed to obtain the vehicle's position and / or orientation over time, which allows offline computation based on the sensor data. The vehicle's position may be an absolute position in real-world coordinates, for example as determined by a GPS sensor.

[0016] In a further implementation of the method according to the first aspect, determining the object area covered by the object comprises mapping from the sensor data to a 3D space, and the future vehicle trajectory is stored in the 3D space. The annotated data may be used for optimization of various ADAS and AD functions.

[0017] In a further implementation of the method according to the first aspect, the vehicle trajectory comprises positions of the vehicle at multiple points in time based on sensor data.

[0018] In a further implementation of the method according to the first aspect, annotating the object as relevant includes annotating the object as future relevant if the object area at a future time point after the current time point is determined to overlap with a future vehicle trajectory. For example, if another object is approaching the vehicle from behind, it may not be considered "currently relevant" because its current position does not overlap with the future vehicle trajectory (which is in front of the vehicle if the vehicle is moving forward). However, its future position (when it approaches or overtakes the vehicle) will overlap with the future vehicle trajectory and therefore may be "future relevant." Thus, "future relevant" objects may be particularly important (and they may be recognized, for example, by a rear-facing sensor on the vehicle).

[0019] By considering future points in time, foresight aspects, which are fundamental for driving procedures and have an impact on the driving behavior of the vehicle, are now taken into account. For example, as mentioned above, a vehicle approaching rapidly from behind can be recognized as "futurely relevant" and may even be considered as particularly important. Foresight aspects are crucial for the functionality of automated driving, since the annotation of sensor data according to relevance provides a training data set that separates unimportant objects from important ones.

[0020] In particular, the future time points considered during annotation of future relevant objects may be within a predetermined time from the current time point, for example, the predetermined time may be a constant value between 0 and 6 seconds, preferably between 0.5 and 3 seconds.

[0021] In a further implementation of the method according to the first aspect, determining the object area covered by the object comprises: determining the position and / or orientation of the object from the sensor data; and determining the dimensions of an object from sensor data, Or determining the type of object from the sensor data and inferring the dimensions of the object using the predetermined dimensions, where the predetermined dimensions are determined based on the determined type of object.

[0022] In addition to sensor data as obtained from one or more sensors of the vehicle, additional annotations such as the type of object (e.g., vehicle type) may be available. This information may help to more accurately determine the object area. The dataset may also include detailed information, such as specific dimensions of the object and further characteristics of the object. Alternatively, the type of object may be determined by methods based on the sensor data.

[0023] In a further implementation of the method according to the first aspect, the object area is a probabilistic area and / or the future vehicle trajectory is a probabilistic trajectory, and if the probability of intersection between the object area and the future vehicle trajectory is greater than a predetermined intersection probability threshold, an overlap is determined. Since various uncertain factors are involved that can affect the vehicle driving process and the movement behavior of objects around the vehicle, it is advantageous to consider probabilistic factors for both the vehicle trajectory and the object area in order to take into account future trends that may affect the vehicle behavior. The probabilistic trajectory may be affected by uncertainty about the position due to errors, for example, when acquiring a GPS signal at high speed.

[0024] In a further implementation of the method according to the first aspect, annotating the object as related includes annotating a plurality of objects as related, and the method further includes annotating one of the plurality of related objects as the lead related object if the lead related object has the smallest distance to the vehicle along the future vehicle trajectory. By further distinguishing the related objects (hereinafter also referred to as related objects), a hierarchy of relatedness may be created, and the most related object from among the related objects may be determined as the lead related. In particular, the lead related object may be the lead vehicle, because the lead vehicle typically travels on the future vehicle trajectory and is therefore always related.

[0025] Another aspect of the invention relates to a sensor data annotation device configured to perform the above method.

[0026] A further aspect of the invention refers to a computer-readable storage medium having stored thereon program code, the program code comprising instructions that, when executed by a processor, perform the method of the second aspect or one of multiple implementations of the second aspect. [Brief explanation of the drawings]

[0027] The accompanying drawings are intended to provide a further understanding of the present invention. In this regard, the accompanying drawings illustrate various steps and stages of a method and, in addition to the description, explain the inventive concepts. Many of the advantages mentioned will become apparent with reference to the drawings. Elements shown in the drawings are not necessarily shown to scale relative to each other. [Figure 1] FIG. 1 is a schematic diagram showing the positions of multiple vehicles and the object area of ​​multiple vehicles, one of the vehicles being identified as a lead vehicle. [Figure 2a] FIG. 2a is a schematic diagram showing the positions of multiple vehicles and their object areas at a current time in a scenario in which a first vehicle is attempting to overtake a second vehicle. [Figure 2b] Figure 2b is a schematic illustration of the scenario of Figure 2a, but considering the location of the vehicle at a future time. [Figure 3] FIG. 3 shows a similar scenario to that shown in FIGS. 1, 2a, 2b, but with a different shaped object area. [Figure 4] FIG. 4 shows a vehicle with sensors surrounding and sensing the vehicle during sensor data acquisition. Detailed Description of the Invention

[0028] Detailed Description of the Embodiments The above description is merely an implementation method of the present invention, and does not limit the scope of the present invention. Any modifications or replacements can be easily made by those skilled in the art. Therefore, the scope of the present invention should be subject to the scope of the attached claims.

[0029] 1 shows a vehicle 101, hereinafter also referred to as ego-vehicle, on a road 100 at a current time t. The ego-vehicle is equipped with sensors to record its surroundings, in particular dynamic objects 102-104 (which may be other vehicles or other dynamic traffic participants surrounding the ego-vehicle 101). The ego-vehicle 101 is moving along a trajectory 110, which includes a past vehicle trajectory 110a and a future vehicle trajectory 110b. The ego-vehicle 101 includes sensors that collect sensor data of the surroundings of the ego-vehicle 101.

[0030] The ego-vehicle is surrounded by second, third, and fourth vehicles 102, 103, and 104. In the preferred embodiment illustrated in FIG. 1, the object area of ​​the objects (here, vehicles) corresponds to the real area occupied by the vehicles, i.e., the vehicles illustrated in FIG. 1 correspond to the object area of ​​the vehicles. Initially, the object area of ​​the third vehicle 103 overlaps with the future vehicle trajectory 110b. Therefore, the third vehicle 103 is labeled as a related object. The fourth vehicle 104 also overlaps with the future vehicle trajectory 110b and therefore is labeled as a related object. The object area of ​​the second vehicle 102 does not overlap with the vehicle trajectories and therefore is not considered related.

[0031] The closest related object to ego vehicle 101 along future vehicle trajectory 110b is vehicle 103, and therefore it is considered the lead related object. From Figure 1, it may be further predicted that when the ego vehicle moves further forward in overtaking the third vehicle 103, its future trajectory will no longer overlap with vehicle 103. At that time, the fourth vehicle 104 becomes the new lead related object.

[0032] It will be understood that in practice, the object area may be defined to be larger than the object itself. For example, the scenario shown in Figure 2a illustrates an embodiment in which the object area is defined as a circular object area 201a, 202a, 203a around the center of the object 201, 202, 203. This may be particularly appropriate for detecting objects that are not directly on the future trajectory of the ego vehicle, but are traveling, for example, on an adjacent lane, and therefore may be of interest.

[0033] Figure 2a shows a scenario in which an ego-vehicle 201 is traveling in the center lane of a road 200 (note that lane markings are not required for the operation of the method). The ego-vehicle 201 is surrounded by second and third vehicles 202, 203 traveling in the same lane and adjacent lanes. The scenario shown in Figure 2a corresponds to an initial point in time, corresponding to the beginning of an overtaking maneuver by the ego-vehicle 201. The ego-vehicle 201 is traveling in the center lane, and its driver is attempting to overtake a second vehicle 202 traveling in front of the ego-vehicle 201 in the center lane. The trajectory 210 of the ego-vehicle 201 can be divided into a past vehicle trajectory 210a, which is the trajectory that the vehicle has traversed up to the current point in time, and a future vehicle trajectory 210b, which is the part of the trajectory that the vehicle will traverse after the current point in time.

[0034] The future trajectory 210b of the ego vehicle reflects the process of changing into the left lane and passing the second vehicle.

[0035] In the left lane is a third vehicle 203. At the current time, the future trajectory 210b overlaps the object area of ​​the second vehicle 202 and the object area of ​​the third vehicle 203. Since the second vehicle 202 is closer than the third vehicle 203, the second vehicle is the lead related vehicle.

[0036] Let us denote the point in time depicted in Figure 2a as time T. Note that in Figure 2a, vehicles 202 and 203 are displayed as relevant objects because their object areas intersect with the future vehicle trajectories.

[0037] Figure 2b shows the same road 200 and the same vehicle as Figure 2a, but at an earlier point in time. Let us denote this earlier point in time as time T-tau, where tau refers to a predetermined time before that point in time, preferably 0 to 3 seconds before. Given the earlier point in time depicted in Figure 2b, the ego-vehicle 201 appears at position 201' in its past trajectory.

[0038] At an earlier time T-tau, depicted in Figure 2b, the object area of ​​the second vehicle 202' does not intersect with the future trajectory 210b' because at this time the second vehicle has not yet moved into the center lane. Therefore, at T-tau, the second vehicle is not (yet) a relevant object. However, it will become a relevant object at the next time T depicted in Figure 2a, and therefore is marked as a future relevant object.

[0039] At time T-tau depicted in Figure 2b, the object area of ​​the third vehicle 203' intersects with the future trajectory given at T-tau and is therefore marked as a relevant object. In addition, it also becomes a relevant object at the next time T depicted in Figure 2a and is therefore marked as a future relevant object. In other words, the third vehicle 203' is a relevant object, a future relevant object and a lead relevant object.

[0040] In the above embodiment, the object area corresponds to the real area covered by the object. However, this is not required. In other embodiments, the object area may be, for example, a circular area around the center of the vehicle. In still other embodiments, the shape of the object area may not be circular but may be any geometric shape. FIG. 3 illustrates an implementation in which the object area is a rectangular shape disposed around the center of the object. The rectangular shape may vary in size depending on the type and dimensions of the object. The shape and size of the object area may be based on predetermined safety parameters, such as minimum headway. A circular shape may also be appropriate for pedestrians or cyclists who require a large buffer area for safety reasons. FIG. 3 also illustrates that vehicles 302-305 may have object areas 312-315 with rectangular shapes around them. These shapes may shift forward from the center of the object. This has the advantage that the object area is not limited to the real area covered by the object but can also reflect an area belonging to a space considered "dangerous." For example, if the ego-vehicle is within 5 meters ahead of another vehicle or within 1 meter to the side of another vehicle, it is considered dangerous. Similarly, vehicle trajectory 310 may include a safety margin 320 of width 322, and because safety margin 320 overlaps object area 314 of fourth vehicle 304, fourth vehicle 304 may be considered to obstruct future vehicle trajectories 310.

[0041] An "other vehicle" may be annotated as relevant if an area of ​​the vehicle, of a predetermined dimension based on the object type, e.g., vehicle type, is determined to interfere with a future vehicle trajectory. Additionally, annotating an object as relevant may include annotating an object as future relevant if the object area at a future time point after the current time point is determined to overlap with a future vehicle trajectory, where the future vehicle trajectory is preferably defined as including a location corresponding to a time point in the future relative to the current time point.

[0042] 4 illustrates various interactions between an ego-vehicle 401 and external entities. The vehicle 401 includes sensors that capture recordings from areas 410a and 410b on the left and right sides of the vehicle, and an area 412 in front of the vehicle. There may be additional rear-facing sensors that are not shown in FIG. 4.

[0043] The vehicle trajectory may be obtained either directly by receiving position and orientation data from GPS signals 420, or indirectly by post-processing sensor data from the surroundings of the ego-vehicle 401 so that the vehicle's position and / or orientation can be calculated over time from the sensor data. As visualized in Figure 4, the surroundings of the ego-vehicle include objects 402, 403, which may be annotated as relevant vehicles, future relevant vehicles, and / or lead vehicles after the ego-vehicle has completed acquiring sensor data and vehicle trajectory (preferably offline).

[0044] The method may be performed offline without any network connection, i.e., relying entirely on pre-recorded data.

[0045] Known methods typically process instantaneous (frame-by-frame) data or data collected in the recent past (the last few samples while maintaining temporal coupling (synchronization) between data recorded from surrounding dynamic objects and data related to the ego-vehicle's own motion). In this way, these methods can use historical data to make inferences about surrounding dynamic objects. They also typically use information from the static world (detected lane markings, road edges, etc.) to individually identify dynamic objects of greater importance, for example by focusing on objects in the ego-vehicle's lane.

[0046] The presented method allows adding semantic labels, in particular related, lead related, or future related, to the base annotations. These semantic labels can reflect the relevance of the objects for a given traffic situation.

[0047] The method is based on the separation of time-synchronous data (measurements of surrounding dynamic objects and the recorded real-world behavior of the ego-vehicle). Consider a scene constructed from basic annotations of surrounding dynamic objects at a given time sample and the integrated behavior of the ego-vehicle giving its overall path over the entire recording. By exploiting the basic properties of the ego-motion data, any past or future section of this path can be projected into a selected scene. Creating a merged scene gives the opportunity to infer past and future correspondences between dynamic actors in a given scene and selected path sections, independent of the original recording time.

[0048] In the presented method, we use this merged scene to determine the relevance of dynamic objects. Traffic-aware semantic annotations can be added to dynamic objects based on future route data. Note that this method only requires basic annotations of dynamic objects, not annotations of surrounding static objects such as lane markings, road edges, etc.

[0049] In a preferred embodiment, extracting lead object characteristics from a continuous time series of elementary annotations can be implemented as follows.

[0050] (1) By overlapping the future trajectory of the ego vehicle onto the current annotation of the object, it can be determined which objects will intersect with the future trajectory of the ego vehicle. From these objects, the object closest to the future vehicle trajectory can be considered as the lead object, and this information is marked in the annotation.

[0051] (2) At a given time t0, consider an object that is marked as a lead object. If this particular object is 0-x If it exists in the base annotation, it is set to t with a predicted range of x seconds. 0-x In this way, objects can be annotated as future lead objects, which adds time prediction functionality.

[0052] Besides the characteristics of the lead object, various other relevant aspects can be annotated in a similar way, for example a traffic situation on a dense highway where several objects (vehicles) are approaching an exit in a single file. When merging into such a file of vehicles, during annotation one can mark vehicles that have a safe gap to merge behind, thus obtaining semantic annotations that assist the merging procedure.

[0053] In multi-lane traffic situations, objects / vehicles can be marked to be taken into account when planning their lane change maneuvers or when providing lane change suggestions to the driver.

[0054] If we can identify relevant objects from the base annotations, the method can be extended to focus specifically on the detection and state estimation of relevant objects. By training a learning-based method on semantically annotated data, an attention-like solution can emerge, where the method can identify which parts of the input are more important; that is, relevant objects (or, as in the example above, lead objects) may have higher importance than others. Having such a focused method can have various benefits, such as reduced computational requirements (e.g., by excluding the detection and tracking of irrelevant objects), clearer and more structured output, and potentially more precise measurements of the method's accuracy.

Claims

1. 1. A computer-implemented method for annotating sensor data recorded from around a vehicle (101, 201, 301, 401) at multiple points in time, the method comprising: determining objects (102-104, 202, 203, 302-305, 403) in the sensor data; determining the current object area (201a, 202a, 203a, 312-315) covered by the object (102-104, 202, 203, 302-305, 403) and determining the future vehicle trajectory (110b, 210b) of the vehicle (101, 201, 301, 401); and annotating the object (102-104, 202, 203, 302-305, 403) as relevant if it is determined that the object area (201a, 202a, 203a, 312-315) obstructs the future vehicle trajectory (110b, 210b); method.

2. If the object area (201a, 202a, 203a, 312-315) is within a predetermined distance from the future vehicle trajectory (110b, 210b), the object area (201a, 202a, 203a, 312-315) is determined to obstruct the future vehicle trajectory (110b, 210b). The method of claim 1.

3. the sensor data includes camera data, LIDAR, and / or RADAR; 3. The method according to claim 1 or 2.

4. collecting position and / or orientation data of the vehicle (101, 201, 301, 401) over time using dedicated position sensors, in particular GPS sensors; or collecting sensor data from around the vehicle (101, 201, 301, 401) and post-processing the sensor data to obtain the position and / or orientation of the vehicle (101, 201, 301, 401) over time, based on determining a trajectory (110, 210) of the vehicle (101, 201, 301, 401); 4. The method according to any one of claims 1 to 3.

5. determining the object area (201a, 202a, 203a, 312-315) covered by the object (102-104, 202, 203, 302-305, 403) includes performing a mapping from the sensor data to a 3D space, and the future vehicle trajectory (110b, 210b) is stored in the 3D space; 5. The method according to any one of claims 1 to 4.

6. The vehicle trajectory includes positions of the vehicles (101, 201, 301, 401) at the multiple points in time based on the sensor data.

6. The method according to any one of claims 1 to 5.

7. The annotating of the objects (102-104, 202, 203, 302-305, 403) as related includes: annotating the objects (102-104, 202, 203, 302-305, 403) as being future relevant if it is determined that the object areas (201a, 202a, 203a, 312-315) at a future time point after the current time point overlap with the future vehicle trajectory (110b, 210b).

7. The method according to any one of claims 1 to 6.

8. the future time point is within a predetermined time from the current time point; The method of claim 7.

9. Determining the object areas (201a, 202a, 203a, 312-315) covered by the objects (102-104, 202, 203, 302-305, 403) includes: determining the position and / or orientation of the object (102-104, 202, 203, 302-305, 403) from the sensor data; and determining the dimensions of said object (102-104, 202, 203, 302-305, 403) from sensor data; or determining a type of the object (102-104, 202, 203, 302-305, 403) from the sensor data and inferring dimensions of the object (102-104, 202, 203, 302-305, 403) using predetermined dimensions, wherein the predetermined dimensions are determined based on the determined type of object (102-104, 202, 203, 302-305, 403); 9. The method according to any one of claims 1 to 8.

10. the object area (201a, 202a, 203a, 312-315) is a probabilistic area and / or the vehicle trajectory is a probabilistic trajectory, and an overlap is determined if the probability of intersection of the object area (201a, 202a, 203a, 312-315) with the vehicle trajectory is greater than a predetermined intersection probability threshold; 10. The method according to any one of claims 1 to 9.

11. annotating the objects (102-104, 202, 203, 302-305, 403) as related includes annotating a plurality of objects (102-104, 202, 203, 302-305, 403) as related, the method further including annotating one of the plurality of related objects as a lead-related object (103, 202, 304) if the lead-related object (103, 202, 304) has a minimum distance to the vehicle (101, 201, 301, 401) along the future vehicle trajectory (110b, 210b); 11. The method according to any one of claims 1 to 10.

12. The object area (201a, 202a, 203a, 312-315) is determined as a predetermined area around the center of the object (102-104, 202, 203, 302-305, 403); 12. The method according to any one of claims 1 to 11.

13. the size of the object area (201a, 202a, 203a, 312-315) depends on the speed of the vehicle (101, 201, 301, 401); The method of claim 12.

14. configured to carry out the method of any one of claims 1 to 13, Sensor data annotation device.

15. comprising a plurality of instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 13; A computer-readable recording medium.

Images