Automatic traffic aware semantic annotation of dynamic objects

Abstract

A computer-implemented method for annotating sensor data recorded from the surroundings of a vehicle at a plurality of points in time, the method comprising:-determining an object in the sensor data; -determining an object area covered by the object at the current point in time and determining a future vehicle trajectory of the vehicle; and marking the object as relevant if it is determined that the area of the object interferes with the future vehicle trajectory.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method and a sensor data annotation apparatus for annotating sensor data recorded from a vehicle's surroundings at multiple points in time.

[0002] The invention also relates to a computer-readable storage medium storing a program code comprising instructions for implementing such a method. Background Art

[0003] In the context of automated and autonomous driving, traffic safety in the sense of avoiding collisions with other objects (e.g., vehicles or vulnerable road users) is of paramount importance. Traffic situations in urban environments are extremely complex and involve a large number of traffic participants that need to be taken into account by automated vehicles. Therefore, sufficiently large training datasets are of great importance for machine learning algorithms for automated and autonomous driving.

[0004] In the prior art, various annotation methods are known for constructing training datasets for supervised learning of autonomous driving. However, existing methods (which involve, for example, manual labeling of acquired sensor data) are too time-consuming and insufficient to provide the large amount of (labeled) training data required to train modern algorithms.

[0005] There is a need for improved methods for not only acquiring training data but also labeling previously acquired training data. Summary of the invention

[0006] The object of the present invention is to provide a computer-implemented method for annotating sensor data in the context of autonomous driving and constructing a training dataset using efficiently annotated data to overcome one or more of the above-mentioned problems of the prior art.

[0007] A first aspect of the invention provides a computer-implemented method for annotating sensor data recorded from around a vehicle at a plurality of points in time, the method comprising:

[0008] - identify objects in sensor data,

[0009] - determining the object area covered by the object at the current point in time and determining the future vehicle trajectory of the vehicle, and

[0010] - If the object region is determined to interfere with the future vehicle trajectory, the object is marked as relevant.

[0011] The method of the first aspect can annotate objects in sensor data according to their relevance. By distinguishing between relevant and irrelevant objects, a training dataset with relevance-oriented annotations can be created. The annotated training dataset can then be used, for example, to optimize supervised training for learning-based methods in the context of autonomous driving (AD). Preferably, the method of the first aspect is implemented after sensor data is recorded, for example by a fleet of vehicles equipped with sensors. The fleet can be driven by human drivers, but the sensor data acquired thereby and the annotations provided by the proposed method can be used as input to a training method for (semi-) autonomous driving.

[0012] The sensor data may include camera data of the vehicle including a recording of the surroundings of the vehicle. The sensor data may also include any data obtained from the vehicle's sensors that records the surroundings of the vehicle and / or the state and / or movement of the vehicle. The sensor data may include a pre-collected database that enables offline calculation of the annotation method.

[0013] Objects can be static or dynamic objects. In particular, dynamic objects can include other moving vehicles or other moving traffic participants, such as cyclists, motorcyclists or pedestrians around the vehicle. Static objects can include non-moving vehicles or other stationary traffic participants.

[0014] The term "future vehicle trajectory" herein refers to the trajectory of the vehicle at a point in time later than the current point in time. The current point in time may be one of a plurality of points in time. Since the sensor data is preferably pre-recorded data, the "point in time" may refer to the recording point in time, i.e., the point in time when the recording is created. That is, the implementation of the method of the first aspect may be considered to be through the recorded sensor data point by point in time.

[0015] By labeling objects determined in the sensor data as relevant, objects that are important for avoiding collisions, for example, can be identified. This creates a training data set in which the data is labeled according to relevance.

[0016] In a first implementation of the method according to the first aspect, an object area is determined to interfere with the future vehicle trajectory if the object area is within a predetermined distance of the future vehicle trajectory. By including the predetermined distance, a safety buffer zone is taken into account. This may be advantageous in situations where the distance between vehicles is small, and a potential collision between a vehicle and an object can be expected. Because marking an object as relevant ultimately implies relevance regarding traffic safety. The predetermined distance can be predetermined based on various factors extracted from the recorded data, such as ego vehicle speed, object speed, object distance, object type, environmental factors or other considerations. For example, a higher speed of the vehicle may correspond to a larger distance. This has the advantage that at higher speeds, when the vehicle should maintain a larger distance from other objects, other objects are more easily identified as relevant.

[0017] In another implementation of the method according to the first aspect, the sensor data comprises camera data. Preferably, the sensor data may be generated using LIDAR, RGBD, stereo camera or a fusion of these sensors to record dynamic objects around the vehicle.

[0018] Preferably, the method further comprises a step of determining the trajectory of the vehicle based on collecting vehicle positioning and / or orientation data over time using dedicated sensors, in particular GPS sensors. Optionally, the step of determining the trajectory of the vehicle can be based on collecting sensor data from the vehicle's surroundings and / or from sensors integrated into the vehicle, such as IMU devices and / or wheel beat sensors. The sensor data is post-processed to obtain the positioning and / or orientation of the vehicle over time. This enables offline calculations based on the sensor data. The vehicle positioning can be an absolute positioning in real-world coordinates, for example as determined by a GPS sensor.

[0019] In another implementation of the method according to the first aspect, determining the object area covered by the object comprises performing a mapping from the sensor data to a 3D space, wherein the future vehicle trajectory is stored in the 3D space.The annotated data may be used to optimize different ADAS and AD functionalities.

[0020] In another implementation of the method according to the first aspect, the vehicle trajectory comprises a location of the vehicle at a plurality of time points based on sensor data.

[0021] In another implementation of the method according to the first aspect, marking an object as relevant includes: if it is determined that the area of ​​the object at a future time point after the current time point overlaps with the future vehicle trajectory, marking the object as future relevant. For example, if another object approaches the vehicle from behind, it may not be considered "currently relevant" because its current position will not overlap with the future vehicle trajectory (if the vehicle is moving forward, the trajectory is in front of the vehicle). However, it may be "future relevant" because its future position (when it approaches or overtakes the vehicle) will overlap with the future vehicle trajectory. Therefore, "future relevant" objects may be particularly important (and they can be identified, for example, by the vehicle's rear-facing sensors).

[0022] By considering future time points, the present invention takes into account foresight aspects that are important for the driving maneuver and have an impact on the driving behavior of the vehicle. For example, as mentioned above, a vehicle approaching quickly from behind can be recognized as "future relevant" and can even be considered to be particularly important. The foresight aspect is crucial for the automated driving function, since the annotation of the sensor data according to relevance provides a training data set that separates unimportant objects from important objects.

[0023] In particular, the future time point considered during the marking of future related objects may be within a predetermined time distance to the current time point. For example, the predetermined time distance may be a constant predetermined time distance between 0 seconds and 6 seconds, preferably between 0.5 seconds and 3 seconds.

[0024] In another implementation of the method according to the first aspect, determining the object area covered by the object includes:

[0025] - determining the position and / or orientation of the object based on the sensor data; and

[0026] - determining the size of the object from the sensor data or determining the type of the object from the sensor data and inferring the size of the object using a predetermined size, wherein the predetermined size is determined based on the determined type of the object.

[0027] In addition to the sensor data obtained from one or more sensors of the vehicle, additional annotations such as object type (e.g., vehicle type) may be available. This information can help determine the object area more accurately. The data set may also include detailed information, such as the specific size and other properties of the object. Alternatively, the object type may be determined by a method based on sensor data.

[0028] In another implementation of the method according to the first aspect, the object area is a probabilistic area and / or the future vehicle trajectory is a probabilistic trajectory, and if the probability that the object area and the future vehicle trajectory intersect is greater than a predetermined intersection probability threshold, an overlap is determined. Due to various uncertainties involved that may affect the driving process of the vehicle and the movement behavior of objects around the vehicle, it is advantageous to consider probabilistic factors of both the vehicle trajectory and the object area to consider future trends that may affect the behavior of the vehicle. Probabilistic trajectories may be affected by uncertainty about positioning due to inaccuracies in acquiring GPS signals at high speeds, for example.

[0029] In another implementation of the method according to the first aspect, marking the object as relevant includes marking a plurality of objects as relevant, and the method further includes marking one of the plurality of relevant objects as a first relevant object when: the first relevant object has the smallest distance to a vehicle along the future vehicle trajectory. By further distinguishing the relevant objects, a hierarchy of relevance can be created, and the most relevant object among the relevant objects can be determined as first relevant. In particular, the first relevant object can be a first vehicle, because the first vehicle usually travels on the future vehicle trajectory, that is, it is always relevant.

[0030] Another aspect of the present invention relates to a sensor data annotation device, which is configured to implement the method described above.

[0031] Another aspect of the present invention relates to a computer-readable storage medium storing program code, wherein the program code comprises instructions which, when executed by a processor, implement the method of the second aspect or one of the implementations of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are intended to provide a further understanding of the present invention. In this regard, the accompanying drawings show the different steps and phases of the method and, in addition to the description, illustrate the concept of the present invention. Many of the advantages mentioned will be apparent with reference to the accompanying drawings. The elements shown in the drawings are not necessarily shown to scale relative to each other.

[0033] Figure 1 is a schematic diagram of the locations of several vehicles and object areas, wherein one of the vehicles is identified as a leading vehicle.

[0034] Figure 2a is a schematic diagram of the locations and object areas of several vehicles at the current time point in a scene where a first vehicle is about to overtake a second vehicle.

[0035] Figure 2b yes Figure 1 Schematic diagram of the scenario of a, but considering the position of the vehicle at a future point in time.

[0036] Figure 3 Shows something like Figure 1 , Figure 2a and Figure 2b The scene shown in , but with object regions of different shapes.

[0037] Figure 4 A vehicle is shown having sensors sensing surrounding vehicles during acquisition of sensor data. DETAILED DESCRIPTION

[0038] The above description is only an implementation of the present invention, and the scope of the present invention is not limited thereto. Those skilled in the art can easily make any changes or substitutions. Therefore, the protection scope of the present invention shall be subject to the protection scope of the attached claims.

[0039] Figure 1 A vehicle 101 (hereinafter also referred to as the ego vehicle) on a street 100 at a current time point t is shown. The ego vehicle is equipped with sensors to record the surroundings, in particular dynamic objects 102 to 104, which may be other vehicles or other dynamic traffic participants around the ego vehicle 101. The ego vehicle 101 moves along a trajectory 110, which includes a past vehicle trajectory 110a and a future vehicle trajectory 110b. The ego vehicle 101 includes sensors that collect sensor data around the ego vehicle 101.

[0040] The self-vehicle is surrounded by a second vehicle 102, a third vehicle 103, and a fourth vehicle 104. Figure 1 In the preferred embodiment shown, the object area of ​​the object (here a vehicle) corresponds to the actual area occupied by the vehicle, i.e. Figure 1 The vehicles shown correspond to the object regions of the vehicles. At the first time point, the object region of the third vehicle 103 overlaps with the future vehicle trajectory 110b. Therefore, the third vehicle 103 is marked as a relevant object. The fourth vehicle 104 also overlaps with the future vehicle trajectory 110b and is therefore also marked as a relevant object. The object region of the second vehicle 102 does not overlap with the vehicle trajectory and is therefore considered unrelated.

[0041] Since the closest relevant object to the ego vehicle 101 along the future vehicle trajectory 110b is the vehicle 103, the vehicle 103 is considered as the first relevant object. Figure 1 It can be expected that when the ego vehicle goes further beyond the third vehicle 103, its future trajectory will no longer overlap with the third vehicle 103. Then, the fourth vehicle 104 will become the new first relevant object.

[0042] It should be understood that, in practice, the object region may be defined as being larger than the object itself. Figure 2aThe scene shown in shows an embodiment in which the object area is defined as a circular object area 201a, 202a, 203a around the center of the object 201, 202, 203. This may be particularly suitable for detecting objects that are not directly on the future trajectory of the ego vehicle but are, for example, traveling in an adjacent lane and may therefore be of importance.

[0043] Figure 2a A scenario is shown of an ego vehicle 201 traveling in the middle lane of a street 200. (Note that lane markings are not required for the method to work.) Surrounding the ego vehicle 201 are a second vehicle 202 traveling in the same lane and a third vehicle 203 traveling in an adjacent lane. Figure 2a The scenario shown in FIG. 2 corresponds to a first time point corresponding to the start of an overtaking maneuver of the ego vehicle 201. The ego vehicle 201 is traveling in the middle lane, and its driver is about to overtake a second vehicle 202 traveling in the middle lane in front of the ego vehicle 201. The trajectory 210 of the ego vehicle 201 can be divided into a past vehicle trajectory 210 a, which is a trajectory that the vehicle has passed before the current time point, and a future vehicle trajectory 210 b, which is a part of the trajectory that the vehicle will pass after the current time point.

[0044] The future trajectory 210 b of the ego vehicle reflects the process of changing lanes to the left lane and overtaking the second vehicle.

[0045] On the left lane, there is a third vehicle 203. At the current time point, the future trajectory 210b overlaps the object area of ​​the second vehicle 202 and the object area of ​​the third vehicle 203. Since the second vehicle 202 is closer than the third vehicle 203, the second vehicle is the first relevant vehicle.

[0046] Will Figure 2a The time point depicted in is denoted as time T. Note that Figure 2a In FIG. 2 , vehicles 202 and 203 appear as relevant objects because their object regions intersect with the future vehicle trajectories.

[0047] Figure 2b Shown with Figure 2a The same street 200 and the same vehicle, but at an earlier point in time. This earlier point in time is denoted as T-tau, where tau is a predetermined time distance before the point in time, preferably between 0 seconds and 3 seconds. Considering Figure 2b , an earlier point in time is depicted, where the ego vehicle 201 appears at location 201 ′ on its past trajectory.

[0048] exist Figure 2bAt the earlier time point T-tau shown in FIG, the object area of ​​the second vehicle 202' does not intersect the future trajectory 210b' because at this time point, the second vehicle has not yet moved to the middle lane. Therefore, at T-tau, the second vehicle is not (yet) a relevant object. However, it is identified as a future relevant object because it will Figure 2a The upcoming time point T depicted in becomes the relevant object.

[0049] exist Figure 2b At the time point T-tau indicated in , the object area of ​​the third vehicle 203' intersects with the future trajectory given in T-tau, so it is identified as a relevant object. In addition, it is also identified as a future relevant object because it will be Figure 2a That is, the third vehicle 203 ′ is a related object, a future related object, or a first related object.

[0050] In the above-described embodiments, the object area corresponds to the actual area covered by the object. However, this is not necessary. In other embodiments, the object area may be a circular area around the center of the vehicle, for example. In other embodiments, the shape of the object area is not a circular shape and may be any geometric shape. Figure 3 An implementation is shown where the target area is a rectangular shape located around the center of the object. The size of the rectangular shape can vary based on the object type and size. The shape and size of the target area can be based on predetermined safety parameters, such as minimum vehicle spacing. The circular shape is also suitable for pedestrians or cyclists who need a wide buffer area for safety reasons. Figure 3 Also shown are object areas 312 to 315 that vehicles 302 to 305 may have rectangular shapes around them. These shapes may be displaced forward from the center of the object. The advantage of this is that the object area may reflect not only the actual area covered by the object, but also the area that falls within the space that is considered "dangerous." For example, if the ego vehicle is within 5 meters in front of another vehicle or within 1 meter to the side of another vehicle, it may be considered dangerous. Similarly, the vehicle trajectory 310 may include a safety margin 320 having a width 322, and the fourth vehicle 304 may be considered to interfere with the future vehicle trajectory 310 because the safety margin 320 overlaps with the object area 314 of the fourth vehicle 304.

[0051] If the area of ​​the vehicle (which is based on the object type, such as a predetermined size of the vehicle type) is determined to interfere with the future vehicle trajectory, the "other vehicle" can be marked as relevant. In addition, marking the object as relevant may include: if it is determined that the object area overlaps with the future vehicle trajectory at a future time point after the current time point, marking the object as future relevant, wherein, preferably, the future vehicle trajectory is defined to include a position corresponding to a time point in the future compared to the current time point.

[0052] Figure 4 Different interrelationships between the ego vehicle 401 and external entities are shown. The vehicle 401 includes sensors that acquire recordings from areas 410a, 410b to the left and right of the vehicle and from an area 412 in front of the vehicle. There may be Figure 4 Additional sensors looking backwards are not shown.

[0053] The vehicle trajectory may be obtained directly by receiving positioning and orientation data from GPS signals 420, or indirectly by post-processing sensor data from around the ego vehicle 401 so that the vehicle's position and / or orientation over time may be calculated from the sensor data. Figure 4 As visualized in , the ego vehicle surroundings include objects 402 , 403 , which may be labeled as relevant, future relevant, and / or leading vehicles after the ego vehicle has completed acquisition of sensor data and the vehicle trajectory may then preferably be offline.

[0054] The method may be performed offline without any connection to any network, ie the method may rely entirely on previously recorded data.

[0055] Known methods usually process instantaneous (per frame) data or data collected a short time ago (the last few samples), while maintaining the temporal coupling (synchronicity) between the data recorded from the surrounding dynamic objects and the data about the own motion of the ego vehicle. In this way, these methods are able to reason about the surrounding dynamic objects using historical data. In addition, information from the static world (e.g. detected lane markings, road edges, etc.) is usually used to separately identify dynamic objects of higher importance, for example by focusing on objects in the ego lane.

[0056] The proposed method can add semantic labels to the basic annotations, specifically: relevant, first relevant, or future relevant. These semantic labels can reflect the relevance of the object with respect to a given traffic situation.

[0057] The approach is based on time-synchronized data - decoupling the measurements of the surrounding dynamic objects from the actual motion of the recorded ego vehicle. Consider a scene built from: basic annotations of the surrounding dynamic objects at a given time sample, and the integrated motion of the ego vehicle giving its overall path over the complete recording. Exploiting the basic properties of the ego-motion data, any past or future part of this path can be projected into the chosen scene. Creating this merged scene gives the opportunity to reason about past and future correspondences between dynamic actors on a given scene and the chosen path part, independent of their original recording time.

[0058] In the proposed method, this merged scene is used to determine the relevance of dynamic objects. Traffic-aware semantic annotations can be added to dynamic objects based on future path data. Note that this method only requires basic annotations of dynamic objects, and does not require annotations of any surrounding static objects such as lane markings, road edges, etc.

[0059] In a preferred embodiment, extracting the first object attribute from the continuous time series of basic annotations can be achieved as follows:

[0060] (1) By overlaying the future trajectory of the ego vehicle on the current annotations of the objects, it is possible to determine which objects will intersect with the future trajectory of the ego vehicle. Among these objects, the object closest to the future trajectory of the vehicle can be considered the first object, and this information can be identified in the annotation.

[0061] (2) Consider the object identified as the first object at a given time point t0. If this particular object 0-x If it exists in the basic annotation at the location, it is marked as t 0-x The first future object in , with a prediction horizon of x seconds. In this way, objects can be annotated as first future objects with increased temporal prediction capabilities.

[0062] In addition to the first object attributes, various other aspects of dependencies can be annotated in a similar manner, such as a dense highway traffic situation with multiple objects (vehicles) in a queue approaching an exit. In the case of merging into such a column of vehicles, the vehicle after which there is a safe gap to merge into can be identified during annotation. Thus, a semantic annotation that supports the merging maneuver can be obtained.

[0063] In multi-lane traffic situations, objects / vehicles that should be taken into account when planning a lane change maneuver or providing lane change advice to the driver can be identified.

[0064] Once relevant objects can be identified from basic annotations, the method can be extended to focus specifically on the detection and state estimation of relevant objects. By training learning-based methods on semantically annotated data, attention-like solutions can emerge, where the method can identify which parts of its input are more important: relevant objects (or the top object as in the example above) may have higher importance than other objects. Mastering such an attention method can have multiple benefits, such as: computational requirements can be reduced (e.g., by eliminating detection and tracking of irrelevant objects), the output is clearer and more structured, the accuracy of the measurement method can be more precise, etc.

Claims

1. A computer-implemented method for annotating sensor data recorded from around a vehicle (101, 201, 301, 401) at a plurality of time points, the method comprising: - determining an object (102-104, 202, 203, 302-305, 403) in the sensor data; - determining an object area (201a, 202a, 203a, 312-315) covered by the object (102-104, 202, 203, 302-305, 403) at a current point in time, and determining a future vehicle trajectory (110b, 210b) of the vehicle (101, 201, 301, 401); as well as - if it is determined that the object area (201a, 202a, 203a, 312-315) interferes with the future vehicle trajectory (110b, 210b), marking the object (102-104, 202, 203, 302-305, 403) as relevant.

2. The method according to claim 1, wherein: If the object area (201a, 202a, 203a, 312-315) is within a predetermined distance of the future vehicle trajectory (110b, 210b), the object area (201a, 202a, 203a, 312-315) is determined to interfere with the future vehicle trajectory (110b, 210b).

3. The method according to one of the preceding claims, wherein: The sensor data includes camera data, LIDAR and / or RADAR.

4. The method according to one of the preceding claims, further comprising the step of determining a trajectory (110, 210) of the vehicle (101, 201, 301, 401) based on: - using dedicated positioning sensors, in particular GPS sensors, to collect vehicle (101, 201, 301, 401) positioning and / or orientation data over time; or - collecting sensor data from around the vehicle (101, 201, 301, 401) and post-processing the sensor data to obtain the location and / or orientation of the vehicle (101, 201, 301, 401) over time.

5. The method according to one of the preceding claims, wherein: Determining an object area (201a, 202a, 203a, 312-315) covered by the object (102-104, 202, 203, 302-305, 403) comprises performing a mapping from the sensor data to a 3D space, and wherein the future vehicle trajectory (110b, 210b) is stored in the 3D space.

6. The method according to one of the preceding claims, wherein: The vehicle trajectory includes the location of the vehicle (101, 201, 301, 401) at the plurality of time points based on the sensor data.

7. The method according to one of the preceding claims, wherein: Marking the object (102-104, 202, 203, 302-305, 403) as relevant includes: if it is determined that the object area (201a, 202a, 203a, 312-315) at a future time point after the current time point overlaps with the future vehicle trajectory (110b, 210b), marking the object (102-104, 202, 203, 302-305, 403) as future relevant.

8. The method according to claim 7, wherein: The future time point is within a predetermined time distance to the current time point.

9. The method according to one of the preceding claims, wherein: Determining the object area (201a, 202a, 203a, 312-315) covered by the object (102-104, 202, 203, 302-305, 403) comprises: - determining the position and / or orientation of the object (102-104, 202, 203, 302-305, 403) based on the sensor data; and -determining the size of the object (102-104, 202, 203, 302-305, 403) based on the sensor data, or determining the type of the object (102-104, 202, 203, 302-305, 403) based on the sensor data and inferring the size of the object (102-104, 202, 203, 302-305, 403) using a predetermined size, wherein the predetermined size is determined based on the determined type of the object (102-104, 202, 203, 302-305, 403).

10. The method according to one of the preceding claims, wherein: The object area (201a, 202a, 203a, 312-315) is a probabilistic area and / or the vehicle trajectory is a probabilistic trajectory, and the overlap is determined if the probability that the object area (201a, 202a, 203a, 312-315) intersects with the vehicle trajectory is greater than a predetermined intersection probability threshold.

11. The method according to one of the preceding claims, wherein: Labeling the objects (102-104, 202, 203, 302-305, 403) as related includes labeling multiple objects (102-104, 202, 203, 302-305, 403) as related, and the method also includes labeling one of the multiple related objects as the first related object (103, 202, 304) when the first related object (103, 202, 304) has the smallest distance to the vehicle (101, 201, 301, 401) along the future vehicle trajectory (110b, 210b).

12. The method according to one of the preceding claims, wherein: The object area (201a, 202a, 203a, 312-315) is determined as a predetermined area around the center of the object (102-104, 202, 203, 302-305, 403).

13. The method according to claim 12, wherein: The size of the object area (201a, 202a, 203a, 312-315) depends on the speed of the vehicle (101, 201, 301, 401).

14. A sensor data annotation device, wherein: The device is configured to implement the method according to one of the preceding claims.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to implement the method according to one of claims 1 to 13.

Images