METHOD AND DEVICE FOR OBJECT TRACKING AND DRIVER ASSISTANCE SYSTEM

Description

[0001] The present invention relates to a method and a device for tracking objects in a surrounding of a vehicle. The invention further relates to a driver assistance system.Prior Art

[0002] Driver assistance systems rely on the detection of objects in a surrounding of the vehicle. Modem vehicles can comprise multiple sensors of different types, such as radar sensors, cameras, infrared sensors, lidar sensors and the like. US 10 936 902 B1 relates to training bounding box selection. US 2021 / 0080558 A1 relates to extended object tracking using radar. US 2016 / 0103213 A1 relates to 3D tracking of objects in a radar system.

[0003] Based on the sensor data, objects can be identified. For example, a radar sensor can emit radar signals which are reflected by objects in a surrounding of the vehicle. The coordinates of the reflecting objects can be determined by analyzing the received radar signals. Each of the recognized objects can be tracked over time. For example, the moving directions or velocities of the tracked objects can be determined. Such information is important to estimate the likelihoods of possible events, such as a possible collision with another vehicle. Based on this information, the driver assistance systems can inform the driver of possibly dangerous situations or can control functions of the vehicle, such as an adjustment of the speed or the steering angle of the vehicle.

[0004] For tracking objects, newly detected measurements must be associated with already tracked objects, if possible. Tracking solutions are taking care of the association problem between objects detected in a subsequent frame by creating dynamic entities with a predefined lifespan. Although there are various solutions, due to the computation resource limitations, usually modeling assumptions have to be introduced. Filtering techniques (like Kalman filtering) can give a procedure-skeleton to such assumptions, in which the main steps are the following: prediction of the state of the observed objects for the next timestamp, the association between the predicted state of the object and the measurement in the next timestamp, and an update of the state of the observed object for further tracking.Disclosure of the Invention

[0005] The present invention provides a method and a device for tracking objects in a surrounding of a vehicle, and a driver assistance system, as recited in the independent claims.

[0006] Preferred embodiments are set out in the dependent claims.

[0007] According to a first aspect, the invention therefore relates to a device for tracking objects in a surrounding of a vehicle, comprising an interface configured to receive, from at least one vehicle sensor of the vehicle, newly acquired measurement data. The device further comprises a computing device. The computing device is configured to determine time coordinates and spatial coordinates of newly measured objects in a surrounding of the vehicle based on the newly acquired measurement data, wherein the spatial coordinates of the newly measured objects are defined relative to a global coordinate system. The computing device is further configured to associate each newly measured object with at most one tracked object of a plurality of tracked objects based on distances between the newly measured object and the plurality of tracked objects, wherein the distance depends on both a difference between the spatial coordinates of the newly measured object and the tracked object and a difference between the time coordinates of the newly measured object and the tracked object. The computing device updates spatial coordinates of the tracked objects using a Kalman filtering method, using the associated newly measured objects.

[0008] According to a second aspect, the invention provides a driver assistance system for a vehicle, comprising at least one vehicle sensor of the vehicle configured to acquire measurement data, and a device for tracking objects in a surrounding of a vehicle according to the first aspect of the invention.

[0009] According to a third aspect, the invention relates to a method for tracking objects in a surrounding of a vehicle. The method comprises receiving, from at least one vehicle sensor of the vehicle, newly acquired measurement data and determining time coordinates and spatial coordinates of newly measured objects in a surrounding of the vehicle based on the newly acquired measurement data, wherein the spatial coordinates of the newly measured objects are defined relative to a global coordinate system. The method further comprises associating each newly measured object with at most one tracked object of a plurality of tracked objects based on distances between the newly measured object and the plurality of tracked objects, wherein the distance depends on both a difference between the spatial coordinates of the newly measured object and the tracked object and a difference between the time coordinates of the newly measured object and the tracked object; and updating spatial coordinates of the tracked objects using a Kalman filtering method, using the associated newly measured objects.Advantages of the invention

[0010] The present invention applies a space-time input representation and a modification to the association step. By using a distance which depends both on spatial coordinates and time coordinates, tracking is more robust to object movement. An improved object detector performance can be achieved. The tracking can be achieved in a space-time representation, where the distance metric is applied. Further, movement-dependent linear patterns and their recognition can be used.

[0011] Relying on the output of multiple kinds of object detectors - wherein each detector's output is transformed into the global coordinate system - one stream of detections can be produced.

[0012] When the ego vehicle's movement is properly tracked by transforming the object detectors' frames (i.e., coordinates) taken at different time instants into a common coordinate system, and a proper multi-dimensional representation is chosen (2D / 3D Cartesian coordinates and a time-like attribute), movement-dependent linear patterns can be formed for each object. Also, if these patterns are detected, their speed and momentum can be calculated to stabilize the classification itself.

[0013] According to the invention, the distance D (i.e. metric) between a newly measured object and a tracked object is computed by the following formula: D 2 = dx 2 + dy 2 + dz 2 − vt 2 , wherein dx, dy and dz denote differences between the spatial coordinates of the newly measured object and the tracked object for three orthogonal axes of the global coordinate system, wherein v denotes a velocity of the vehicle, and wherein t denotes a difference between the temporal coordinate of the newly measured object and the tracked object.

[0014] That is, the distance response to a space-time interval. The distance is selected in a way that static objects (i.e., moving with the speed of the vehicle) have zero distance. The association of the object therefore takes the movement of the vehicle into account by using a distance which takes care of the ego vehicle movement. This leads to a more robust association. Also, by applying the above formula for the distance, asynchronous measurements merged into one continuous input stream can be handled, as a way of multimodal late-fusion technique. For the multimodal setup, a synchronization method may be present (e.g., timestamp-based).

[0015] According to a further embodiment of the device, the computing device is configured to determine the global coordinate system as a current pose of the vehicle, based on a current velocity of the vehicle. After each measurement cycle, the global coordinate system can be recomputed.

[0016] According to a further embodiment of the device, the computing device is configured to associate the newly measured object with the at most one tracked object using the Hungarian method. Herein, the above metric defined in formula (1) can be used instead of a conventional Euclidean metric.

[0017] According to a further embodiment, the device further comprises a first-in first-out, FIFO, memory device, wherein the computing device is configured to store the time coordinates and spatial coordinates of newly measured objects in the FIFO memory device. For an efficient space-time representation, multiple past frames (i.e., time coordinates and spatial coordinates) can be kept.

[0018] According to a further embodiment of the device, the computing device is further configured to transform spatial coordinates of the plurality of tracked objects into the global coordinate system before associating each newly measured object with at most one tracked object of the plurality of tracked objects. Therefore, the newly measured objects can easily be associated with the tracked object

[0019] According to a further embodiment of the device, the computing device is further configured to generate a new tracked object based on a newly measured object if said newly measured object is not associated with any tracked object of the plurality of tracked objects. Therefore, the list of tracked objects can be extended.

[0020] According to a further embodiment of the device, the computing device is further configured to delete a tracked object of the plurality of tracked objects if no newly measured object is associated with the tracked object. Therefore, objects no longer present in the field of view of the at least one vehicle sensor can be deleted. Such objects are generally no longer relevant for driver assistance systems or the like.

[0021] Short description of the drawings Figure 1shows a block diagram of a driver assistance system comprising a device for tracking objects in a surrounding of a vehicle according to an embodiment of the invention; Figure 2shows a schematic illustration of tracked objects; and Figure 3shows a flow chart of a method for tracking objects in a surrounding of a vehicle according to an embodiment of the invention.

[0022] The numbering of process steps is for clarity and is generally not intended to imply any particular chronological order. In particular, several process steps can also be carried out simultaneously.Description of the embodiments

[0023] Figure 1 shows a block diagram of a driver assistance system 6 comprising a device 1 for tracking objects in a surrounding of a vehicle. The device 1 comprises an interface 2 which communicates with one or more vehicle sensors 5 of the vehicle. Preferably, the vehicle sensors may comprise at least one radar sensor. The vehicle sensors 5 can also comprise sensors like cameras, lidar sensors, infrared sensors and the like.

[0024] The interface 2 may communicate with the vehicle sensors 5 via a CAN bus or the like. The device 1 receives via the interface 2 newly acquired measurement data from the vehicle sensor 5. That is, after each measurement time (time stamp), newly acquired measurement data is provided to the device 1. The device 1 may receive raw data from the radar sensor 5. According to other embodiments, the data may already be preprocessed.

[0025] The device 1 comprises a computing device 4, such as a microprocessor, microcontroller, a programmable gate array, or the like. The computing device 4 determines time coordinates and spatial coordinates of newly measured objects in a surrounding of the vehicle based on the newly acquired measurement data. Herein, the term "newly measured object" relates to a particular measurement (i.e., within a present measurement cycle) of an object. The object can be an object already tracked by the device 1 or a new object.

[0026] The spatial coordinates of the newly measured objects are defined relative to a global coordinate system, which may be given as a current pose of the vehicle, e.g., based on a current velocity of the vehicle.

[0027] The computing device 4 associates each newly measured object with one or none tracked object of a plurality of tracked objects. At each time, multiple objects may be tracked, e.g., corresponding to further vehicles, barriers, road signs, pedestrians and so on.

[0028] The computing device 4 associates newly measured objects with a tracked object by computing for each tracked object a distance between the newly measured object and said tracked object. The distance is computed based on both a difference between the spatial coordinates of the newly measured object and the tracked object and a difference between the time coordinates of the newly measured object and the tracked object. According to the invention, the distance is computed using formula (1) above. In general, the difference between the time coordinates of the newly measured object and the tracked object corresponds to the time difference between two subsequent measurements.

[0029] The computing device 4 further updates the spatial coordinates of the tracked objects using a Kalman filtering method.

[0030] The computing device 4 may comprise a tracking pool for gathering the available Kalman filter instances, i.e., every tracked object has one corresponding instance.

[0031] The device 1 further comprises a first-in first-out, FIFO, memory device 3 which stores the time coordinates and spatial coordinates corresponding to several previously measured objects. Tracking means monitoring one stream that contains subsequent frames in which the detected objects are located. For the space-time representation, multiple past frames can be kept. If these frames are in the same global coordinate system, linear patterns can be captured. Further, the memory device 3 may store the global coordinate system.

[0032] Figure 2 shows a schematic illustration of tracked objects. Herein, there are moving objects 21 and 22 and static objects 23 and 24.

[0033] Figure 3 shows a flow chart of a method for tracking objects in a surrounding of a vehicle. The method can be carried out using the device 1 described above. Likewise, the device 1 can be configured to carry out any of the following method steps or aspects of the method.

[0034] In a step S1, tracking is initialized. Further, a tracking pool is initialized which may have an instance count of 0. Further, FIFO memory storage may be allocated. The size can be maximized, e.g., restricted to 5 frames. The current pose of the vehicle may be extracted and stored as a global coordinate system. However, the invention is not restricted to the vehicle coordinate system as a global coordinate system. According to other embodiments, different common global coordinate systems can be used. The memory storage may be relocated.

[0035] After a drift occurs in ego vehicle positioning, the global coordinate system changes. The current ego vehicle is then again extracted as a global coordinate system. All tracked objects from the tracking pool are transformed by transforming their spatial coordinates into the new global coordinate system. The new ego vehicle pose is stored as the global coordinate system.

[0036] Next, measurement initiates. The current ego vehicle speed is extracted. At least one vehicle sensor of the vehicle acquires measurement data. Newly acquired measurement data is provided to the device 1 for tracking the objects. The device 1 may be located in the vehicle. At least some functions of the device 1 can also be located outside of the vehicle, e.g., in a distant server.

[0037] In a second step S2, the device 1 determines time coordinates and spatial coordinates (i.e., a measurement frame) of newly measured objects in a surrounding of the vehicle based on the newly acquired measurement data. The time coordinates may simply be timestamps. The spatial coordinates of the newly measured objects are defined relative to a global coordinate system. The spatial coordinates may first be defined in a non-global coordinate system and may then be transformed into the global coordinate system.

[0038] Together with the measurement data, the ego vehicle pose is tracked in the global coordinate system (i.e., common or "world" coordinate system) for each time instant where measurements are taken.

[0039] Moreover, a proper calibration may be required to be able to do affine transformation based coordination system changes between sensor coordinate systems, the space-time representation and the transformation into the global coordinate system.

[0040] In a step S3, the device 1 associates each newly measured object with at most one tracked object of a plurality of tracked objects based on distances between the newly measured object and the plurality of tracked objects. The distance depends on both a difference between the spatial coordinates of the newly measured object and the tracked object under consideration and a difference between the time coordinates of the newly measured object and the tracked object under consideration.

[0041] The difference between the time coordinates can be set to 0 in the first iteration and can otherwise be set to the difference between a current timestamp and a stored timestamp.

[0042] It can also be possible to store a 4-dimensional vector {x, y, z, vt} associated with each newly acquired object, comprising the spatial coordinates x, y, z and a parameter νt = deltaT ⋅ ego vehicle speed − length value , where deltaT corresponds to the difference between a current timestamp and a stored timestamp and the length value is set based on a length of the vehicle. The transformed measurement frame {x, y, z, vt} is pushed into the memory storage 3.

[0043] In a fourth step S4, the spatial coordinates of the tracked objects are updated, using a Kalman filtering method. In space-time tracking, the past information is used to stabilize the association step of the selected Kalman filter, by the movement derived from the state change of the continuously updated tracklet (tracked object).

[0044] Object detectors do have some noise arising from the measuring technique. Also, the tracking process can have ambiguity. These have to be modeled, a common choice is to use an additional Normal distribution, and both have a dataset dependent hyperparameter (variance). The tracking is based on Kalman filtering. Herein, the state-transition model, the observation model, the measurement noise and the process noise are identical to the standard Kalman filtering method. The noises can be the above-mentioned normal distributions, e.g., with a variance of 0.01, and a constant state-transition model can be used.

[0045] In the Kalman-update step, the current tracked state of the object is updated. Herein, the associated newly measured object relates to new measurements taken into account in the Kalman method. If there is no match for some newly measured object, a new tracked object is generated. If none of the newly measured objects is associated with one of the existing tracked objects, the tracked object is deleted.

[0046] Optionally, for currently available tracked objects, further classification can be applied. the classification can be dynamic or static, depending on the tracked object attributes available in Kalman filter inner states, e.g., momentum or speed.

[0047] The method can then repeat again from step S1.

Claims

1. A device (1) for tracking objects in a surrounding of a vehicle, comprising: an interface (2) configured to receive, from at least one vehicle sensor (5) of the vehicle, newly acquired measurement data; and a computing device (4) configured to: determine time coordinates and spatial coordinates of newly measured objects in a surrounding of the vehicle based on the newly acquired measurement data, wherein the spatial coordinates of the newly measured objects are defined relative to a global coordinate system; associate each newly measured object with at most one tracked object (21-24) of a plurality of tracked objects (21-24) based on distances between the newly measured object and the plurality of tracked objects (21-24), wherein the distance depends on both a difference between the spatial coordinates of the newly measured object and the tracked object and a difference between the time coordinates of the newly measured object and the tracked object; and update spatial coordinates of the tracked objects (21-24) using a Kalman filtering method, using the associated newly measured objects; characterized in that the distance D between a newly measured object and a tracked object (21-24) is computed by the following formula: D 2 = dx 2 + dy 2 + dz 2 − vt 2 , wherein dx, dy and dz denote differences between the spatial coordinates of the newly measured object and the tracked object (21-24) for three orthogonal axes of the global coordinate system, wherein v denotes a velocity of the vehicle, and wherein t denotes a difference between the temporal coordinate of the newly measured object and the tracked object (21-24).

2. The device (1) according to claim 1, wherein the computing device (4) is configured to determine the global coordinate system as a current pose of the vehicle, based on a current velocity of the vehicle.

3. The device (1) according to claim 1 or 2, wherein the computing device (4) is configured to associate the newly measured object with the at most one tracked object (21-24) using the Hungarian method.

4. The device (1) according to any of the preceding claims, further comprising a first-in first-out, FIFO, memory device (3), wherein the computing device (4) is configured to store the time coordinates and spatial coordinates of newly measured objects in the FIFO memory device (3).

5. The device (1) according to any of the preceding claims, wherein the computing device (4) is further configured to transform spatial coordinates of the plurality of tracked objects (21-24) into the global coordinate system before associating each newly measured object with at most one tracked object (21-24) of the plurality of tracked objects (21-24).

6. The device (1) according to any of the preceding claims, wherein the computing device (4) is further configured to generate a new tracked object (21-24) if a newly measured object is not associated with a tracked object (21-24) of the plurality of tracked objects (21-24).

7. The device (1) according to any of the preceding claims, wherein the computing device (4) is further configured to delete a tracked object (21-24) of the plurality of tracked objects (21-24) if no newly measured object is associated with the tracked object (21-24).

8. A driver assistance system (6) for a vehicle, comprising at least one vehicle sensor (5) of the vehicle, configured to acquire measurement data; and a device (1) for tracking objects in a surrounding of a vehicle according to any of the preceding claims.

9. A method for tracking objects in a surrounding of a vehicle, comprising the following steps: receiving (S1), from at least one vehicle sensor (5) of the vehicle, newly acquired measurement data; determining (S2) time coordinates and spatial coordinates of newly measured objects in a surrounding of the vehicle based on the newly acquired measurement data, wherein the spatial coordinates of the newly measured objects are defined relative to a global coordinate system; associating (S3) each newly measured object with at most one tracked object (21-24) of a plurality of tracked objects (21-24) based on distances between the newly measured object and the plurality of tracked objects (21-24), wherein the distance depends on both a difference between the spatial coordinates of the newly measured object and the tracked object and a difference between the time coordinates of the newly measured object and the tracked object; and updating (S4) spatial coordinates of the tracked objects (21-24) using a Kalman filtering method, using the associated newly measured objects; characterized in that the distance D between a newly measured object and a tracked object (21-24) is computed by the following formula: D 2 = dx 2 + dy 2 + dz 2 − vt 2 , wherein dx, dy and dz denote differences between the spatial coordinates of the newly measured object and the tracked object (21-24) for three orthogonal axes of the global coordinate system, wherein v denotes a velocity of the vehicle, and wherein t denotes a difference between the temporal coordinate of the newly measured object and the tracked object (21-24).