Method for predicting behavior of target object

Abstract

The invention relates to a method for predicting the behavior of a target object (2) in a traffic scenario, the method comprising the following method steps: Step I: selecting a traffic scenario in which measurement points of a radar measurement process of a vehicle (1) radar sensor have been generated for detecting objects in the vehicle (1) surroundings which are assigned to the measurement points; Step II: labeling the measurement points, wherein measurement points which are not assigned to a target object (2) are removed; Step III: transmitting the labeled measurement points to an offline radar tracker which predicts the behavior of the target object (2) in the respective traffic scenario in such a way that the offline radar tracker is adjusted by at least one external parameter in order to predict the behavior of the target object (2) for the respective traffic scenario; Step IV: applying a smoothing algorithm to the predicted behavior of the target object (2) in order to further optimize the behavior of the target object (2).

Description

Technical Field

[0001] This invention relates to a method for predicting the behavior of target objects in traffic scenarios, and the application of such behavior as a reference benchmark for radar sensors in online radar trackers. Background Technology Technical Background

[0003] Modern vehicles, such as cars and motorcycles, are increasingly equipped with driver assistance systems. These systems use sensor systems to perceive the surrounding environment, identify traffic conditions, and provide assistance to the driver, such as implementing braking or steering interventions, or issuing visual or auditory warnings. Sensor systems for environmental perception typically employ radar sensors, lidar sensors, camera sensors, or similar sensors. From the sensor data obtained, the surrounding environment can then be inferred. Environmental perception using radar sensors is based on the emission and reflection of focused electromagnetic waves, such as those reflected by other road users, obstacles on the road, or roadside structures. In this process, the radar sensor generates measurement points or detection results, which are then assigned to an object, for example, and subsequently tracked by a radar object tracker. Pedestrian detection is typically performed using camera sensors, but radar sensors are also increasingly being used.

[0004] For systems of the aforementioned types, radar sensors can also be integrated with other sensor technologies, such as camera sensors or lidar sensors. One advantage of radar sensors is their reliable operation even in adverse weather conditions, and in addition to detecting object distance, they can directly measure the radial relative velocity of objects through the Doppler effect. Transmission frequencies typically used are 24 GHz, 77 GHz, and 79 GHz. As the functional range of such systems continues to expand, the requirements for the systems are also continuously increasing, especially the maximum detection distance. Besides vehicle environmental perception, vehicle interior monitoring is also receiving increasing attention for these types of systems, such as for identifying which seats are occupied; in this process, frequencies in the 60 GHz band may be used, for example.

[0005] In this context, the present invention relates to the verification of radar data generated by radar sensors, and more particularly to the verification of the output of a radar object tracker in a (test) vehicle equipped with a radar sensor and a radar object tracker. This output is the tracked radar object, characterized or identifiable by its kinematic properties (e.g., position, velocity, acceleration, etc.). According to existing technology, external sensors or reference systems, such as differential GPS (Global Positioning System), lidar, or camera sensors, can also be used for this purpose. These external sensors provide the kinematic properties of a corresponding reference object, which the radar tracker output can be compared to for verification and / or accuracy estimation. For this purpose, the external sensors must also be arranged or installed in the test vehicle and properly calibrated, which can result in significant cost expenditure. Another disadvantage is that the measurement characteristics of external sensors differ from those of radar sensors, thus often requiring additional post-processing work to finally compare the object from the radar object tracker with the object from the external reference sensor.

[0006] Existing technical documents

[0007] According to US 2023 / 0258794 A1, a method is known in which radar data is combined with previous radar datasets to create a combined radar dataset. Each radar dataset and the combined radar dataset can contain distinct data points, and each data point can be assigned to a specific feature or category. The combined dataset can then be preprocessed to identify the distance, location, and velocity associated with a particular data point, so that each corresponding data point can be characterized and represented using data that identifies the location, velocity, and other physical details of the corresponding data point. This preprocessed data can then be processed by a machine learning process to identify kinematic information, which can then be provided to the AV's tracking system. Therefore, the basis of this method is the cumulative collection of radar data to simplify correlation and create classifications as a pre-tracking step. Here, an online mechanism is involved, where the focus is not on tracking / object creation, but rather on employing a deep learning approach where no offline reference benchmark is generated. Summary of the Invention

[0008] The task of this invention

[0009] Based on existing technology, the objective of this invention is to provide a method for predicting the behavior of a target object, thereby reducing computational or post-processing workload in a cost-effective manner.

[0010] Solution to the task

[0011] The aforementioned task is solved by all the technical solutions of claim 1 and the parallel independent claims. Advantageous embodiments of the invention are claimed in the dependent claims.

[0012] According to the present invention, the proposed method for predicting the behavior of a target object in a traffic scenario (i.e., the kinematic state or kinematic properties of the target object, which can be determined, for example, by its position and / or velocity and / or acceleration and / or extended dimensions) includes the following method steps:

[0013] Step 1: Select a traffic scenario in which measurement points for the radar measurement process of the vehicle's radar sensor have been generated to detect objects in the environment surrounding the vehicle that belong to the measurement points (i.e., an online traffic condition dataset).

[0014] Step II: Mark or label the measurement points, where measurement points not assigned to the target object (e.g., static objects, roadside buildings, other traffic participants, VRU measurement points, unintended measurement points of the actual target object, such as outliers, or similar cases) are removed;

[0015] Step III: The marked measurement points are transmitted to an offline radar tracker, which (offline) predicts the behavior of target objects in the corresponding traffic scenario by adjusting the offline radar tracker using at least one external parameter to predict the behavior of target objects for the corresponding traffic scenario.

[0016] Step IV: Apply a smoothing algorithm to the predicted behavior of the target object to further optimize its behavior.

[0017] Advantageously, object initialization, context object initialization, or filter behavior can be set as extrinsic parameters. However, alternatively or additionally, detection point-object association parameters or initial variance can also be used.

[0018] Preferably, as object initialization, parameters such as the orientation, velocity, and acceleration of the target object are set, which cannot be measured by radar in a single cycle.

[0019] In addition, as a smoothing algorithm, Rauch-Tung-Striebel smoothing and / or improved Bryson-Frazier smoothing and / or minimum variance smoothing and / or dual-filter smoothing can be set.

[0020] Advantageously, the measurement points marked in step II can be stored in such a way that these measurement points correspond one-to-one with / uniquely match the corresponding traffic scenarios.

[0021] Preferably, the method further includes step V, wherein the result of step IV is used as a reference benchmark for the online tracker.

[0022] Furthermore, this invention also claims an application that uses the behavior of a target object determined by the method according to the invention as a reference benchmark for an online radar tracker of a radar sensor. Attached Figure Description

[0023] Description of the present invention with reference to embodiments

[0024] The invention will now be further described with reference to advantageous embodiments. In the accompanying drawings:

[0025] Figure 1 A simplified diagram illustrating the design scheme of the method flow according to the present invention is shown;

[0026] Figure 2 A simplified schematic diagram showing the radar measurement results of the vehicle's radar sensor, and

[0027] Figure 3 Show Figure 2 A simplified schematic diagram of the radar measurement results, wherein the radar measurement results have been cleaned according to the present invention. Detailed Implementation

[0028] This invention specifically relates to creating "optimal" detection images by labeling or marking individual cycles, thereby minimizing the workload of developing trackers because no exceptions or special treatments are needed in the algorithm, which is often necessary, for example, in online trackers. Furthermore, the data association step is completely omitted because all remaining detection points will be tracked. Additionally, data smoothing or smoothing processing is preferably performed, on the one hand, by different smoothing algorithms, and on the other hand, by forward and backward traversal of the data (dual-filter smoothing). In this process, the data is forward-traversed and tracked once, as in an online system, where the final value of the last cycle is used as the starting value for the backward traversal. Inverse vehicle motion compensation can be performed during this process, and the final result is a weighted average of the two tracking traversals.

[0029] Here, a key aspect of the invention is the generation of a reference benchmark (“radar as reference benchmark”) using the same radar data input as that used by a radar tracker employing a similar radar sensor, such as one operating in real time. By applying pre-processing and post-processing measures and significantly relaxing hardware limitations (e.g., in terms of memory and runtime), this so-called reference tracking system can provide high-quality object kinematics characteristics without an external reference sensor. Specifically, the invention relates to the verification of the radar object tracker output. This output is typically the tracked radar object, characterized by its kinematic properties (e.g., position, velocity, acceleration). Furthermore, there is no need to set up external reference systems, such as differential GPS, lidar, or camera sensors, which provide reference object kinematics; the radar tracker output can be compared to these reference object kinematics for verification and accuracy estimation. Since there is therefore no need to install and properly calibrate external sensors in the test vehicle, costs and manufacturing effort can be significantly reduced. Furthermore, post-processing effort can be reduced because there is no need to match the measurement characteristics of different sensors to each other.

[0030] The reference baseline is generated using the same radar data input as mass-produced object trackers that operate in real-time and run on the hardware of actual radar sensors. High-quality object kinematics can be provided without external reference sensors by applying pre-processing and post-processing measures and by significantly relaxing hardware boundary conditions (memory, runtime). The created reference baseline can also be created for static objects, such as stationary cars or static environmental objects. Furthermore, multiple reference objects can be simultaneously labeled in each traffic scene, such as multiple cars traveling in front and behind, all appearing within the sensor's detection range and meaningful to the reference baseline. While this makes data correlation difficult, it can still be simplified in a practical way, for example, by pre-setting the correlation range using external parameters.

[0031] The process itself, or the method of the present invention, is an offline process applied after measuring a specific traffic scenario and storing the measurements in a measurement file, which is then used as the starting point for a reference benchmark generation process. Therefore, the method can also be executed as a computer-implemented method. Here, "computer-implemented method" in the sense of the present invention describes a process planning or operation mode implemented or executed by a computer. Here, the computer can process data through programmable computational rules. Therefore, the method can also be executed later, for example, through a new program, multiple new programs, algorithms, etc. Here, the computer can be designed as a control device, an IC (integrated circuit) module, a microcontroller, a processor, or a system-on-a-chip (SoC) or similar form.

[0032] Preferably, the method according to the invention utilizes... Figure 1 The following steps should be performed:

[0033] Select a scenario or a traffic scenario (Step I)

[0034] First, select the traffic scenario for which you want to create a reference baseline. This can be any relevant situation, such as objects or traffic participants crossing the road, objects or traffic participants turning, static objects, roundabouts, distant objects, VRUs (vulnerable road users) such as pedestrians or bicycles, or similar situations. Figure 2 The diagram shows an example of radar measurement results from the vehicle 1's radar sensor. Simplified measurement points are indicated by asterisks. Besides the target object 2, the measurement results could be, for example, mirrored measurement results 3 (or mirrored pseudo-targets or mirrored objects), road boundaries such as guardrails 4, roadside buildings or roadside vegetation, or other static objects 5. Furthermore, so-called outliers 6 can be detected; these outliers are also objects but are excluded due to their location or orientation.

[0035] Mark or label traffic scenes (Step II)

[0036] During the labeling or marking process, all radar measurements that are irrelevant to the object of interest or target object 2 in the selected traffic scene are removed. This primarily involves the static environment, other traffic participants, and the identification of selected objects that cannot be used for tracking, such as mirrored measurements 3, guardrails 4, other static objects 5, and outlier measurements 6, which may distort the tracking process. These marked detection points are stored in a unique label file specific to the corresponding traffic scene.

[0037] Transmit the radar measurement results to the radar object tracker (Step III)

[0038] After data cleaning of radar images or radar measurement results (see...) Figure 3The relevant radar measurements are fed into a conventional (forward) radar object tracker, such as an electronic control unit (ECU). This offline reference tracker can then be adjusted using external parameters to optimize its behavior for the corresponding traffic scenario. These parameters may include, for example, object initialization, filter behavior, or similar parameters. Object initialization involves executing the offline reference tracker to learn the object's motion or behavior, as the online system is unaware of the object's motion upon first viewing. By executing the offline reference tracker, the behavior can be learned, and the object can be correctly configured for motion orientation and kinematic characteristics (e.g., acceleration, velocity). Various traffic scenarios with different dynamic behavior types can be adapted using, for example, the filter behavior of a tracking filter. This requires some form of scene recognition and response in the online system. However, in the current offline reference tracker, the traffic scenario is known, and the correct filter dynamic characteristics can be selected for the corresponding traffic scenario. These parameters are therefore also stored for this traffic scenario, where the development effort of the tracker can be minimized by adjusting the external parameters. As output, the kinematic characteristics of the tracked target object can be set, as in the online system, but optimized for the given traffic scenario.

[0039] Apply the smoothing algorithm (Step IV)

[0040] By applying a smoothing algorithm to the output of step III, the current behavior estimate can be improved. This smoothing algorithm preferably works countercausally, allowing for the incorporation of future knowledge about the target's motion. This post-processing step is not available in online systems—but can be applied in offline systems. After executing the (forward) object tracker, the kinematic state and variance of the target object / objects are available, whereby a smoothing algorithm is subsequently applied to these results, preferably "backward-traversing the data" and optimizing the state. For example, a Rauch-Tung-Striebel (URTSS) smoother or a dual-filter smoother can be set as the smoothing algorithm, where the output of the smoother is the optimized kinematic properties of the object / objects of interest. In a preferred embodiment, a Rauch-Tung-Striebel (RTS) smoother is used, with an additional dual-filter smoother that traverses both forward and backward in time.

[0041] Create a reference datum format (Step V)

[0042] The result of smoothing is optimized object kinematics, which is now used as a reference benchmark for the online tracker. The online tracker can then use this benchmark as a target performance objective.

[0043] Furthermore, the reference benchmark format can be a suitable format for an existing black-box testing toolchain, allowing the reference benchmark to be added accordingly (step VI), whereby the tracker can then repeat the so-called benchmark testing. Additionally, the online tracker can test these reference benchmarks after each code change and check for improvements or shortcomings compared to previous iterations.

[0044] The reference tracker can be implemented arbitrarily in practice. Since there are no limitations on runtime and memory, real-time capabilities are not a concern. Possible implementations could be CTRA trackers, CA trackers, or IMM trackers, or other methods such as particle filters. For example, it could be implemented as an tunable Kalman filter with constant acceleration (an "extended Kalman filter" using a CA model with states x, VxAbs, AxAbs, y, VyAbs, AyAbs) or an IMM tracker with CTRA (constant rotational speed and acceleration) models, CA (constant acceleration) and CV (constant velocity) models.

[0045] In a preferred implementation, the adjustable parameters are object initialization characteristics (such as orientation, span, velocity, and / or acceleration) that cannot be measured by radar within a single cycle, and filter dynamics that define how quickly or slowly the filter responds to, for example, changes in the orientation of the target object. Typically, all internal parameters of the tracker can be made adjustable via externally stored parameters specific to the traffic scenario. These could be, for example, detection point-object association parameters or initial variance. Furthermore, for smoothing, all known smoothing algorithms with fixed intervals can be used, such as Rauch-Tung-Striebel smoothing, a modified Bryson-Frazier smoother, or minimum variance smoothing and / or similar algorithms.

[0046] While the marking or labeling process can also be performed manually, this requires a high time-intensive workload to mark multiple radar detection points for each measurement cycle. In a preferred embodiment, the marking process is partially, particularly up to about 90%, automated in that the detection points associated with the selected object(s) are marked in a non-parameter-adapted association step. Afterwards, manual inspection and data cleaning can also be performed to remove unwanted identifications or add missed identifications or detection points.

Claims

1. A method for predicting the behavior of a target object (2) in a traffic scenario, the method comprising the following steps: Step 1: Select a traffic scene in which measurement points for the radar measurement process of the vehicle (1) radar sensor have been generated to detect objects in the environment surrounding the vehicle (1) that belong to the measurement points. Step II: Mark the measurement points, where, Measurement points not assigned to target object (2) were removed; Step III: The marked measurement points are transmitted to an offline radar tracker, which predicts the behavior of the target object (2) in the corresponding traffic scenario by adjusting the offline radar tracker with at least one external parameter to predict the behavior of the target object (2) for the corresponding traffic scenario. Step IV: Apply the smoothing algorithm to the predicted behavior of the target object (2) to further optimize the behavior of the target object (2).

2. The method according to claim 1, characterized in that, As an external parameter, it sets the object initialization or filter behavior.

3. The method according to claim 1 or 2, characterized in that, As object initialization, parameters that cannot be measured by radar in a single cycle are set, such as, in particular, the orientation, extended dimensions, velocity and / or acceleration of the target object (2).

4. The method according to any one of the preceding claims, characterized in that, As a smoothing algorithm, Rauch-Tung-Striebel smoothing and / or an improved Bryson-Frazier smoother and / or a minimum variance smoother and / or a dual-filter smoother are set.

5. The method according to any one of the preceding claims, characterized in that, The measurement points marked in step II are stored in such a way that the marked measurement points are unique for the corresponding traffic scenario.

6. The method according to any one of the preceding claims, characterized in that, The method also includes step V, in which the result of step IV is used as a reference benchmark for the online tracker.

7. An application that uses the behavior of a target object (2) as a reference benchmark for an online radar tracker of a radar sensor, wherein the behavior of the target object is determined according to the method of any one of the preceding claims.

Images