Method for predicting the behavior of a target object

Abstract

The present invention relates to a method for predicting the behavior of a target object (2) in a traffic scenario, which comprises the following method steps: Step I: Selecting a traffic scenario in which measuring points of a radar measurement of a radar sensor of a vehicle (1) were generated for detecting objects assigned to the measuring points in the surroundings of the vehicle (1), Step II: Labeling the measurement points, removing measurement points that are not assigned to a target object (2); Step III: Transmitting the labelled measurement points to an offline radar tracker, which predicts the behaviour of the target object (2) in the respective traffic scenario by adjusting the offline radar tracker by at least one external parameter in order to predict the behaviour of the target object (2) for the respective traffic scenario, Step IV: Applying a smoothing algorithm to the predicted behavior of the target object (2), which further optimizes the behavior of the target object (2).

Description

[0001] The present invention relates to a method for predicting the behavior of a target object in a traffic scenario and the use of such behavior as a reference for an online radar tracker of a radar sensor. Technological background

[0002] Modern means of transport, such as cars or motorcycles, are increasingly being equipped with driver assistance systems which, with the help of sensors, record the surroundings, recognise traffic situations and can support the driver, e.g. by braking or steering intervention or by issuing a visual or acoustic warning. Radar sensors, lidar sensors, camera sensors or similar are regularly used as sensor systems for environmental detection. Conclusions about the surroundings can then be drawn from the data acquired by the sensors. Environmental detection using radar sensors is based on the emission of bundled electromagnetic waves and their reflection, e.g. by other road users, obstacles on the road or the buildings along the edge of the road. The radar sensor generates measuring points or detections which can then be used, for example, tobe assigned to an object, which can then be tracked using a radar object tracker. Pedestrian detection is often done using camera sensors, but radar sensors are also increasingly being used.

[0003] For systems of the type described above, radar sensors can also be used in fusion with sensors using other technologies, such as camera or lidar sensors. Radar sensors have the advantage, among other things, that they operate reliably even in poor weather conditions and can measure not only the distance between objects but also their radial relative velocity using the Doppler effect. The transmission frequencies typically used are 24 GHz, 77 GHz, and 79 GHz. Due to the increasing functional scope of such systems, the requirements are constantly increasing, particularly with regard to the maximum detection range. In addition to environmental detection of motor vehicles for systems of the type described above, the focus is now also on interior monitoring of motor vehicles, e.g. to detect which seats are occupied; frequencies in the 60 GHz range, for example, are used for this purpose.

[0004] In this context, the present invention deals with the validation of the radar data generated by the radar sensor, in particular with the validation of the radar object tracker output of a (test) vehicle which has a radar sensor with a radar object tracker. This output is a tracked or tracked radar object which is identified or characterized by its kinematics (e.g. position, speed, acceleration or the like). According to the prior art, external sensors or reference systems such as differential GPS (Global Positioning System), LIDAR or camera sensors can also be used for this purpose. These external sensors provide the kinematics for the respective reference object, with which the radar tracker output can be compared in order to validate it and / or to carry out an accuracy estimate. For this purpose, the external sensors must also be arranged ormust be installed and properly calibrated, which can be extremely costly. Another disadvantage is that the external sensors have different measurement properties than the radar sensor, so additional post-processing is often required to finally compare an object from the radar object tracker with an object from an external reference sensor. Printed state of the art

[0005] US 2023 / 0258794 A1 discloses a method in which radar data is combined with previous sets of radar data to create a combined set of radar data. Each of these radar data sets and the combined radar data set may contain various data points, and each of these data points may be associated with certain specific features or classifications. The combined data set may then be preprocessed to identify distances to be associated with specific data points, locations to be associated with these data points, and speeds associated with the data points, so that each of the respective data points can be mapped and labeled with data identifying positions, speeds, and other physical details of the respective data points.This preprocessed data can then be processed through a machine learning process to identify kinematic information, which can then be made available to an AV's tracking system. The method is based on the approach of collecting radar data in an accumulative manner to facilitate association and create classifications as a precursor to a tracker. This is an online mechanism that does not focus on tracking / object creation, but rather pursues a deep learning approach, with no offline references generated. Object of the present invention

[0006] Based on the prior art, the object of the present invention is to provide a method for predicting the behavior of a target object, with which the computational or post-processing effort is reduced in a cost-effective manner. Solution to the task

[0007] The above object is achieved by the entire teaching of claim 1 and the subordinate claim. Advantageous embodiments of the invention are claimed in the subclaims.

[0008] According to the invention, the presented method for predicting the behavior of a target object (i.e., for example, the kinematic states or kinematics of the target object, which can be determined, for example, by its position and / or speed and / or acceleration and / or extension) in a traffic scenario comprises the following method steps: Step I: Selecting a traffic scenario in which measurement points of a radar measurement of a radar sensor of a vehicle were generated to detect objects in the surroundings of the vehicle assigned to the measurement points (i.e., for example, also data sets of traffic situations that were generated online), Step II: Labeling of the measurement points, whereby measurement points that are not assigned to a target object (such as measurement points of static objects, peripheral buildings, other road users, VRU, unwanted measurement points of the actual target object, such as outliers, etc., or the like) are removed; Step III: Transmitting the labelled measurement points to an offline radar tracker, which predicts the behavior of the target object in the respective traffic scenario (offline) by adjusting the offline radar tracker by at least one external parameter in order to predict the behavior of the target object for the respective traffic scenario, Step IV: Applying a smoothing algorithm to the predicted behavior of the target object, thereby further optimizing the behavior of the target object.

[0009] An object initialization, a situational object initialization, or a filter behavior can be used as an external parameter. Alternatively or additionally, detection-object association parameters or initial variances can also be used.

[0010] Preferably, parameters are provided for object initialization that cannot be measured by a radar in a single cycle, such as orientation, speed and acceleration of the target object.

[0011] Furthermore, a Rauch-Tung-Striebel smoothing and / or a modified Bryson-Frazier smoother and / or a minimum variance smoothing and / or a two-filter smoother can be provided as the smoothing algorithm.

[0012] It is advisable to store the measurement points marked in step II in such a way that they are unique for the respective traffic scenario.

[0013] Preferably, the method additionally comprises step V, after which the result from step IV is used as a reference for an online tracker.

[0014] Furthermore, the present invention also claims the use of a behavior of a target object, which was determined using a method according to the invention, as a reference for an online radar tracker of a radar sensor. Description of the invention based on exemplary embodiments

[0015] The invention is described in more detail below using practical examples. They show: Fig. 1 a simplified representation of an embodiment of a method sequence according to the invention; Fig. 2 a simplified schematic representation of a radar measurement of a radar sensor of a vehicle, as well as Fig. 3 a simplified schematic representation of the radar measurement from Fig. 2, in which the radar measurement was corrected according to the invention.

[0016] The present invention is particularly concerned with creating an "optimal" detection image by labeling or annotating individual cycles, whereby the effort required to develop a tracker is minimized, since no exceptions or special treatments need to be incorporated into the process algorithm, which is regularly necessary, for example, with an online tracker. The entire data association step is also eliminated, since all remaining detections are tracked. Furthermore, the data is preferably smoothed using various smoothing algorithms, but also by forward and backward traversing of the data (two-filter smoothing). The data is traversed forward once and tracked, as in an online system, with the end values of the last cycle serving as starting values for the backward traversal of the data. This can include, among other things,a reverse ego compensation takes place, with the final result being a weighted average of both track runs.

[0017] A key approach of the present invention is to generate a reference using the same radar data input ("radar-as-a-reference") as used by a radar tracker of a generic radar sensor, for example, operating in real time. By applying pre- and post-processing measures and by significantly relaxing the hardware restrictions (e.g., in memory and runtime), this so-called reference tracking system can deliver high-quality object kinematics without an external reference sensor. The invention is particularly concerned with the validation of the radar object tracker output. This output is typically a tracked radar object characterized by its kinematics (e.g., position, velocity, acceleration).Furthermore, there is no need to provide external reference systems, such as differential GPS, LIDAR, or camera sensors, which provide the reference object kinematics against which the radar tracker output can then be compared for validation and accuracy estimation. This eliminates the need to install and properly calibrate external sensors in a test vehicle, significantly reducing costs and manufacturing effort. Furthermore, post-processing effort can be reduced, as there is no need to adjust the measurement properties of different sensors.

[0018] The reference is generated using the same radar data input as used by a standard object tracker, which operates in real time and runs on the hardware of the actual radar sensor. By applying pre- and post-processing measures and by significantly relaxing the hardware constraints (memory, runtime), high-quality object kinematics can be provided without an external reference sensor. The generated reference can also be used for stationary objects, such as a stationary car or stationary environmental objects. Furthermore, multiple reference objects can be labeled simultaneously per traffic scene, such as several cars driving one behind the other that appear simultaneously in the sensor detection range and are of interest for a reference. This would then complicate data association but would still facilitate it in a practical way, since, for example,the association range can also be specified via an external parameter.

[0019] The process itself, or rather the method according to the invention, is an offline process that is applied after a measurement of a specific traffic scenario has been carried out and saved in a measurement file, which then serves as the starting point for the reference generation process. The method can therefore also be carried out as a computer-implemented method. The term "computer-implemented method" in the sense of the invention describes the process planning or procedure that is realized or carried out using the computer. The computer can process the data using programmable calculation rules. With regard to the method, essential properties can therefore also be subsequently implemented, e.g., through a new program, new programs, an algorithm, or the like.The computer can be designed as a control device, as an IC (Integrated Circuit) component, microcontroller, processor or system-on-chip (SoC) or the like.

[0020] The method according to the invention is preferably carried out using the Fig. 1 shown process steps are carried out: Selection of the scenario or traffic scenario (Step I) First, a traffic scenario is selected for which the reference is to be created. This can be any relevant situation, such as intersecting objects or road users, turning objects or road users, stationary objects, roundabouts, distant objects, VRU (vulnerable road users) such as pedestrians or bicycles, or the like. Fig. Figure 2 shows an example of a radar measurement from a radar sensor of a vehicle 1 – simplified measurement points are represented as stars – where, in addition to the target object 2, the measurements can be, for example, mirror measurements 3 (or mirror targets or "mirrorojects"), road boundaries (roadboarders), such as guardrails 4 or roadside buildings or vegetation, or other stationary objects 5. Furthermore, so-called outliers 6 can also be detected, which also belong to an object but are sorted out due to their location or positioning. Labeling of the traffic scenario (Step II) During labeling, all radar measurements that are not relevant to the object of interest or target object 2 in the selected traffic scenario are removed. This essentially involves the stationary environment, other road users, and detections of the selected object that are not usable for tracking, such as the detections of mirror measurements 3, guardrails 4, the other stationary object 5, and outlier measurements 6, which can distort the tracking process. These labeled detections are stored in a label file that is unique for the respective traffic scenario. Transmitting the radar measurements to a radar object tracker (Step III) After cleaning the radar image or radar measurement (see Fig.3) The relevant radar measurements are fed into a conventional (forward) radar object tracker, e.g., a control unit (ECU; Electronic Control Unit). This offline reference tracker can then be adjusted using external parameters to optimize its behavior for the respective traffic scenario. Parameters can include, for example, object initialization, filter behavior, or the like. Object initialization involves running an offline reference tracker to determine the object's movement or behavior, since online systems do not know the object's movement when it is first seen. By running an offline reference tracker, this behavior is known, and the object can be correctly adjusted in terms of movement orientation and kinematics (e.g., acceleration, velocity).Through the filter behavior of a tracking filter, for example, different types of traffic scenarios with different types of dynamic behavior can be adapted. In an online system, this would require some kind of scenario detection and response. However, in the present offline reference tracker, this traffic scenario is known, and the correct filter dynamics for the respective traffic scenario can be selected. These parameters are thus also stored for this traffic scenario, whereby the adaptation using external parameters minimizes the development effort for a tracker. The output can then be, for example, the kinematics of the tracked target object as implemented in the online system, but optimized for the given traffic scenario. Applying a smoothing algorithm (Step IV) By applying a smoothing algorithm to the output from step III, which preferably operates anticausally so that future knowledge about the target movement can be incorporated, the current behavior estimate can be improved. This post-processing step would not be available in an online system but can be applied in offline systems. After running the (forward) object tracker, the kinematic states and variances for the target object(s) are available. A smoothing algorithm is then applied to these results, preferably "walking backward through the data" and optimizing the states. For example, the smoothing algorithm can be an RTS (Rauch-Tung-Striebel) smoother (URTSS), a two-filter smoother, with the output of the smoother being the optimized kinematics of the object(s) of interest.In a preferred embodiment, the Rauch-Tung-Striebel (RTS) smoother is used, with an additional two-filter smoother being provided which runs forward and backward in time. Creating a reference format (Step V) The result of smoothing is the optimized object kinematics, which now serves as a reference for the online tracker. The online tracker can then use this reference as a target performance objective.

[0021] Furthermore, the reference format can be a suitable reference format for an existing black-box testing toolchain, so that this reference can be added accordingly (step VI), and the tracker can then repeatedly perform a so-called benchmark. Furthermore, the online tracker can check these references after each code change and determine whether there is an improvement or disadvantage compared to a previous run.

[0022] In practical terms, the reference tracker can be implemented arbitrarily. Since there are no runtime or memory constraints, real-time capability does not need to be considered. Possible implementations could be CTRA, CA, or IMM trackers, or other approaches such as particle filters. For example, it can be implemented as an adjustable Kalman filter with constant acceleration ("extended Kalman filter" using the CA model with the states x, VxAbs, AxAbs, y, VyAbs, AyAbs) or an IMM tracker with CTRA models (constant angular rate with acceleration), CA (constant acceleration), and CV models (constant velocity).

[0023] The adjustable parameters in the preferred implementation are object initialization properties, such as orientation, size, speed, and / or acceleration, which cannot be measured in a single radar cycle, and the filter dynamics, which define how quickly or slowly the filter reacts, for example, to a change in the direction of a target object. In general, all internal parameters of a tracker can be made adjustable for the respective traffic scenario using externally stored parameters. Possible parameters include, for example, detection-object association parameters or initial variances. Furthermore, all known fixed-interval smoothing algorithms can be used for smoothing, such as Rauch-Tung-Striebel smoothing, the modified Bryson-Frazier smoother, or minimum variance smoothing and / or the like.

[0024] The labeling process could also be performed manually, but this would be very time-consuming to label multiple radar detections per measurement cycle. In a preferred embodiment, the labeling process is partially automated, in particular up to approximately 90%, by marking the detections associated with the selected object(s) in an uncoordinated association step. Afterward, inspection and cleanup can also be performed manually to remove unwanted detections or add missed detections. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2023 / 0258794 A1

[0005]

Claims

[1] Method for predicting the behavior of a target object (2) in a traffic scenario, comprising the following method steps: Step I: Selecting a traffic scenario in which measuring points of a radar measurement of a radar sensor of a vehicle (1) were generated for detecting objects assigned to the measuring points in the surroundings of the vehicle (1), Step II: Labeling the measurement points, removing measurement points that are not assigned to a target object (2); Step III: Transmitting the labelled measurement points to an offline radar tracker, which predicts the behaviour of the target object (2) in the respective traffic scenario by adjusting the offline radar tracker by at least one external parameter in order to predict the behaviour of the target object (2) for the respective traffic scenario, Step IV: Applying a smoothing algorithm to the predicted behavior of the target object (2), which further optimizes the behavior of the target object (2). [2] Method according to claim 1, characterized by that an object initialization or a filter behavior is provided as an external parameter. [3] Method according to claim 1 or 2, characterized by that parameters are provided for object initialization which cannot be measured by a radar in a single cycle, such as in particular orientation, extent, speed and / or acceleration of the target object (2). [4] Method according to one of the preceding claims, characterized by that the smoothing algorithm used is a Rauch-Tung-Striebel smoothing and / or a modified Bryson-Frazier smoother and / or a minimum variance smoothing and / or a two-filter smoother. [5] Method according to one of the preceding claims, characterized bythat the measuring points marked in step II are stored in such a way that they are unique for the respective traffic scenario. [6] Method according to one of the preceding claims, characterized by that the method additionally comprises step V, after which the result from step IV is used as a reference for an online tracker. [7] Use of a behavior of a target object (2), which was determined using a method according to one of the preceding claims, as a reference for an online radar tracker of a radar sensor.

Images