Method for determining the state of an object in the environment of a vehicle

Abstract

The invention relates to a method (100) for determining the state of an object in the environment of a vehicle (1), comprising: - Providing (101) sensor data, wherein the sensor data result from a detection by at least one sensor (2), in particular a radar sensor, of the vehicle (1), wherein the object (3) is represented in the sensor data, - Determine (102) at least one characteristic of the state using a machine learning model (50) based on the provided sensor data, wherein the machine learning model (50) is trained for object detection and / or classification, - Determining (103) at least one further characteristic of the state using at least one physical model based on the provided sensor data. The invention further relates to a computer program, a device and a storage medium for this purpose.

Description

[0001] The invention relates to a method for determining the state of an object in the environment of a vehicle. The invention further relates to a computer program, a device, and a storage medium for this purpose. State of the art

[0002] Assisted and (highly) automated driving functions use a variety of sensor modalities to create a representation of the environment. This is referred to as perception. The environment here includes the static world (infrastructure, vegetation, etc.) as well as moving objects (vehicles, pedestrians, etc.).

[0003] To enable reliable detection, sensors such as radar sensors are used. These are particularly advantageous at night, in fog, spray, and rain. They emit an electromagnetic wave, which is reflected and received back by the sensor. This results in location data (reflections, locations) on both static objects and moving targets. The measurement space is primarily polar, and a location essentially comprises the distance, the radial relative velocity, the azimuth angle, and, in newer radar sensors, also the elevation angle. Additionally, the radar cross-section is estimated, which (in simplified terms) indicates the reflectivity of the target.

[0004] To track objects in the environment stably and without interruption, a stochastic observer, such as a Kalman filter or Bernoulli filter, is typically used. In this context, a distinction is made between two fundamental tasks. On the one hand, the initial conditions of an object must be estimated (object creation / spawning / track initiation), and on the other hand, object maintenance (track maintenance) must be performed, meaning the object must be preserved over time and the estimation error minimized. The requirements for these tasks differ considerably.

[0005] When creating objects for these tasks, the focus is primarily on avoiding false positives and false negatives, as well as estimating the initial state as accurately as possible. Since an object may be tracked over a longer time horizon, object maintenance focuses on correcting the currently estimated state. Disclosure of the invention

[0006] The invention relates to a method with the features of claim 1, a computer program with the features of claim 9, a device with the features of claim 10, and a computer-readable storage medium with the features of claim 11. Further features and details of the invention will become apparent from the respective dependent claims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program, the device, and the computer-readable storage medium according to the invention, and vice versa, so that mutual reference is always possible with regard to the disclosure of the invention.

[0007] The invention relates in particular to a method for determining the state of an object in the environment of a vehicle, comprising: - Providing sensor data, wherein the sensor data results from the acquisition of at least one sensor, in particular a radar sensor, of the vehicle, wherein the object is represented in the sensor data, wherein, as an alternative to the radar sensor, a camera, ultrasonic or LiDAR sensor is also conceivable, - Determining at least one characteristic of the state using a machine learning model, in particular a (deep) neural network, based on the provided sensor data, wherein the machine learning model is trained for object detection and / or classification, - Determine at least one further characteristic of the state using at least one physical model based on the provided sensor data.

[0008] The at least one physical model can be, for example, a kinematic model in which motion is determined based on velocity, acceleration, and / or changes in direction. This also includes models that, for example, make advantageous use of a rigid body property of an object. Furthermore, the at least one physical model can be a dynamic model that also considers forces, friction, or mass to provide more complex descriptions of motion. The at least one physical model can also be a radar- or lidar-based model in which signal propagation is used to estimate distance and velocity. Finally, the at least one physical model can be an optical flow model in which motion is determined based on the apparent flow of objects in a scene.It is also conceivable that at least one physical model is a vehicle dynamics model that models the vehicle's own behavior, e.g. when cornering or on a wet road surface.

[0009] It is therefore possible to determine a more comprehensive and accurate state, in particular a state vector, of an object by combining sensor data with the output of the machine learning model and at least one physical model. This can improve the accuracy of object detection. Furthermore, the integration of various characteristics can enable a more detailed description of the object and its environment.

[0010] At least one characteristic can be, for example, a class (e.g., "passenger car," "motorcycle," or "pedestrian"), a Cartesian position, an orientation, a Cartesian velocity, a speed, a yaw rate, and / or an extent of the object. At least one or all of these quantities or characteristics can be given in or derived from a vehicle coordinate system, e.g., with respect to the Cartesian position or the Cartesian velocity. The state is preferably represented by a state vector that includes, in particular, the various characteristics.

[0011] This offers the advantage of a comprehensive and detailed description of the object's state. Providing various characteristics such as class, position, orientation, velocity, and extent enables a precise understanding of the object within the vehicle's environment. Furthermore, using a state vector to represent the state allows for efficient processing and integration of the information into information technology systems such as the vehicle's tracking system.

[0012] Furthermore, it is optionally provided that, as part of the detection process using the machine learning model, at least the object's class, Cartesian position, orientation, and extent are determined and provided in the form of a bounding box, specifically an oriented bounding box, i.e., a bounding box where an orientation is also available. The use of the machine learning model enables particularly precise object recognition with regard to the object's class, position, orientation, and extent. The bounding box can clearly display this information and serve as the basis for further analysis and control functions within the vehicle. A bounding box is, in particular, a rectangular frame that can be drawn around a detected object in an image or video, especially in radar data.This marks, in particular, the coordinates of an area in which the object is located and can be used to distinguish the object from its surroundings.

[0013] Furthermore, within the scope of the invention, it is optionally possible to determine at least the Cartesian velocity and / or the velocity of the object using the at least one physical model, preferably based on the provided bounding box. The use of a physical model allows for a particularly precise determination of the object's motion. For this purpose, for example, different positions of the bounding box and corresponding timestamps for each position can be used to determine the Cartesian velocity and / or the velocity of the object.

[0014] Furthermore, within the scope of the invention, it is conceivable that the sensor data includes timestamps representing a point in time when the respective sensor data was acquired. The determination of the time is carried out using the machine learning model and / or the determination using the at least one physical model, taking the timestamps into account. In this way, the temporal sequence of the sensor data can be considered, and older sensor data can also be accessed when determining the state. The combination of sensor data with timestamps enables, in particular, a more precise analysis of the object's state, since changes over time can be better captured. This is especially relevant for dynamic environments in which the properties of an object can change continuously.

[0015] Furthermore, it is conceivable that the procedure may also include: - Performing object tracking of the object using the determined state of the object, wherein, within the scope of object tracking, the state of the object is continuously updated using the machine learning model and / or the at least one physical model, in particular as long as the object tracking (for the object) is active, for example at regular intervals or triggered by certain sensor values ​​in the sensor data or triggered by the vehicle, for example at a start or during a specific driving maneuver.

[0016] In this way, it is possible to track the object's position and movement over time, and preferably also its size and class. The continuous updating of the state using the machine learning model and / or the physical model enables, in particular, a precise representation of the object's movement.

[0017] Advantageously, the invention may provide that the method further comprises: - Performing a maturity assessment based on the sensor data and / or the at least one characteristic determined using the machine learning model, wherein, within the scope of the maturity assessment, the sensor data and / or the at least one characteristic determined using the machine learning model are checked with regard to at least one defined quality criterion, Object tracking is only performed if a maturity check result indicates that at least one defined quality criterion is met. For example, the defined quality criterion might stipulate that sufficient sensor data specific to the presence of the object must be available. Here, for instance, several time periods can be examined, and if the object is represented in the sensor data in a defined number of measurements or for a defined time period, the defined quality criterion can be considered fulfilled. Furthermore, the defined quality criterion might stipulate that a bounding box for the object exists, meaning it was successfully created by the machine learning model. The maturity check ensures, in particular, that object tracking is only performed for valid objects.Object candidates, is performed, which can reduce the number of false positives and / or false negatives.

[0018] According to an advantageous embodiment of the invention, the method may further comprise: - Initiating vehicle control based on the determined state of the object.

[0019] For example, a braking maneuver could be triggered if an object is located in close proximity to the vehicle and is not moving in the same direction, thus indicating a likely collision. Furthermore, a distance control system could be implemented based on the detected situation, maintaining a safe distance from a vehicle ahead.

[0020] It is possible that the method according to the invention is used in a vehicle. The vehicle can be, for example, a motor vehicle and / or passenger vehicle and / or autonomous vehicle. The vehicle can have vehicle equipment, for example, for providing an autonomous driving function and / or a driver assistance system. The vehicle equipment can be designed to control the vehicle at least partially automatically and / or accelerate and / or brake and / or steer.

[0021] The machine learning model is or was specifically trained for classification and / or object detection. Accordingly, the training can result in a trained machine learning model that can be used for classification and / or object detection. Its use, and thus its inference capabilities, can be implemented, for example, in a vehicle. The input data points can be, for example, pixels from image data or be based on them, in order to perform the classification and / or object detection of the data points based on the pixels. The input data can include sensor and / or image data, which at least partially results from acquisition with a sensor, preferably a camera sensor or radar sensor, and / or which has been at least partially synthesized, thus specifically replicating the real-world data of a sensor.Specifically, the values ​​of image points, preferably pixels, in the image data can represent the environment of a sensor and / or a vehicle and / or a traffic scene. A classification, preferably image classification and / or object detection, based on these values ​​can be provided. This enables, for example, the detection of objects within the traffic scene. The classification can also take the form of semantic segmentation (i.e., pixel- or area-based classification) and / or object detection. The image data can be, for example, images or point clouds from a radar sensor and / or an ultrasonic sensor and / or a LiDAR sensor and / or a thermal imaging camera. Accordingly, the images can also be presented as radar images and / or ultrasonic images and / or thermal images and / or LiDAR images.

[0022] The invention also relates to a computer program, in particular a computer program product, comprising instructions which, when executed by a computer, cause the computer to execute the method according to the invention. Thus, the computer program according to the invention offers the same advantages as those described in detail with reference to a method according to the invention.

[0023] The invention also relates to a data processing device configured to execute the method according to the invention. The device can, for example, be a computer that executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. Alternatively, a non-volatile data storage device can be provided in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0024] The invention may also relate to a computer-readable storage medium which contains the computer program according to the invention and / or includes instructions which, when executed by a computer, cause the computer to execute the method according to the invention. The storage medium is, for example, designed as a data storage device such as a hard drive and / or non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0025] Furthermore, the method according to the invention can also be implemented as a computer-implemented method. Alternatively or additionally, at least one of the disclosed method steps can be computer-implemented and / or carried out automatically.

[0026] Further advantages, features, and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can each be essential to the invention individually or in any combination.

[0027] They show: Fig. 1 a schematic visualization of a method, a machine learning model, a device, a storage medium and a computer program according to exemplary embodiments of the invention, Fig. 2 a schematic visualization of a vehicle with a sensor and an object according to exemplary embodiments of the invention.

[0028] In Fig. Figure 1 shows a method 100, a machine learning model 50, a device 10, a storage medium 15 and a computer program 20 according to exemplary embodiments of the invention.

[0029] In Fig. Figure 2 shows a vehicle 1 with a sensor 2 and an object 3 according to exemplary embodiments of the invention schematically.

[0030] Fig.Figure 1 shows in particular an embodiment of a method 100 for determining the state of an object in the environment of a vehicle 1. In a first method step 101, sensor data are provided, wherein the sensor data result from the acquisition of at least one sensor 2, in particular a radar sensor, of the vehicle 1, and the object 3 is represented in the sensor data. In a second step 102, at least one characteristic of the state is determined using a machine learning model 50 based on the provided sensor data, wherein the machine learning model 50 is trained for object detection and / or classification. In a third step 103, at least one further characteristic of the state is determined using at least one physical model based on the provided sensor data.

[0031] According to exemplary embodiments of the present invention, a concept for object creation is provided. In particular, results of a machine learning model 50, especially a deep neural network (DNN), are combined with model-based methods, i.e., especially physical models.

[0032] This allows the following problems to be addressed and solved to their advantage: A more robust maturity level check of the object candidate can be provided, thus reducing false positives and false negatives. Furthermore, faster object creation can be achieved. In addition, the state, especially the initial state, of objects can be determined. Object candidates can be identified more effectively, particularly with regard to an estimation of their extent and / or a classification of the object candidate.

[0033] According to exemplary embodiments, the strengths of machine learning models 50 such as DNNs and model-based approaches are combined using physical models for object creation, whereby the current weaknesses of the individual approaches can be compensated for.

[0034] The advantage of the invention lies, for example, in the fact that the detection of the machine learning model 50 can be considered a method for estimating at least part of the state of an object 3, or initial conditions of an object candidate for object tracking. Furthermore, this results in a particularly independent detection path to ensure that the object 3 to be created is not a ghost object (false positive). Additionally, the time between a first detection and object creation can be reduced. A major advantage of this is, for example, that the time from the first detection of a (radar) sensor 2 until the driving function reacts can also be significantly reduced. This is particularly relevant for controlling the vehicle 1, especially in the context of safety-relevant driver assistance functions (e.g.,AEB) of vehicle 1 is important, as even a few milliseconds can decide whether a full braking maneuver can be initiated in time.

[0035] The following describes an example of a software architecture for model-based object creation. If a (radar) sensor 2 detects a potential object 3, an object hypothesis can be generated. Within the scope of the present invention, this object hypothesis can also be referred to as an object candidate. For this object candidate, suitable sensor data, particularly radar data, can be collected from all sensors, especially radar sensors, in subsequent cycles and compiled into so-called location sets. A location set preferably comprises only sensor data, or radar data, from one sensor 2 at a given timestamp. Over time, several location sets are preferably collected for the object candidate, potentially from different sensors.

[0036] This sensor data can now be used to determine the initial conditions of the object candidate. Various strategies with diverse tactics can be employed for this purpose. If one of the strategies is successful and has been able to estimate the initial conditions of a state vector of the object candidate, object 3 is preferably created after a successful maturity test and transferred to a track cycle and object maintenance.

[0037] In contrast to classical tracking of objects, the (initial) state vector is preferably not determined recursively. Instead, according to exemplary embodiments of the invention, all sensor data or radar data are stored in the location sets as a batch. Thus, all sensor data are preferably available with corresponding timestamps. One advantage of this approach is, for example, that different strategies and tactics can now be used in each cycle to estimate the probability of existence as well as the initial state of the state vector.

[0038] A generalized state vector can be used for an initial estimate of the initial conditions. This is preferably not a minimal representation and not a state vector in the control engineering sense, as the latter can contain redundant states. x=[pxpyψvxvyvωLW] px, py are in particular a Cartesian position of object 3, or object candidates, ψ an orientation, v x , v y A Cartesian ground velocity, v a velocity, ω a yaw rate, and L, W a dimension. All quantities can be given in a vehicle coordinate system of vehicle 1.

[0039] In particular, different strategies are employed to achieve the most accurate possible estimate of the state, or initial conditions, of covariance matrices and a dimension. Each of these strategies encompasses different tactics. The strategy preferably terminates once all entries of the generalized state have been successfully estimated. If, however, the computation fails, another strategy can be used.

[0040] One tactic in particular is an algorithm that estimates a part of the state vector, the covariance matrices and / or the dimension of object 3, or object candidates for object tracking.

[0041] If a strategy was successful and the generalized state vector could be fully estimated, a maturity check is preferably performed before the object candidate is passed to the tracking system for object maintenance. Various heuristic criteria can be used for this purpose, such as the number of location sets and the total number of locations. The goal of this maturity check is, for example, to prevent the creation of false positives in the tracking loop.

[0042] In addition to the generalized state vector and the variance, object creation can also determine an object class. However, current algorithms do not offer satisfactory performance, which is why every object created during object creation is generally classified as an "obstacle".

[0043] The machine learning model 50 can be used to identify oriented boxes, especially oriented bounding boxes, obb = [px, py, ψ, L, W] T with an additional class label. These can be used directly to populate the generalized state vector. Over, for example, two measurement cycles, these bounding boxes can then be collected similarly to location sets. Using this time-based collection, it is then possible to calculate the velocities.

[0044] Within the framework of a software architecture for hybrid object creation, an association is established between the bounding boxes and objects 3, or object candidates. If the machine learning model 50 provides a suitable (spatial) bounding box for the object candidate, this can be linked and managed as a bounding box alongside the location sets. The bounding boxes can be considered as measurements from a virtual (radar) sensor 2.

[0045] Furthermore, a new strategy can be introduced according to the exemplary implementations. In particular, the performance of the extension and classification of the machine learning model 50 surpasses the model-based tactics. The goal can now be to design and implement new tactics with the bounding boxes to achieve a more accurate estimation of the initial state.

[0046] As part of a tactic, an extent estimation can be performed. In this tactic, the extent estimate of the bounding boxes can be incorporated into the generalized state vector of the object candidate.

[0047] Another tactic involves classification. The class label of the bounding box can then be directly applied to the object candidate.

[0048] Furthermore, other corresponding tactics are also conceivable that fill in the remaining quantities of the state vector.

[0049] Furthermore, a new maturity assessment can be performed. Since the machine learning model 50 can be considered a virtual (radar) sensor 2 and is stored alongside the location sets for an object candidate, it can also be used to calculate a more robust maturity assessment. The previous model-based approach relied primarily on the attributes of the locations collected for an object candidate. However, clutter, angular ambiguities, and other effects of sensor 2 can lead to false positives. Because the machine learning model 50 has learned many of these effects, it is now possible to incorporate this advantage into the maturity assessment. Thus, before track initiation, it can be checked whether at least one or more bounding boxes could be associated with an object candidate. This reduces the number of false positive tracks.

[0050] Another advantage is that not only can the false positive rate be reduced, but the object candidate can also be transferred to object tracking or track maintenance much faster if the above maturity level check was successful.

[0051] The preceding explanation of the embodiments describes the present invention solely by way of examples. Naturally, individual features of the embodiments can be freely combined with one another, provided this is technically feasible, without departing from the scope of the present invention.

Claims

[1] Method (100) for determining the state of an object (3) in the environment of a vehicle (1), comprising: - Providing (101) sensor data, wherein the sensor data result from a detection by at least one sensor (2), in particular a radar sensor, of the vehicle (1), wherein the object (3) is represented in the sensor data, - Determine (102) at least one characteristic of the state using a machine learning model (50) based on the provided sensor data, wherein the machine learning model (50) is trained for object detection and / or classification, - Determine (103) at least one further characteristic of the state using at least one physical model based on the sensor data provided. [2] Method (100) according to claim 1, characterized by, that at least one characteristic is a class, a Cartesian position, an orientation, a Cartesian velocity, a velocity, a yaw rate and / or an extent of the object (3) and the state is preferably represented by a state vector. [3] Method (100) according to claim 2, characterized by , that in the context of the determination (102) using the machine learning model (50) at least the class, the Cartesian position, the orientation and the extent of the object (3) is determined and provided in the form of a bounding box. [4] Method (100) according to one of claims 2 or 3, characterized by , that in the course of determining (103) using the at least one physical model at least the Cartesian velocity and / or the velocity of the object (3) is determined, preferably on the basis of the provided bounding box. [5] Method (100) according to any one of the preceding claims, characterized by , that the sensor data include timestamps representing a time of acquisition of the respective sensor data, wherein the determination (102) is carried out using the machine learning model (50) and / or the determination (103) is carried out using the at least one physical model taking the timestamps into account. [6] Method (100) according to any one of the preceding claims, characterized by , that the procedure (100) further includes: - Performing object tracking of the object (3) using the determined state of the object (3), wherein, as part of the object tracking, the state of the object (3) is continuously updated using the machine learning model (50) and / or the at least one physical model. [7] Method (100) according to claim 6, characterized by , that the procedure (100) further includes: - Performing a maturity level check based on the sensor data and / or the at least one characteristic determined using the machine learning model, wherein, within the scope of the maturity level check, the sensor data and / or the at least one characteristic determined using the machine learning model are checked with regard to at least one defined quality criterion, wherein object tracking is only carried out if a result of the maturity level check indicates that the at least one defined quality criterion is met. [8] Method (100) according to any one of the preceding claims, characterized by , that the procedure (100) further includes: - Initiating control of the vehicle (1) based on the determined state of the object (3). [9] Computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause it to execute the method (100) according to any of the preceding claims. [10] Device (10) for data processing which is configured to carry out the method (100) according to any one of claims 1 to 8. [11] Computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause it to perform the steps of the method (100) according to any one of claims 1 to 8.

Images