Method, computer program, device and memory medium for ascertaining state of object in vehicle environment

Abstract

The invention relates to a method (100) for ascertaining a state of an object (3) in the environment of a vehicle (1), comprising:-providing (101) sensor data, the sensor data being generated from the detection of at least one sensor (2), in particular a radar sensor, of the vehicle (1), the object (3) being represented in the sensor data, -ascertaining (102) at least one characterization of the state on the basis of the provided sensor data using a machine learning model (50), the machine learning model (50) being trained for object detection and / or classification,-ascertaining (102) at least one characterization of the state on the basis of the provided sensor data using at least one physical model, at least one further characterization of the state is ascertained (103). The invention further relates to a computer program, a device and a memory medium for this purpose.

Description

Technical Field

[0001] This invention relates to a method for determining the state of an object in a vehicle environment. Furthermore, the invention also relates to a computer program, an apparatus, and a memory medium for this purpose. Background Technology

[0002] Assisted and (highly) automated driving functions utilize multiple sensor modalities to create images of the environment. This is called perception. Here, the environment includes the static world (infrastructure, vegetation, etc.) and moving objects (vehicles, pedestrians, etc.).

[0003] To enable reliable sensing, sensors, such as radar sensors, are employed. Advantageously, these sensors can also be used, especially at night, in fog, in water droplets, and in rain. They emit electromagnetic waves that are reflected and then received by the sensor again. This results in localization (reflection, locations) on both static objects and moving targets. Here, the measurement space is particularly polar space, and localization essentially includes distance, radial relative velocity, and azimuth, and for newer radar sensors, elevation. Additionally, the radar cross-section (simplified) indicating the reflectivity of the target is estimated.

[0004] To enable stable and uninterrupted tracking of objects in an environment, random observers, such as Kalman filters or Bernoulli filters, are typically employed. In this regard, a distinction is made between two fundamental tasks. On the one hand, the initial conditions of the object must be estimated (object creation / spawning / track-initialization), and on the other hand, object maintenance must be performed (track-maintenance), i.e., the object must be retained over time while minimizing the estimation error. In some cases, the requirements for these tasks are clearly distinct.

[0005] When creating objects for these tasks, the focus is primarily on avoiding false positive and false negative tracking, and estimating the initial state as accurately as possible. Since objects may be tracked over long periods, object maintenance focuses on correcting for the currently estimated state. Summary of the Invention

[0006] The subject of this invention is a method according to the invention, a computer program according to the invention, an apparatus according to the invention, and a machine-readable memory medium according to the invention. Further features and details of the invention will become apparent from the description and drawings. Hereinafter, the features and details described in conjunction with the method according to the invention also apply to the computer program according to the invention, the apparatus according to the invention, and the computer-readable memory medium according to the invention, and vice versa, so that they can always be referenced to each other in relation to the content of this invention.

[0007] The subject of this invention is, in particular, a method for determining the state of an object in a vehicle environment, the method comprising: - Provide sensor data, wherein the sensor data is generated from the detection of at least one sensor of the vehicle, particularly a radar sensor, wherein the object is represented in the sensor data, wherein, instead of a radar sensor, a camera sensor, an ultrasonic sensor or a lidar sensor may also be conceived. - Based on the provided sensor data, at least one representation of the state is obtained using a machine learning model, particularly a (deep) neural network, wherein the machine learning model is trained for object detection and / or classification. - Based on the provided sensor data, and using at least one physical model, obtain at least one additional characterization of the state.

[0008] The at least one physical model can be, for example, a kinematic model in which motion is determined based on changes in velocity, acceleration, and / or direction. It also includes models that, for example, advantageously utilize the rigid body properties of an object. Furthermore, the at least one physical model can be a dynamic model that also considers forces, friction, or mass to provide a more complex description of motion. The at least one physical model can also be a radar-based or lidar-based model in which signal propagation is used for distance and velocity estimation. Additionally, the at least one physical model can be an optical flow model in which motion is determined based on the apparent optical flow of objects in the scene. Similarly, it is conceivable that the at least one physical model is a driving dynamics model that models the vehicle's own behavior, such as when turning or when the roadway is wet.

[0009] Therefore, by combining sensor data with the output of a machine learning model and at least one physical model, a more comprehensive and accurate state of an object, especially its state vector, can be obtained. This improves the accuracy of object detection. Furthermore, integrating different representations allows for a more detailed description of the object and its environment.

[0010] The at least one representation may be, for example, the category of the object (e.g., "passenger vehicle," "motorcycle," or "pedestrian"), Cartesian position, orientation, Cartesian velocity, speed, yaw rate, and / or extended dimensions. At least one or all of these parameters, or representations, may be given or derived from the vehicle's vehicle coordinate system, for example, in terms of Cartesian position or Cartesian velocity. The state is preferably represented by a state vector, which in particular includes the different representations.

[0011] This offers the following advantages: It enables a comprehensive and detailed description of an object's state. It provides different representations, such as category, location, orientation, velocity, and extended dimensions, allowing for an accurate understanding of objects within the vehicle's environment. Furthermore, using state vectors to represent states allows for efficient processing and integration of information in information technology applications, such as in vehicle tracking systems.

[0012] Furthermore, optionally, within the scope of the calculation using a machine learning model, at least the object's category, Cartesian position, orientation, and extended dimensions should be determined and provided in the form of a bounding box, particularly an oriented bounding box, where the orientation can also be provided within the bounding box. Using a machine learning model, object recognition can be performed with particularly high precision in terms of object category, position, orientation, and extended dimensions. The bounding box clearly displays this information and can serve as the basis for subsequent analysis and control functions in the vehicle. The bounding box is, in particular, a rectangular box that can be drawn around the identified object in an image or video, especially in radar data. This bounding box specifically marks the coordinates of the area where the object is located and can be used to demarcate the object from its environment.

[0013] Furthermore, optionally, within the scope of the invention, it is possible to: within the range of calculations performed using at least one physical model, at least the Cartesian velocity and / or velocity of the object be calculated, preferably based on the provided bounding box. The motion of the object can be determined with particular accuracy by using a physical model. For this purpose, for example, the Cartesian velocity and / or velocity of the object can be determined using different positions of the bounding box and corresponding timestamps of those positions.

[0014] Furthermore, within the scope of this invention, it is conceivable that sensor data includes timestamps representing the points in time when the corresponding sensor data was detected, wherein the determination is performed considering the timestamps when using a machine learning model and / or when using at least one physical model. In this way, the time series of sensor data can be considered, and earlier sensor data can be utilized when determining the state. The combination of sensor data and timestamps particularly enables more accurate analysis of the object's state because changes over time can be better detected. This is especially relevant for dynamic environments where the characteristics of an object may continuously change.

[0015] Furthermore, it can be envisioned that this method also includes: - Perform object tracking on the object using the obtained object state, wherein, within the scope of the object tracking, the state of the object is continuously updated using the machine learning model and / or the at least one physical model, in particular as long as the object tracking is active, for example, updating at regular intervals or triggered by determined sensor values ​​in sensor data or by vehicle-triggered updates, such as at startup or when performing a specific driving maneuver.

[0016] In this way, the position and movement of an object can be tracked over time, and preferably, the object's extended dimensions and category can also be tracked. Continuously updating the state using machine learning models and / or physical models makes it particularly possible to accurately represent object movement.

[0017] Advantageously, within the scope of the invention, the method may further include: - A maturity test is performed based on the sensor data and / or based on at least one representation obtained using the machine learning model, wherein, within the scope of the maturity test, the sensor data and / or at least one representation obtained using the machine learning model are examined in terms of at least one defined quality criterion. Specifically, object tracking is performed only if the result of the maturity test indicates that "at least one of the defined quality criteria is met".

[0018] For example, the defined quality criterion can be set as follows: there is sufficient sensor data specifically for the existence of the object. Here, for example, multiple time periods can be investigated, and the defined quality criterion can be met if the object is represented in the sensor data in a defined number of measurements or within a defined time interval. Furthermore, the defined quality criterion can also specify that a bounding box exists for the object, i.e., the bounding box can be successfully created by the machine learning model. This is particularly achieved through the maturity test: object tracking is performed only on valid objects, or object candidates, thereby reducing the number of false positives and / or false negatives.

[0019] According to an advantageous improvement of the present invention, the method may further include: - Control of the vehicle is initiated based on the state obtained from the object.

[0020] For example, if an object is located very close to the vehicle in front and is not moving in the same direction, thus making a collision assessment possible, then braking maneuvers can be triggered. Furthermore, distance adjustments relative to the vehicle ahead can be performed based on the obtained state.

[0021] The method according to the invention can be used in vehicles. Vehicles can be configured, for example, as motor vehicles and / or passenger vehicles and / or autonomous vehicles. Vehicles may have, for example, vehicle facilities for providing autonomous driving functions and / or driver assistance systems. These vehicle facilities can be implemented for at least partially automatic control and / or acceleration and / or braking and / or steering of the vehicle.

[0022] Machine learning models are specifically trained or have been trained for classification and / or object detection. Accordingly, trained machine learning models can be generated from training, which can be used for classification and / or object detection. For example, the use can be set up in a vehicle, and thus the inference can be set up. The data points of the input data can be, for example, pixels of image data or based on image data, so that classification and / or object detection can be performed on the data points based on the pixels. The input data can include sensor data and / or image data, which are generated at least in part from detections performed by sensors, preferably camera sensors or radar sensors, and / or these data are at least in part synthesized, thus particularly mimicking real sensor data. Specifically, the environment of the sensor and / or vehicle and / or traffic scene can be represented by the values ​​of image points (preferably pixels) of the image data. Classification, preferably image classification and / or object detection can be set based on these values. This enables, for example, the detection of objects in the traffic scene. Here, the classification can also be set in the form of semantic segmentation (i.e., classification by pixels or by regions) and / or object detection. Image data can be, for example, images or point clouds from radar sensors and / or ultrasonic sensors and / or lidar sensors and / or thermal imaging devices. Accordingly, images can also be implemented as radar images and / or ultrasonic images and / or thermal images and / or lidar images.

[0023] Similarly, the subject of this invention is also a computer program, and more particularly a computer program product, comprising instructions that, when implemented by a computer, cause the computer to perform the method according to the invention. Therefore, the computer program according to the invention provides the same advantages as those described in detail with respect to the method according to the invention.

[0024] Similarly, the subject of this invention is also an apparatus for data processing, configured to implement the method according to the invention. For example, a computer implementing a computer program according to the invention can be configured as the apparatus. The computer may have at least one processor for implementing the computer program. A non-volatile data memory may also be provided, in which the computer program can be stored and can be read from the memory by the processor for implementation.

[0025] Similarly, the subject matter of the invention can also be a computer-readable storage medium having a computer program according to the invention and / or including instructions that, when executed by a computer, cause the computer to perform the method according to the invention. This storage medium is, for example, constructed as a data storage device, such as a hard disk and / or non-volatile memory and / or a memory card. This storage medium can, for example, be integrated into the computer.

[0026] Furthermore, the method according to the invention can also be implemented as a computer-based method. Alternatively or additionally, at least one of the disclosed method steps can be computer-based and / or executed automatically.

[0027] Further advantages, features, and details of the invention will become apparent from the following description, in which embodiments of the invention will be described in detail with reference to the accompanying drawings. Here, the features mentioned in the application documents may be substantially significant to the invention individually or in any combination. Attached Figure Description

[0028] The attached diagram shows: Figure 1 Schematic visualizations of methods, machine learning models, apparatuses, memory media, and computer programs according to embodiments of the present invention. Figure 2 A schematic visualization of a vehicle and object equipped with sensors according to an embodiment of the present invention. Detailed Implementation

[0029] exist Figure 1 The diagram schematically illustrates a method 100, a machine learning model 50, an apparatus 10, a memory medium 15, and a computer program 20 according to embodiments of the present invention.

[0030] exist Figure 2 The image schematically illustrates a vehicle 1 equipped with sensor 2 and an object 3 according to an embodiment of the present invention.

[0031] Figure 1 In particular, an embodiment of a method 100 for determining the state of an object in the environment of a vehicle 1 is shown. In a first method step 101, sensor data is provided, wherein the sensor data is generated from detections by at least one sensor 2 of the vehicle 1, particularly a radar sensor, wherein the object 3 is represented in the sensor data. In a second step 102, based on the provided sensor data, at least one representation of the state is determined using a machine learning model 50, wherein the machine learning model 50 is trained for object detection and / or classification. In a third step 103, based on the provided sensor data, at least one additional representation of the state is determined using at least one physical model.

[0032] According to an embodiment of the present invention, a scheme for object creation is provided. Here, in particular, the results of machine learning model 50, especially the results of deep neural network (DNN), are combined with model-based methods, that is, especially with physical models.

[0033] This allows for more robust maturity testing of object candidates, thereby reducing false positives and false negatives. Furthermore, it enables faster object formation. Additionally, it allows for improved determination of the state of objects or object candidates, especially the initial state, particularly in extended size estimation and / or object candidate classification.

[0034] According to the embodiments, the strengths of machine learning model 50, such as the probing strengths of DNN, are combined with the strengths of model-based methods when using physical models for object creation, which can compensate for the current shortcomings of individual solutions.

[0035] The advantages of this invention include, for example, that the detection of the machine learning model 50 can be used as a means to estimate at least a portion of the state of object 3 or to estimate the initial conditions of object candidates for object tracking. Furthermore, this specifically yields independent detection paths to ensure that the object 3 to be created is not a false object (false positive). Additionally, the time between the initial detection and object creation can be reduced. A significant advantage in this regard is that the time from the initial detection of the (radar) sensor 2 to the response of the driving function can also be significantly reduced. This is important, for example, for the control of vehicle 1, especially within the scope of safety-related driver assistance functions (e.g., AEB) of vehicle 1, where milliseconds can determine whether full braking can be triggered in a timely manner.

[0036] The software architecture for model-based object creation is described below as an example. If (radar) sensor 2 detects a potential object 3, an object hypothesis can be created. Within the scope of this invention, this object hypothesis may also be referred to as an object candidate. In subsequent cycles, matching sensor data from all sensors, particularly radar data, can be collected for this object candidate and aggregated into a so-called location set. Here, a location set preferably includes only sensor data or radar data from one sensor 2 at a single timestamp. Preferably, more location sets are collected for the object candidate over time, and these location sets may also come from different sensors.

[0037] At this point, the sensor data can be used to determine the initial conditions for the object candidates. Different strategies with various methods can be employed for this purpose. If one of these strategies is successful and can estimate the initial conditions of the object candidate's state vector, then object 3 is preferably created after successfully completing the maturity check, and the object is then delivered to the tracking cycle and object maintenance, or object tracking.

[0038] Unlike classic object tracking, the (initial) state vector is preferably not determined recursively, but rather, according to embodiments of the invention, all sensor or radar data in the location set is stored as a batch. Therefore, all sensor data is preferably available with accompanying timestamps. The advantage of this approach is, for example, that different strategies and methods can be used to estimate the existence probability and the initial state of the state vector in each cycle.

[0039] An initial estimate of the initial conditions can be obtained using a generalized state vector. This generalized state vector is preferably not a minimal representation, and is not a state vector in the sense of a control technique, as it may contain redundant states.

[0040] , Especially the Cartesian positions of object 3 or object candidates. It's the direction. , It concerns the Cartesian velocity of the ground. It's speed. Yaw rate and , This refers to the extended dimensions. All parameters can be given in the vehicle coordinate system of vehicle 1.

[0041] Different strategies are employed to achieve the most accurate estimations possible of the state or initial conditions, the covariance matrix, and the dimensions. Each of these strategies includes different means. A strategy preferably terminates once all terms of the generalized state have been successfully estimated. Conversely, if computation fails, other strategies can be utilized.

[0042] One approach is particularly an algorithm that estimates a portion of the state vector of an object 3 or an object candidate for object tracking, including its covariance matrix and / or dimension.

[0043] If a strategy has been successful in the past and the generalized state vector can be fully estimated, it is preferable to perform a maturity check before delivering object candidates to a tracking system for object maintenance. For this purpose, different heuristics can be used, such as the size of the location set and the total number of locations. The goal of this maturity check is, for example, to avoid creating false positive objects in the tracking loop.

[0044] In addition to generalized state vectors and variances, object creation can also determine object categories. However, current algorithms, in particular, do not provide satisfactory performance; therefore, each object 3 from object creation is generally classified as an "obstacle."

[0045] Machine learning model 50 can be used to obtain oriented bounding boxes with additional class labels, especially directed bounding boxes. These bounding boxes can be directly used to fill the generalized state vector. These bounding boxes can then be collected, for example, after two measurement cycles in a manner similar to the localization set. The velocity can then be calculated using this temporal collection.

[0046] Within the scope of the software architecture for hybrid object creation, the association from bounding boxes to objects 3 or object candidates is particularly important. If the machine learning model 50 provides matching (spatial) bounding boxes for use with object candidates, these matching bounding boxes can be associated, and managed as bounding boxes in addition to the localization set. Here, the bounding box can be viewed as a measurement by a virtual (radar) sensor 2.

[0047] Furthermore, according to the embodiments, new strategies can be introduced. Machine learning model 50 performs particularly well over model-based methods in terms of extended dimensions and classification. In this case, the goal could be to design and employ new methods incorporating bounding boxes to achieve a more accurate estimation of the initial state.

[0048] Within one approach, extended dimension estimation can be performed. In this approach, the extended dimension estimate of the bounding box can be incorporated into the generalized state vector of the object candidate.

[0049] Another approach is to perform classification. Here, the category labels of the bounding boxes can be used directly for object candidates.

[0050] In addition, other corresponding methods can be envisioned to fill in the remaining parameters of the state vector.

[0051] Furthermore, a new maturity test can be performed. Since the machine learning model 50 can be treated as a virtual (radar) sensor 2 and stored as object candidates in addition to the localization set, this can also be used to compute a more robust maturity test. Current model-based approaches are particularly based on attributes of localization already collected for object candidates. However, clutter, angular ambiguity, and other effects of sensor 2 can lead to false positives being initialized. Since the machine learning model 50 has learned several of these effects, this advantage can now be incorporated into the maturity test. Therefore, it can be tested before tracking initialization whether at least one or more bounding boxes can be associated with an object candidate. This can reduce the number of false positive tracking.

[0052] Another advantage is that if the maturity test is successful, not only can the false positive rate be reduced, but the candidate object can also be delivered to the object tracking or Track-Maintenance much faster.

[0053] The above explanation of the embodiments is merely a description of the invention within the scope of examples. Of course, various features of the embodiments, as long as they are technically meaningful, can be freely combined with each other without departing from the scope of the invention.

Claims

1. A method (100) for determining the state of an object (3) in the environment of a vehicle (1), the method comprising: - Provide (101) sensor data, wherein the sensor data is generated from the detection of at least one sensor (2) of the vehicle (1), particularly a radar sensor, wherein the object (3) is represented in the sensor data. - Based on the provided sensor data, at least one representation of the state is obtained using a machine learning model (50), wherein the machine learning model (50) is trained for object detection and / or classification. - Based on the provided sensor data, using at least one physical model, obtain at least one additional characterization of the state described in (103).

2. The method (100) according to claim 1, characterized in that: The at least one characterization is the category, Cartesian position, orientation, Cartesian velocity, speed, yaw rate and / or extended size of the object (3), and the state is preferably represented by a state vector.

3. The method (100) according to claim 2, characterized in that: Within the scope of the determination (102) performed using the machine learning model (50), at least the category, Cartesian position, orientation, and extended dimensions of the object (3) are determined and provided in the form of bounding boxes.

4. The method (100) according to claim 2 or 3, characterized in that: Within the scope of the determination (103) performed using the at least one physical model, at least the Cartesian velocity and / or velocity of the object (3) shall be determined, preferably based on the provided bounding box.

5. The method (100) according to any one of the preceding claims, characterized in that: The sensor data includes a timestamp, which represents the time point at which the corresponding sensor data was detected. The timestamp is taken into account when calculating (102) using the machine learning model (50) and / or when calculating (103) using the at least one physical model.

6. The method (100) according to any one of the preceding claims, characterized in that: The method (100) further includes: - Perform object tracking of the object (3) using the obtained state of the object (3), wherein, within the scope of the object tracking, the state of the object (3) is continuously updated using the machine learning model (50) and / or the at least one physical model.

7. The method (100) according to claim 6, characterized in that: The method (100) further includes: - A maturity test is performed based on the sensor data and / or based on at least one representation obtained using the machine learning model (50), wherein, within the scope of the maturity test, the sensor data and / or at least one representation obtained using the machine learning model (50) are examined in terms of at least one defined quality criterion. Specifically, object tracking is performed only if the result of the maturity test indicates that "at least one of the defined quality criteria is met".

8. The method (100) according to any one of the preceding claims, characterized in that: The method (100) further includes: - Control of the vehicle (1) is initiated based on the state obtained from the object (3).

9. A computer program (20) comprising instructions that, when executed by a computer (10), cause the computer to perform the method (100) according to any one of the preceding claims.

10. An apparatus (10) for data processing, configured to implement the method (100) according to any one of claims 1 to 8.

11. A computer-readable storage medium (15) comprising instructions that, when executed by a computer (10), cause the computer to perform the steps of the method (100) according to any one of claims 1 to 8.

Images