Machine-learned traffic situation completion

Abstract

The invention relates to a method for generating a model for completing data on other traffic participants (3) in a traffic situation incompletely detected using sensors, having the steps of: providing (S1) a complete data set on a traffic situation, comprising the trajectories of an ego vehicle (1) and all of the other traffic participants (3); reducing (S2) the complete data set by the trajectories of one or more of the other traffic participants (3); and generating (S3) the model by means of a machine learning process using input variables from the reduced data set and output variables for the model from the complete data set such that the model is designed to generate a complete data set of the respective current traffic situation from an incomplete data set, said complete data set comprising the trajectories of the other traffic participants (3). In the process, the step of generating the model or the algorithm by means of a machine learning process is repeated for a plurality of different reduced data sets.

Description

[0001] MACHINE-LEarned traffic situation completion

[0002] The invention relates to a method for generating a model or an algorithm for completing data about other road users in a traffic situation that is incompletely recorded by sensors from an ego perspective of a vehicle, as well as a system for completing data about other road users in a traffic situation that is incompletely recorded by sensors from an ego perspective of a vehicle.

[0003] To control automated vehicles or to implement a driver assistance system in a manually driven vehicle, the behavior of other road users is typically predicted, particularly with regard to their trajectories—in terms of the actions and reactions of the road users. For the purpose of prediction, relevant scenarios are typically considered, which are used, for example, in a birds-eye view simulation (a simulation with information from a bird's perspective) to model traffic environments with acting and reacting road users. Ideally, these traffic environment models are based on complete descriptions of scenarios, the information for which is obtained, for example, through the use of drones for traffic monitoring.However, such bird's-eye view data, obtained, for example, through the use of unmanned aerial vehicles or other stationary traffic monitoring systems, is generally not available during regular vehicle operation. In all these cases, the alternative option of using data from a vehicle's first-person perspective while it is operating in traffic must be used – this is also known as shadowing. However, this data, unmodified, cannot per se provide complete knowledge of the environment, since the vehicle's sensors cannot detect all the information available from a bird's-eye view (for example, due to obscuring road users or a restricted perspective).

[0004] The object of the invention is therefore to enable the utilization of incomplete traffic data, which were recorded in particular from the ego perspective of a road user (according to the so-called individual shadowing), reliably by completing it.

[0005] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0006] A first aspect of the invention relates to a method for generating a model or an algorithm for completing data about other road users in a traffic situation that is incompletely recorded by sensors from a first-person perspective of a vehicle, comprising the steps:

[0007] - Providing a complete data set on a traffic situation, wherein the complete data set includes the trajectory of an ego vehicle and the trajectories of all other road users in the traffic situation within a given time window;

[0008] - Reducing the complete data set by the trajectories of one or more of the other road users to a reduced data set;

[0009] - Generating the model or algorithm by machine learning with predetermined input variables from the reduced data set and predetermined output variables of the model or algorithm from the complete data set, so that the model or algorithm is designed to generate a complete data set of the current traffic situation, including the trajectories of the other road users, from incomplete data by means of plausible reconstruction; in this case, the generation of the model or algorithm by machine learning is repeated for a large number of different reduced data sets.

[0010] The model or algorithm generated using the method according to the invention serves—when fully generated—to complete data on other road users in a traffic situation captured by sensors from a first-person perspective of a vehicle. Whether a model or an algorithm is used is a matter of interpretation. While an artificial neural network can be used for a model, other estimators can also be applied, such as a hidden Markov model, in which the incomplete data is considered as emissions from the system. In contrast to such models, an algorithm is a sequential series of instructions. For example, a Kalman filter is an algorithm, not a model.Although the model and algorithm differ in the points mentioned above, they are functionally equivalent in order to provide a system suitable for completing data about other road users in a traffic situation recorded by sensors from an ego perspective of a (ego) vehicle.

[0011] The traffic situation includes at least the trajectories of the vehicle under consideration, also referred to as the ego vehicle, as well as those of other road users. However, the term "trajectory" is also justified when the ego vehicle or the other road users are not moving, since the term "trajectory" indicates a trajectory with temporal information, and thus, even a static position can be described by a trajectory with temporal information.

[0012] The complete dataset therefore describes a traffic situation in which both the ego vehicle and the other road users are located. The ego vehicle is the vehicle under consideration which, analogous to the training phase, could apply the fully trained model or the fully designed algorithm to complete data based on the sensor data acquired from its ego perspective in its regular operation. It is not necessary for such a model or algorithm to already be provided in the complete dataset for the ego vehicle, because the complete dataset only serves to train such a model or algorithm, and the system for executing the completed model or algorithm is intended for the subsequent regular operation of a vehicle and is not necessarily yet functionally included in the complete dataset.

[0013] However, the role of the ego vehicle in the complete dataset defines the ego perspective, according to which the data of the complete dataset or the already reduced dataset are to be aligned, as if the data had been sensorily captured from this ego perspective. It may be necessary to process the reduced dataset as if the data from the respective dataset were captured from the ego perspective of the vehicle with the system for executing the model or the simulation.of the algorithm; then it does not matter whether the complete data set is first transformed (if it is not yet ready) as if it had been sensorily ascertained from the ego perspective of this ego vehicle in question, or whether the complete data set is first reduced accordingly and the reduced data set is transformed (if it is not already) as if it had been sensorily ascertained from the ego perspective of this ego vehicle in question.

[0014] The above can be particularly true if the complete dataset in its original form includes information from a bird's-eye view. Possible training datasets from a drone perspective would be, for example, the HighD dataset ("HighD Dataset" from www.highd-dataset.com) or, in urban areas, the InD dataset ("inD Dataset" from www.ind-dataset.com). The completeness of the data provides a so-called "ground truth." However, such a bird's-eye view cannot be determined from the ego vehicle's own sensors. A transformation may then be necessary that references the data to the ego vehicle's sensor view. This serves to maintain consistency, so that when training the model or algorithm, input data with a comparable reference and comparable perspectives are used as in later operation of the vehicle when applying the fully trained model or fully designed algorithm.

[0015] However, a transformation into the ego vehicle's ego perspective is not necessary if already interpreted sensor data is used in the respective dataset both for training the model or designing the algorithm, as well as during the subsequent operation of the fully trained model or fully designed algorithm, for example, determined trajectories of other road users detectable by the ego vehicle's sensors. In this case, the input data of the model or algorithm no longer contains any information about the perspective from which the other road users were recorded. Rather, the perspective is resolved and abstracted into information about the traffic situation.

[0016] The reduction of the complete data set by one or more of the additional road users to a reduced data set is preferably only carried out to the extent that at least one additional road user remains in the reduced data set in order to provide a data basis for estimating the behavior of additional users based on the behavior of this one additional road user.

[0017] The model or algorithm is created through machine learning. In the case of artificial neural networks, this can be done through what is known as "back propagation." In particular, machine learning is carried out in a form of supervised learning. When creating the model or algorithm, in the case of an artificial neural network, the parameters (weights) – and in rare, complex cases, optionally also the structure of the artificial neural network, such as the number of levels (so-called "layers") – are adjusted. Regardless of whether a model or an adaptable algorithm is used, a prerequisite for creation is always that input data is specified and output data corresponding to the input data is also specified, and the transmission by the model or algorithm is adjusted, particularly iteratively, until the model or algorithm isThe algorithm independently uses only the input data to arrive at output data that corresponds to the specified ones. Once this state is reached, the fully trained model or the fully designed algorithm is capable of using current input data from reality to determine corresponding output data that may, as intended, correspond to reality—in this case, a correct completion of data about other road users.

[0018] Preferably, machine learning is performed with the specified output variables of the model or algorithm from the complete dataset in such a way that data from the complete dataset is used that is not present in the reduced dataset. In particular, the specified output variables include all data from the complete dataset.

[0019] Preferably, the model or algorithm is also capable of detecting whether all relevant actors of the scenario are present in the dataset. In addition, the reduced dataset, and in particular a complete dataset reduced in several ways, can serve other purposes, in particular training, validation, and testing of the resulting models and algorithms. Furthermore, the term "model" can include a multitude of submodels, for example, by executing a separate, definable submodel for each estimated additional road user and their trajectories. However, the multitude of these submodels is understood under the term "model" used above and below. The same applies to the algorithm.

[0020] Machine learning particularly exploits implicitly known situations and patterns, such as the reaction of another road user observable by the ego vehicle's sensors to a third road user who is, however, not recognizable by the ego vehicle's sensors. Such a reaction could, for example, be an evasive maneuver, a yielding of right-of-way, or something similar. From the reaction of the other road user, which is detected by sensors, the presence and trajectory of the third road user can be deduced – this corresponds to the completion of the data set, which in the present example is initially incomplete and initially only includes the other road user observable by the ego vehicle's sensors.

[0021] If the generation of the model or algorithm through machine learning is repeated for a multitude of different reduced datasets, the same complete dataset can be used to provide the initial data in each iteration, or the complete dataset can be replaced to provide a new database for a further multitude of reduced datasets. The term "plausible" is used here because the model or algorithm is intended to complete the traffic situation in a realistic way, so that a reconstructed traffic situation with completion would result in the same dataset for the recording vehicle. Nevertheless, the actually recorded situation may differ from the reconstructed one. For example, a real ego vehicle would slow down when encountering intersecting traffic, regardless of whether the intersecting traffic is a truck or a bicycle.

[0022] It is therefore an advantageous effect of the invention to provide a model or algorithm that enables the plausible completion of incompletely recorded data about a traffic situation. Thus, the so-called "ground truth" is reconstructed from individual shadowing recordings without the need for stationary traffic monitoring. Rather, additional information is obtained from existing data to estimate reality not captured by sensors.

[0023] According to an advantageous embodiment, a plurality of different reduced data sets are obtained from a single complete data set.

[0024] According to a further advantageous embodiment, the model or the algorithm comprises a plausibility check, wherein the plausibility check is provided with an estimate of a respective trajectory of the road users that can be detected by sensors and those that cannot be detected by sensors by the ego vehicle, wherein in the event of a deviation of an estimate of a trajectory of a further road user that can be detected by sensors from a sensor-based determined trajectory of this further road user from the model or the algorithm, an estimated trajectory of a further road user that cannot be detected by sensors is generated for completion, wherein the plausibility check is adapted by machine learning when the model or the algorithm is generated.

[0025] According to a further advantageous embodiment, heuristics that remain unchanged during generation are implemented in the model or in the algorithm.

[0026] The basic procedure for training the model or designing the algorithm is that one or more plausible trajectories are estimated for each road user based on one or more models. If a road user deviates from these trajectories, i.e., behaves in an unusual or implausible manner, an attempt is made to convert this deviation into plausible behavior by adding additional road users.

[0027] Since there are in principle an infinite number of possibilities for this, various points are taken into account when adding road users: If a vehicle suddenly moves to the left and then back to the middle of the lane, a road user, e.g. a pedestrian who walks onto the road, is estimated to be placed on the right side of the vehicle.

[0028] According to a further advantageous embodiment, the machine learning during the generation of the model or algorithm is carried out with the restriction that additional road users that cannot be detected by sensors are placed by estimation at locations that lie outside a sensory detection range of the ego vehicle.

[0029] According to a further advantageous embodiment, when reducing the complete data set, only those of the other road users that lie outside a sensory detection range of the ego vehicle are removed.

[0030] A further aspect of the invention relates to a system for completing data about other road users in a traffic situation that is incompletely recorded by sensors from an ego perspective of a vehicle, wherein the system is designed to execute a model or an algorithm, wherein input data based on sensor information of an ego vehicle and comprising data about the trajectories of other road users, which may be incomplete, are used to execute the model or the algorithm, and a complete data set generated by plausible reconstruction is obtained as output data of the model or the algorithm, which data set includes trajectories of the other road users for the respective current traffic situation that are not directly contained in the sensor information of the ego vehicle.

[0031] According to a further advantageous embodiment, the system is designed for a first ego vehicle and to link sensor information transmitted from a second ego vehicle through data fusion in order to expand the first ego vehicle's own sensor information and reduce the amount of data to be completed. According to a further advantageous embodiment, the system is designed to determine and output a plurality of possible plausibly completed data.

[0032] Preferably, the respective completed data are provided with a value of the respective calculated probability for it.

[0033] According to a further advantageous embodiment, the system is designed to carry out a plausibility analysis for plausible reconstruction, which includes a completeness analysis, wherein the completeness analysis checks whether the existing estimated number of other road users is sufficient to be able to plausibly explain the trajectories of all other road users, wherein the trajectories of all other road users include the trajectories of other road users determined by sensor data and the trajectories of other estimated existing road users estimated by completion.

[0034] Advantages and preferred developments of the proposed system result from an analogous and analogous transfer of the statements made above in connection with the proposed method.

[0035] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0036] They show:

[0037] Fig. 1 : A method for generating a model or an algorithm for completing data about other road users in a traffic situation that is incompletely detected by sensors from an ego perspective of a vehicle, according to an embodiment of the invention.

[0038] Fig. 2: A system for completing data about other road users in a traffic situation that is incompletely recorded by sensors from a first-person perspective of a vehicle, according to an embodiment of the invention. The representations in the figures are schematic and not to scale.

[0039] Fig. 1 shows a method for generating a model or algorithm for completing data about other road users 3 in a traffic situation that is incompletely recorded by sensors from a first-person perspective of a vehicle. The steps of the method include:

[0040] - Providing S1 a complete data set about a traffic situation, wherein the complete data set includes the trajectory of an ego vehicle 1 and the trajectories of all other road users 3 of the traffic situation within a predetermined time window;

[0041] - Reducing S2 of the complete data set by the trajectories of one or more of the other road users 3 to a reduced data set;

[0042] - generating S3 of the model or algorithm by machine learning with predetermined input variables from the reduced data set and predetermined output variables of the model or algorithm from the complete data set, so that the model or algorithm is designed to generate a complete data set of the respective current traffic situation comprising the trajectories of the other road users 3 from incomplete data by means of plausible reconstruction;

[0043] Fig. 2 shows a system in use for completing data about other road users 3 in a traffic situation that is incompletely recorded by sensors from a vehicle's ego perspective. The system is based on the results of the method in Fig. 1 and involves running a model, wherein input data based on sensor information from an ego vehicle 1 and including data about the trajectories of other road users 3, which may be incomplete, are used to run the model. The ego vehicle 1 has a large number of sensors, such as lidar, radar, ultrasonic distance sensors, stereo cameras, etc.; on the one hand, the sensors all have finite ranges, and on the other hand, in some situations, such as the one shown in Fig. 2, other road users 3 in the immediate vicinity of the ego vehicle 1 cannot be recorded by sensors due to obscuration by trees, buildings, and the like.The additional road users 3 shown in Figure 2 are both relevant for the ego vehicle 1 in order to be able to assess the scenario shown at an intersection and to specify appropriate driving maneuvers via a decision-making module. However, since the sensor detection radius R is limited, only the additional road user 3 on the lower right is detected, but not the additional road user 3 approaching from above. This lies outside the radius R and is also obscured by a building at the intersection. The additional road user 3 approaching from the right-hand road branch is, however, detected by the sensors of the ego vehicle 1, so that its trajectory can be determined directly from the sensor data of the ego vehicle 1.From this trajectory of the additional road user 3 approaching from the right, the ego vehicle 1 recognizes that this additional road user 3 is behaving differently in response to something, namely that it is braking and slowly approaching the road branch turning in from the right from its perspective. The model trained according to Fig. 1 for a respective road user is compared with the data from a digital map, the position determination of the ego vehicle 1 and the determined position of the right-hand additional road user 3 detected by sensors, and the system output of this model generates an additional road user 3 by estimate, which cannot be detected by the sensors for the ego vehicle 1. This additional road user 3, determined and added by estimate, corresponds in Fig.2 the additional road user 3 approaching from above, whose existence is henceforth assumed with a certain probability for the ego vehicle 1. The trajectory of the additional road user 3 is therefore known for the ego vehicle 1 insofar as the initial data of the model includes that this additional road user 3 is located on the upper road from the perspective of the ego vehicle 1, with the direction of travel downwards, so that this additional road user 3 appears in a right-of-way situation for the sensor-detectable additional road user 3 and it can be expected that this additional road user 3 from above will soon be in a relevant area around the ego vehicle 1.At this point, alternatives of a bifurcation arise - both alternatives are output by the model and transferred to a decision module of the ego vehicle 1 to determine the further behavior.

[0044] Although the invention has been illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned by way of example are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

[0045] List of reference symbols

[0046] 1 ego vehicle

[0047] 3 road users

[0048] S1 Provide S2 Reduce

[0049] S3 Create

[0050] R sensor detection radius

Claims

Patent claims 1. A method for generating a model or an algorithm for completing data about other road users (3) in a traffic situation that is incompletely recorded by sensors from a first-person perspective of a vehicle, comprising the steps: - Providing (S1) a complete data set about a traffic situation, wherein the complete data set comprises the trajectory of an ego vehicle (1) and the trajectories of all other road users (3) of the traffic situation within a predetermined time window; - reducing (S2) the complete data set by the trajectories of one or more of the other road users (3) to a reduced data set; - generating (S3) the model or algorithm by machine learning with predetermined input variables from the reduced data set and predetermined output variables of the model or algorithm from the complete data set, so that the model or algorithm is designed to generate a complete data set of the respective current traffic situation comprising the trajectories of the other road users (3) from an incomplete data set by plausible reconstruction; wherein the generation of the model or algorithm by machine learning is repeated for a plurality of different reduced data sets.

2. The method according to claim 1, wherein a plurality of different reduced data sets are obtained from a single complete data set.

3. Method according to one of the preceding claims, wherein the model or the algorithm comprises a plausibility check, wherein the plausibility check is provided with an estimate of a respective trajectory of the road users (3) that can be detected by sensors and those that cannot be detected by sensors by the ego vehicle (1), wherein in the case of a deviation of an estimate of a trajectory of a further road user (3) that can be detected by sensors from a trajectory of this further road user (3) determined on the basis of sensors, the model or the algorithm generates an estimated trajectory of a further road user (3) that cannot be detected by sensors and that has been added for completion, wherein the plausibility check is carried out when generating the The model or the algorithm is adapted by machine learning. Method according to claim 3, wherein heuristics which remain unchanged during generation are implemented in the model or in the algorithm. Method according to one of claims 3 to 4, wherein the machine learning during generation of the model or algorithm is carried out with the restriction that further road users (3) which cannot be detected by sensors and are added for completion are virtually placed at locations which lie outside a detection range of the ego vehicle (1). Method according to one of the preceding claims, wherein when reducing the complete data set, only those of the further road users (3) which lie outside a detection range of the ego vehicle (1) are removed.System for completing data about other road users (3) in a traffic situation that is incompletely recorded by sensors from an ego perspective of a vehicle, wherein the system is designed to execute a model or an algorithm, wherein input data based on sensor information of an ego vehicle (1) and comprising data about the trajectories of other road users (3), which may be incomplete, are used to execute the model or the algorithm, and a complete data set generated by plausible reconstruction is obtained as output data of the model or the algorithm, which data set includes trajectories of the other road users (3) for the respective current traffic situation that are not directly contained in the sensor information of the ego vehicle (1).The system according to claim 7, wherein the system is for a first ego vehicle (1) and is configured to combine sensor information transmitted from a second ego vehicle (1) by data fusion in order to expand the first ego vehicle's (1) own sensor information and reduce the amount of data to be completed. The system according to any one of claims 7 to 8. wherein the system is designed to determine and output a plurality of possible plausibly completed data. System according to claim 9, wherein the system is designed to carry out a plausibility analysis for plausible reconstruction, which includes a completeness analysis, wherein the completeness analysis checks whether the existing estimated number of additional road users (3) is sufficient to plausibly explain the trajectories of all additional road users (3), wherein the trajectories of all additional road users (3) include the trajectories of additional road users (3) determined by sensor data and the trajectories of additional estimated existing road users (3) estimated by completion.

Images