Vehicle system and method for increasing the reliability of predicting future trajectories

Abstract

Vehicle system (1, 1a, 1b) for increasing the reliability of a prediction of future trajectories of road users in the vicinity of an ego vehicle comprising a sensor system (2) which is designed to detect the environment of the ego vehicle using sensors based on sensor data, and an environment module (3) which is designed to extract the road users as well as movement data of the road users from the past sensor data and which is further designed to generate a previous movement trajectory of the road users within a scene based on the recorded movement data. a trajectory prediction module (4) which is trained to generate one or more future trajectories for each detected road user based on the previous movement trajectory, each with an assigned probability of occurrence for the road user, wherein the trajectory prediction module (4) includes at least one trained machine learning method which is trained on training data as an in-distribution pattern (ID pattern) to generate one or more future trajectories based on a previous movement trajectory and to assign a probability of occurrence to each of the future trajectories, characterized by the fact that a recognition unit (6) is provided which is designed to recognize out-of-distribution (OOD) patterns present within a scene based on the previous motion trajectory and / or the motion data and / or to determine the deviations between out-of-distribution (OOD) patterns and in-distribution (ID) patterns, and wherein the recognition unit (6) is further designed to use the existing deviations as a reliability of the probability of occurrence of the future trajectories or to determine the reliability based on the deviation, wherein a motion planning module (5) is provided which is configured to generate an ego trajectory for the ego vehicle, based on one or more future trajectories, each with the associated probability of occurrence of the road users, as well as on the determined reliability of the probability of occurrence of the future trajectories, wherein a sensor-OOD module (7) is present and the sensor-OOD module (7) is configured to receive the sensor data and, at least in the presence of an out-of-distribution (OOD) pattern, to detect a semantic or non-semantic shift of the sensor data compared to the training data and the sensor-OOD module (7) has artificially generated attack data, wherein the sensor-OOD module (7) is configured to detect out-of-distribution (OOD) patterns based on the artificially generated attack data.

Description

[0001] The invention relates to a vehicle system for increasing the reliability of predicting future trajectories of road users in the vicinity of an ego-vehicle, comprising a sensor system configured to detect the environment of the ego-vehicle using sensors based on sensor data, and an environment module configured to extract road users and movement data of the road users from the past sensor data and further configured to generate a previous movement trajectory of the road users within a scene based on the recorded movement data, as well as a trajectory prediction module configured to generate one or more future trajectories for each detected road user based on the previous movement trajectory, each with an assigned probability of occurrence for each road user.wherein the trajectory prediction module comprises at least one trained machine learning method which is trained on training data as an in-distribution pattern (ID pattern) to generate one or more future trajectories based on a previous movement trajectory and to assign a probability of occurrence to each of the future trajectories. Furthermore, the invention relates to a method.

[0002] An important component, particularly relevant in automated vehicles, is the module for predicting the possible / probable trajectories of other road users. Human drivers also perform this task more or less unconsciously by observing the recent behavior of other road users and a range of additional information, such as their position relative to infrastructure like lanes, lane markings, traffic lights, etc.

[0003] Based on its "internal" prediction of how other road users will move within the next few seconds, the vehicle plans and executes its own approach / route / action. An automated vehicle operating in mixed traffic must also be able to handle this task.

[0004] It is therefore an object of the invention to provide an improved vehicle system and a method for the safe generation of future trajectories of road users in the vicinity of an ego vehicle.

[0005] State of the art is disclosed, for example, in DE 10 2021 203 588 A1, DE 10 2019 215 147 A1, DE10 2019 209 736 A1 and US 2021 / 0 001 897 A1.

[0006] The problem is solved by a vehicle system having the features of claim 1 and a method having the features of claim 15.

[0007] The task is solved by a vehicle system for increasing the reliability of predicting future trajectories of road users in the vicinity of an ego-vehicle, comprising a sensor system trained to capture the ego-vehicle's environment using sensors based on sensor data, and an environment module trained to extract road users and their movement data from past sensor data and further trained to generate a previous movement trajectory of the road users within a scene based on the recorded movement data, as well as a trajectory prediction module trained to generate one or more future trajectories for each detected road user based on the previous movement trajectory, each with an assigned probability of occurrence for each road user.wherein the trajectory prediction module comprises at least one trained machine learning algorithm which is trained on training data as in-distribution patterns (ID patterns) to generate one or more future trajectories based on a previous movement trajectory and to assign a probability of occurrence to each of the future trajectories, wherein a recognition unit is provided which is trained to recognize existing out-of-distribution (OOD) patterns based on the previous movement trajectory and / or the movement data, as well as to determine the deviations between out-of-distribution (OOD) patterns and the in-distribution (ID) patterns, and wherein the recognition unit is further trained to use the existing deviations as a reliability of the probability of occurrence of the future trajectories or to determine the reliability based on the deviation, and, wherein a motion planning module is provided which is designed to generate an ego trajectory for the ego vehicle based on one or more future trajectories, each with the assigned probability of occurrence of the road users, as well as based on the determined reliability of the probability of occurrence of the future trajectories.

[0008] Reliability can be understood as the dependability with respect to the predicted probability of occurrence of the individual predicted future trajectories; in the broadest sense, reliability can thus be understood as the probability that the trajectory prediction module functions correctly. Such reliability can, for example, be expressed as a confidence level.

[0009] The machine learning method can, for example, be a deep learning model that is trained using training data.

[0010] Out-of-distribution (OOD) patterns can be, for example, new patterns / sensor data that differ significantly from the training data. In a real-world setting, out-of-distribution (OOD) patterns can belong to any category, such as a wrong-way driver, a vehicle making a wrong turn, or a vehicle traveling at extremely high speed.

[0011] Out-of-distribution patterns can therefore be, for example, anomalies, unknown patterns, or outliers.

[0012] According to the invention, it was recognized that increasing the reliability of the occurrence of a predicted future trajectory of road users in the vicinity of an ego vehicle is essential for carrying out safe control of the ego vehicle.

[0013] According to the invention, a sensor system and an environment module are provided for this purpose. The sensor system includes sensors such as cameras, radar, maps, etc., to capture sensor data from the environment of the ego-vehicle.

[0014] The environment module extracts road users and their movement data from past sensor data. Past sensor data covers a short period up to the currently recorded sensor data. The environment module also includes a representation of the scene, for example, a map with traffic signs, lane markings, etc.

[0015] Based on the recorded movement data, the environment module generates a previous movement trajectory of road users on the map. This means that the environment module generates the temporal development of individual road users within the recorded environment in the form of movement trajectories within the scene.

[0016] According to the invention, a trajectory prediction module is provided which generates one or more future trajectories for each detected road user, each with an assigned probability of occurrence. This means that the trajectory prediction module predicts how road users might move / behave in the future, based on multiple future trajectories (multi-future prediction), each with an assigned probability of occurrence.

[0017] The trajectory prediction module employs a trained machine learning algorithm, specifically a deep neural network, which is trained on training data to generate one or more future trajectories based on a past movement trajectory and to assign a probability of occurrence to each of these future trajectories. This training data represents the in-distribution patterns (ID patterns) that the deep neural network can process based on its training.

[0018] Furthermore, a recognition unit is provided which is designed to recognize existing out-of-distribution (OOD) patterns based on the previous movement trajectory and / or the movement data.

[0019] This means that the recognition unit identifies patterns that are new or unknown to the machine learning process, for example, because they are anomalies or outliers. Furthermore, the deviation between out-of-distribution (OOD) patterns and in-distribution (ID) patterns is determined. The recognition unit is trained to determine the reliability of the probability of occurrence of individual future trajectories based on an existing deviation, or to use this deviation as a basis for reliability. Thus, if the recognition unit detects an out-of-distribution (OOD) pattern, it can be assumed, for example, that the reliability of the trajectory prediction module—that is, the reliability of the probability of occurrence of individual future trajectories—is lower than normal. The magnitude of the deviation can serve as a measure of this reduced reliability.

[0020] Furthermore, a motion planning module is provided that generates an ego trajectory for the ego vehicle based on one or more future trajectories, each with an assigned probability of occurrence for the road users, using the specific reliability of the probability of occurrence of the future trajectories. For example, if an out-of-distribution (OOD) pattern is detected, the motion planning module can plan a safer ego trajectory, perhaps with a lower speed. The motion planning module plans not only the ego trajectory but also the corresponding speed at each of the ego trajectory points.

[0021] This means that the motion planning module decides within the next few seconds how the autonomous vehicle should move based on an ego trajectory and transmits this information to the actuator control for execution. This significantly increases the safety of automated driving functions / operations.

[0022] The vehicle system according to the invention thus increases the reliability of the output probabilities of occurrence with regard to their actual occurrence. If the reliability is low, it can be assumed that the specified future trajectories will not occur with the stated probability of occurrence.

[0023] By recognizing out-of-distribution (OOD) patterns and the magnitude of the deviation, which is passed on to the motion planning module, a different and potentially less risky motion plan can be carried out by the motion planning module.

[0024] In further training, the recognition unit is configured to store out-of-distribution (OOD) patterns along with subsequent sensor data and / or motion trajectories. Furthermore, in additional training, the trajectory prediction module can be configured to train the machine learning algorithm using the stored out-of-distribution (OOD) patterns and subsequent sensor data and / or motion trajectories. The out-of-distribution (OOD) patterns are stored locally in the ego-vehicle to refine the machine learning algorithm.

[0025] Alternatively or additionally, the vehicle system can have an interface and be configured to transmit out-of-distribution (OOD) patterns along with subsequent sensor data to an external server. There, the machine learning algorithm can be further trained, for example, with increased computing power on the external server, and the update can be transmitted back to the vehicle system. Furthermore, such an update can be distributed to all autonomous vehicles equipped with this vehicle system, allowing for continuous improvement of the machine learning algorithm. Additionally, the stored subsequent sensor data can be used to manually verify whether an out-of-distribution (OOD) pattern is indeed present and whether the deviation has been correctly identified. This further enhances the machine learning algorithm.

[0026] In further development, the vehicle system can include a remote control for externally controlling the ego vehicle and an interface. The detection unit is configured to transmit any deviation above a predefined threshold to a remote control center via this interface, enabling monitoring or control of the ego vehicle. This means, for example, that a control center for the ego vehicle can be informed as a remote control center. Depending on the severity of the deviation from the out-of-distribution (OOD) patterns and the criticality of the situation, the control center can remotely connect to the ego vehicle and at least monitor the situation, or even actively intervene if necessary.

[0027] In further training, the vehicle system has a storage unit with simulated and / or real Out-Of-Distribution (OOD) attack patterns, whereby the detection unit is trained to compare the current Out-Of-Distribution (OOD) patterns with the simulated and / or real Out-Of-Distribution (OOD) attack patterns.

[0028] Such out-of-distribution (OOD) attack patterns can be used as adversarial attacks to target the vehicle system, for example, sensor data or generated motion trajectories. These attack patterns are, for example, external attacks that trigger signals which generate malfunctions. This allows, for instance, the detection of a critical out-of-distribution (OOD) pattern with a high degree of deviation, enabling immediate notification of the remote control center or the initiation of emergency measures. Out-of-distribution (OOD) attack patterns can affect all kinds of data; even training data can be manipulated to produce such patterns.

[0029] Furthermore, the detection unit can be trained to determine the deviation between out-of-distribution (OOD) patterns and in-distribution patterns by comparing the deviation between current out-of-distribution (OOD) patterns and simulated and / or real out-of-distribution (OOD) attack patterns. Such a comparison can, for example, easily identify a deviation.

[0030] According to the invention, a sensor-OOD module is provided, wherein the sensor-OOD module is configured to receive the sensor data and, at least in the presence of an Out-Of-Distribution (OOD) pattern, to detect a semantic or non-semantic shift of the sensor data compared to the training data.

[0031] The sensor-OOD module can be operated in parallel with the detection unit and can, for example, be integrated into the detection unit as a software module. The sensor-OOD module directly receives the sensor data, either pre-processed or unprocessed raw sensor data. Based on the sensor data, non-semantic shifts in the sensor data, for example, due to different weather conditions, lighting, etc., can be detected more effectively. Non-semantic shifts can be understood as logical shifts resulting from different conditions, such as varying weather. This allows semantic or non-semantic shifts to be easily detected when an out-of-distribution (OOD) pattern is present.

[0032] Furthermore, the motion planning module is designed to generate an ego trajectory for the ego vehicle, based on one or more future trajectories, each with its assigned probability of occurrence, as well as the reliability of the probability of occurrence of the future trajectories and any detected semantic or non-semantic shift in the sensor data. This provides the motion planning module with better information about the reasons for the occurrence of the out-of-distribution (OOD) pattern, e.g., whether the shift in the distribution of the sensor data compared to the training data is primarily semantic or non-semantic. This information can, for example, be taken into account when generating an ego trajectory. Thus, the occurrence of an out-of-distribution (OOD) pattern can be treated as less dangerous if a non-semantic shift is present.

[0033] According to the invention, the sensor-OOD module includes artificially generated attack data, and the sensor-OOD module is configured to recognize such artificially generated attack data as out-of-distribution data. Artificially generated attack data includes, for example, external attacks such as signals that generate fault functions.

[0034] In further training, the recognition unit is trained to represent the previous movement trajectory of road users and / or sensor data in an embedding space and to recognize the existing out-of-distribution (OOD) patterns within the embedding space.

[0035] In such an embedding space, for example, the history of the sensor data of the ego vehicle, the history of the sensor data of other road users and information from a map, such as lanes, lane markings and traffic lights, etc., can be entered.

[0036] This means that in such a particularly complex and non-homogeneous embedding space, the history of the considered ego-vehicle, the history of other road users, and map information such as lanes, lane markings, and traffic lights, etc., are present. This information can be represented in the embedding space, and the algorithms for OOD detection can be performed within it.

[0037] Because the data is represented using the embedding space, complex, context-dependent situations can be detected, for example, to detect a driver traveling against the lane markings (wrong-way driver) or perpendicular to the lane markings. Trajectories alone are insufficient for this, as the relative orientation of these trajectories with respect to the lane markings is also required, which is made possible by the embedding space.

[0038] Such an embedding space is designed as a relatively low-dimensional space. Furthermore, the machine learning algorithm may also have been trained on this low-dimensional embedding space. Examples of such embedding spaces include JoMase Embedding, VectorNet Global Graph Embedding, and MATF Embedding.

[0039] In further training, the recognition unit is trained to detect out-of-distribution (OOD) patterns within the embedding space by applying a Mahalanobis distance to the distribution of in-distribution patterns, i.e., the known patterns. Since the patterns are represented using the input space, even complex, context-dependent situations can be detected.

[0040] In further training, the vehicle system is trained to extract and store ground-truth movement trajectories of road users from the sensor data, whereby the trajectory prediction module is trained to perform a calibration of the machine learning method with regard to the probability of occurrence, if necessary, based on the ground-truth movement trajectory of the respective road users and the corresponding assigned future trajectories with probability of occurrence.

[0041] The ground-truth motion trajectory is the actual motion trajectory of the road user as it will be traveled in the future.

[0042] By comparing the ground-truth motion trajectory with the predicted future trajectories, it is first possible to assess the reliability of the probability of occurrence relative to the actual occurrence of a future trajectory. If necessary, in cases of significant deviation, this probability can then be calibrated to better align with future reality. Such calibration can be performed offline (i.e., not in real time) or externally. The calibrated trajectory prediction module or machine learning algorithm can then be used. This calibration / improvement of the trajectory prediction module is independent of any standard improvement that can be achieved through typical retraining of the trajectory prediction module or machine learning algorithm.This allows the recognition unit to detect reliability in online mode, while the trajectory prediction module can be calibrated, for example, in offline mode or not in real time.

[0043] In further training, the trajectory prediction module is trained to perform calibration with respect to the probability of occurrence using parametric or non-parametric methods. Such probability calibration can include, for example, parametric methods like temperature scaling or Platt scaling and non-parametric methods like histogram binning and isotonic regression, which must be adjusted accordingly, since trajectory prediction with multiple outputs differs significantly from classification. There is an inherent aleatory uncertainty in learning the one-to-many mapping, which is usually higher than the aleatory uncertainty in learning the one-to-one mapping of the classification. Furthermore, an output does not uniquely represent a category, since, for example,The first output of a model could be a left-turn mode for one sample and a right-turn mode for another sample.

[0044] In further training, the trajectory prediction module is configured to only use the calibrated machine learning method when the recognition unit detects no out-of-distribution (OOD) patterns in the previous movement trajectory and / or movement data assigned to the respective road user. It was observed that applying out-of-distribution (OOD) patterns can degrade the trajectory prediction module through probability calibration. Removing the out-of-distribution (OOD) patterns, however, always improves the trajectory prediction module through probability calibration.

[0045] Furthermore, the task is solved by a procedure to increase the reliability of a prediction of future trajectories of road users in the vicinity of an ego-vehicle, comprising the following steps: - Capturing the environment of the ego vehicle using sensors based on sensor data, - Extracting road users and their movement data from past sensor data and generating a previous movement trajectory of road users within a scene based on the recorded movement data, - Generate one or more future trajectories for each detected road user based on the previous movement trajectory, each with an assigned probability of occurrence for the road user, using a trajectory prediction module, wherein the trajectory prediction module has at least one trained machine learning method which is trained on training data as an in-distribution pattern (ID pattern) to generate one or more future trajectories based on a previous movement trajectory and to assign a probability of occurrence to each of the future trajectories. - Detect existing out-of-distribution (OOD) patterns based on the previous movement trajectory and / or movement data using a recognition unit, - Determining the deviation between out-of-distribution (OOD) patterns and in-distribution (ID) patterns by the recognition unit, - Using the existing deviation as a measure of the reliability of the probability of future trajectories occurring, or determining the reliability based on the deviation, - Generating an ego trajectory for the ego vehicle by a motion planning module based on one or more future trajectories, each with an assigned probability of occurrence of the road users, as well as based on the determined reliability of the probability of occurrence of the future trajectories.

[0046] The method can be carried out on a vehicle system according to the invention. Furthermore, the advantages as well as the advantageous embodiments of the vehicle system according to the invention can be transferred to the method.

[0047] Further features and advantages of the present invention will become apparent from the following description with reference to the accompanying figures. These show: Fig. 1: a vehicle system in accordance with the state of the art, Fig. 2: a vehicle system according to the invention, Fig. 3: a further embodiment of a vehicle system according to the invention, Fig. 4: a further embodiment of a vehicle system according to the invention.

[0048] Fig. Figure 1 shows a state-of-the-art vehicle system 101 for an ego vehicle. This system includes a sensor system 102 which captures the environment / surroundings of the ego vehicle as sensor data using sensors such as cameras, radar, etc.

[0049] The sensor data also includes map information, which can be transmitted to the sensor system 102 or the subsequent environment module 103, for example, via a digital map.

[0050] The sensor data is transmitted to the environment module 103. The environment module 103 extracts the road users in the vicinity from the sensor data, ranging from past to present, and then forwards this data, in the form of previous movement trajectories (e.g., a movement trajectory over the last few seconds), to the trajectory prediction module 104. The environment module 103 also includes a representation of the scene, such as a map with traffic signs, lane markings, etc.

[0051] The trajectory prediction module 104 generates several future trajectories for each detected road user based on the previous movement trajectory, each with an assigned probability of occurrence (multi-future prediction).

[0052] For this purpose, the trajectory prediction module 104 features a trained artificial neural network which is trained on training data as an in-distribution pattern (ID pattern) to generate one or more future trajectories based on a movement trajectory and to assign a probability of occurrence to each of the future trajectories.

[0053] In-distribution patterns (ID patterns) are, for example, learned overtaking maneuvers, traffic stations, or traffic scenarios whose handling was learned by the artificial neural network from real data and / or simulation data.

[0054] Based on the future trajectories with the associated occurrence probabilities, a motion planning module 105 generates an ego trajectory for the ego vehicle, i.e., the motion planning module 105 decides how the ego vehicle should move within the next few seconds and passes this on to the actuator control.

[0055] However, in order to be able to perform safe actions for controlling the ego vehicle, it is essential to know how reliable the prediction of the trajectory prediction module 104 is.

[0056] Fig. Figure 2 shows a schematic representation of a vehicle system 1 according to the invention for an ego vehicle.

[0057] This system also features a sensor system 2, which uses sensors such as cameras, radar, and maps to capture sensor data about the surroundings of the ego vehicle. This sensor data also includes map information, which can be transmitted to sensor system 2 or the subsequent environment module 3, for example, via a digital map.

[0058] The sensor data is transmitted to the environment module 3. Environment module 3 extracts the road users in the vicinity from the sensor data, ranging from past to present, and then forwards this data, in the form of previous movement trajectories (arrow Pf1), for example, a movement trajectory over the last few seconds, to the trajectory prediction module 4. Environment module 3 also includes a representation of the scene, such as a map with traffic signs, lane markings, etc.

[0059] Trajectory prediction module 4 generates multiple future trajectories for each detected road user based on their previous movement trajectory, each with an assigned probability of occurrence (multi-future prediction). To achieve this, trajectory prediction module 4 utilizes an artificial neural network trained on in-distribution patterns (ID patterns) using training data. This network is trained to generate one or more future trajectories from a previous movement trajectory and to assign a probability of occurrence to each of these future trajectories.

[0060] Furthermore, a recognition unit 6 is provided which, based on the previous movement trajectories transmitted by the environment module 3, recognizes existing out-of-distribution (OOD) patterns.

[0061] Here, the detection unit 6 essentially analyzes the output data of the environment module 3, which simultaneously serves as the input data for the trajectory prediction module 4. Detection unit 6 identifies when out-of-distribution (OOD) patterns occur in the data at a specific point in time. Out-of-distribution (OOD) patterns can be, for example, new patterns / sensor data that differ significantly from the training data, such as anomalies, outliers, or new traffic situations. Thus, in Fig. 2 For example, a scenario with a wrong-way driver F is represented as an Out-Of-Distribution (OOD) pattern.

[0062] The decisive factor for such out-of-distribution (OOD) patterns is that the trajectory prediction module 4 has never processed such a type of pattern / data before, or that the artificial neural network cannot handle these out-of-distribution (OOD) patterns or can only handle them unreliably.

[0063] If the recognition unit detects 6 such OOD patterns, the consequence is that the reliability of the trajectory prediction module 4, or of the occurrence probabilities of the future trajectories determined by the trajectory prediction module 4, can be assumed to be lower than normal.

[0064] To determine reliability, the recognition unit 6 determines the deviation between out-of-distribution (OOD) patterns and in-distribution patterns. The reliability of the trajectory prediction module 4 or the occurrence probabilities of the future trajectories can be used as the deviation itself, or they can be determined based on the deviation.

[0065] Furthermore, a motion planning module 5 is available. Information about out-of-distribution (OOD) patterns, or the degree of deviation as a measure of reliability, is passed on to motion planning module 5. Based on the future trajectories with their associated probabilities of occurrence, as well as the information about the deviation and reliability from the trajectory prediction module 4, motion planning module 5 generates an ego trajectory for the ego vehicle. This means that motion planning module 5 decides how the ego vehicle should move within the next few seconds. With this knowledge of reliability, motion planning module 5 can, for example, implement a different and potentially less risky motion plan if the reliability is low, such as planning a lower speed.

[0066] Furthermore, the recognition unit 6 can locally store the out-of-distribution (OOD) patterns along with the subsequent sensor data and / or motion trajectories. The artificial neural network can then be further trained, i.e., improved, using this stored sensor data and / or motion trajectories. This training can be performed offline.

[0067] The out-of-distribution (OOD) patterns, along with subsequent sensor data and / or motion trajectories, can also be transmitted to an external server, such as a cloud, via an interface. There, the artificial neural network can be trained and the resulting training sent back to the ego-vehicle as an update.

[0068] The vehicle system 1 may also have a remote control for operation.

[0069] In the event of a particularly severe deviation, for example above a predefined threshold and depending on the perceived criticality, a Remote Control Center 8 can be informed in real time. Using the remote control, this center can actively take over or at least monitor the control of the ego vehicle.

[0070] Furthermore, vehicle system 1 can have a storage unit containing simulated and / or real out-of-distribution (OOD) attack patterns. These OOD attack patterns can relate to any type of data, including ID patterns, i.e., essentially the training data. This data can be manipulated to generate an OOD attack pattern. Such simulated OOD attack patterns can be used as adversarial attacks against vehicle system 1, for example, targeting the sensor data itself or the generated motion trajectories. These attack patterns include, for example, external attacks such as signals that generate error functions.

[0071] Thus, the detection unit 6 can compare current out-of-distribution (OOD) patterns with simulated and / or real OOD attack patterns and, based on this comparison, determine both the deviation and the severity of the deviation between the ID pattern and the OOD pattern. This allows, for example, the detection of a critical OOD pattern with a high deviation, enabling immediate notification of the Remote Control Center 8 or the initiation of emergency measures such as a shutdown. The detection unit 6 can also detect current OOD patterns without comparing them to simulated and / or real OOD patterns, for example, by measuring the distance to the ID data.

[0072] Fig. Figure 3 shows a further embodiment of a vehicle system 1a according to the invention.

[0073] This also features sensor system 2, which captures the environment / surroundings of the ego vehicle as sensor data using sensors such as camera, radar, maps.

[0074] The sensor data is transmitted to the environment module 3. The environment module 3 extracts the road users in the vicinity from the sensor data, ranging from past to present, and then forwards the sensor data, in the form of previous movement trajectories of the road users, to the trajectory prediction module 4.

[0075] The trajectory prediction module 4 generates several future trajectories for each detected road user based on the previous movement trajectory, each with an assigned probability of occurrence (multi-future prediction).

[0076] The trajectory prediction module 4 features the trained artificial neural network.

[0077] Furthermore, the recognition unit 6 is present, which recognizes existing out-of-distribution (OOD) patterns based on the previous movement trajectories transmitted by the environment module 3.

[0078] If the recognition unit detects 6 such OOD patterns, the consequence is that the reliability of the trajectory prediction module 4, or the probability of occurrence of the future trajectories determined by the trajectory prediction module 4, can be assumed to be lower than normal.

[0079] To determine reliability, the recognition unit 6 determines the deviation between out-of-distribution (OOD) patterns and in-distribution patterns. The reliability of the trajectory prediction module 4 or the probability of occurrence of the future trajectories can be used as the deviation itself, or it can be determined based on the deviation.

[0080] Furthermore, the recognition unit 6 can locally store the Out-Of-Distribution (OOD) patterns with the subsequent sensor data and / or the subsequent motion trajectories, which can be used to improve the artificial neural network, or transmit them to the external server along with the Out-Of-Distribution (OOD) pattern.

[0081] The vehicle system 1a can also have remote control for external intervention in case of particularly severe deviation.

[0082] In parallel to the detection unit 6, a sensor OOD module 7 can be present. This module can receive sensor data from sensor system 2. Since the input to sensor OOD module 7 comes directly from sensor system 2, it is able to detect, in the presence of an out-of-distribution (OOD) pattern, a semantic shift in the sensor data, or a non-semantic shift in the sensor data, such as changes in weather conditions, lighting, etc., compared to the training data, and forward this information to the motion planning module 5. This provides motion planning module 5 with better information about the reasons for the occurrence of OOD patterns. This knowledge can, for example, be taken into account when generating an ego trajectory. Thus, the occurrence of an out-of-distribution (OOD) pattern can be treated as less dangerous if a non-semantic, i.e.,There is a logical shift in the sensor data compared to the ID patterns or training data. Such a shift can occur, for example, due to changing weather conditions or snow.

[0083] Subsequently, the motion planning module 5 can generate the ego trajectory for the ego vehicle, based on one or more future trajectories of the road users, each with an assigned probability of occurrence, as well as on the reliability of the probability of occurrence of the future trajectories, and on a detected semantic or non-semantic shift in the sensor data when an out-of-distribution (OOD) pattern occurs. For example, the occurrence of an out-of-distribution (OOD) pattern can be treated as less dangerous if a non-semantic shift is present.

[0084] Fig. Figure 4 shows a further embodiment of a vehicle system 1b according to the invention.

[0085] This also features sensor system 2, which captures the environment / surroundings of the ego vehicle as sensor data using sensors such as camera, radar, maps.

[0086] The sensor data is transmitted to the environment module 3. The environment module 3 extracts the road users in the vicinity from the sensor data, ranging from past to present, and then forwards the sensor data, in the form of previous movement trajectories, to the trajectory prediction module 4.

[0087] The trajectory prediction module 4 generates several future trajectories for each detected road user based on the previous movement trajectory, each with an assigned probability of occurrence (multi-future prediction).

[0088] The trajectory prediction module 4 features the trained artificial neural network.

[0089] The vehicle system 1b extracts the ground truth movement trajectory of the road users from the sensor data and stores it.

[0090] The ground-truth motion trajectory is the actual motion trajectory of the road user as it will be traveled in the future.

[0091] Trajectory prediction module 4 is designed to compare the ground-truth movement trajectory of the respective road users with the correspondingly assigned future trajectories with their probability of occurrence, and, if necessary, to calibrate the artificial neural network (KknN, Calibrated Artificial Neural Network) with respect to the probability of occurrence. In particular, this can be done offline, i.e., not in real time.

[0092] For this purpose, the artificial neural network is loaded with future trajectories with their probability of occurrence, as well as the corresponding ground truth movement trajectory, and tested. The comparison then examines how reliable the probability of occurrence is in relation to the actual occurrence of a future trajectory.

[0093] If necessary, in cases of significant deviation, this probability of occurrence can subsequently be adjusted through calibration so that it better reflects reality in the future. Such calibration can be performed offline, i.e., not in real time, or externally.

[0094] The trajectory prediction module 4 can perform calibration with respect to the probability of occurrence using parametric methods such as temperature scaling or Platt scaling, or non-parametric methods such as histogram binning or isotonic regression.

[0095] To apply, for example, temperature scaling to the trajectory prediction module 4, new parameters can be added to a known softmax activation function.

[0096] Furthermore, the recognition unit 6 is present, which recognizes existing out-of-distribution (OOD) patterns based on the previous movement trajectories transmitted by the environment module 3.

[0097] The recognition unit 6 can represent the previous movement trajectories of road users and / or sensor data in an embedding space Ω and recognize the existing out-of-distribution (OOD) patterns within this generated embedding space Ω. Such an embedding space Ω can, for example, represent the history of the ego vehicle as sensor data, the history of other road users as sensor data, and map information such as lanes, lane markings, and traffic lights. This embedding space Ω is designed as a relatively low-dimensional space. Furthermore, the artificial neural network can also have been trained using this lower-dimensional embedding space Ω. Examples of such embedding spaces Ω include JoMase Embedding VectorNet, Global Graph Embedding, and MATF Embedding.

[0098] The recognition unit 6 can detect out-of-distribution (OOD) patterns within the embedding space Ω using a Mahalanobis distance applied to a distribution of in-distribution patterns, i.e., known patterns. The squared Mahalanobis distance of an ID pattern x is given by: D2(x)=(x−μ)T⋅Σ−1⋅(x−μ) where µ and Σ are the mean and covariance matrix, respectively, of the ID patterns within the embedding space Ω. Since the data are represented using the input space Ω, complex, context-dependent traffic situations can be easily detected.

[0099] For example, traffic situations involving a driver traveling against the lane markings (wrong-way driver) or perpendicular to the lane markings can be detected. For this, knowledge of the movement trajectories alone is insufficient, as the relative orientation of these trajectories with respect to the lane markings is also required, which is provided by the embedding space Ω.

[0100] If the recognition unit 6 detects such out-of-distribution (OOD) patterns, the consequence is that the reliability of the trajectory prediction module 4, or of the occurrence probabilities of the future trajectories determined by the trajectory prediction module 4, can be assumed to be lower than normal.

[0101] To determine reliability, the recognition unit 6 determines the deviation between out-of-distribution (OOD) patterns and in-distribution patterns. The reliability of the trajectory prediction module 4 or the probability of occurrence of the future trajectories can be used as the deviation itself, or it can be determined based on the deviation.

[0102] Furthermore, the calibrated artificial neural network of the trajectory prediction module 4 is only used if a recognition unit 6 does not detect any out-of-distribution (OOD) patterns in the previous movement trajectories and / or movement data assigned to the respective road user. This means that the calibrated model is only applied to the ID patterns where it is able to output reliable probabilities. This approach reduces the epistemic uncertainty, which is closely related to the parts of the feature space not covered by the training data.

[0103] It was found that applying out-of-distribution (OOD) patterns can degrade the trajectory prediction module 4 through probability calibration. Removing the out-of-distribution (OOD) patterns always results in an improvement through probability calibration. This approach reduces epistemic uncertainty.

[0104] The appropriately calibrated trajectory prediction module 4, or the artificial neural network, can then be used accordingly. This calibrated artificial neural network enables an improved trajectory prediction module 4 with respect to ID patterns.

[0105] This calibration / improvement of the trajectory prediction module 4 is independent of any standard improvement that can be made as a result of typical retraining of the trajectory prediction module or the artificial neural network.

[0106] The detection unit 6 can detect reliability in online mode, while the trajectory prediction module 4 can be calibrated, for example, in offline mode or not in real time.

[0107] Furthermore, motion planning module 5 is present. Information about out-of-distribution (OOD) patterns, or the strength of the deviation as a measure of reliability, is passed on to motion planning module 5. Based on the future trajectories with their associated probability of occurrence, as well as the information about the deviation and reliability from trajectory prediction module 4, motion planning module 5 generates an ego trajectory for the ego vehicle; that is, motion planning module 5 decides how the ego vehicle should move within the next few seconds.

[0108] Furthermore, the recognition unit 6 can locally store the Out-Of-Distribution (OOD) patterns with the subsequent sensor data and / or the subsequent motion trajectories, which can be used to improve the artificial neural network, or transmit them to the external server along with the Out-Of-Distribution (OOD) pattern.

[0109] The vehicle system 1b can also have remote control for external intervention in case of particularly severe deviation. Reference symbol list 101 Vehicle system according to the state of the art 102 Sensor system according to the state of the art 103 Environment module according to the state of the art 104 Trajectory prediction module according to the state of the art 105 Motion planning module according to the state of the art 1,1a,1b Vehicle system 2 Sensor system 3 Environment module 4 Trajectory Prediction Module 5 Motion Planning Module 6 Recognition unit 7 Sensor-OOD module 8 Remote Control Center Ω Embedding space F Wrong-way driver Pf1 Arrow kknN calibrated artificial neural network

Claims

[1] Vehicle system (1,1a,1b) for increasing the reliability of a prediction of future trajectories of road users in the vicinity of an ego vehicle comprising a sensor system (2) which is designed to detect the environment of the ego vehicle using sensors based on sensor data, and an environment module (3) which is designed to extract the road users as well as movement data of the road users from the past sensor data and which is further designed to generate a previous movement trajectory of the road users within a scene based on the recorded movement data, a trajectory prediction module (4) which is trained to generate one or more future trajectories for each detected road user based on the previous movement trajectory, each with an assigned probability of occurrence for the road user, wherein the trajectory prediction module (4) includes at least one trained machine learning method which is trained on training data as an in-distribution pattern (ID pattern) to generate one or more future trajectories based on a previous movement trajectory and to assign a probability of occurrence to each of the future trajectories, characterized by , that a recognition unit (6) is provided which is designed to recognize out-of-distribution (OOD) patterns present within a scene based on the previous motion trajectory and / or the motion data and / or to determine the deviations between out-of-distribution (OOD) patterns and in-distribution (ID) patterns, and wherein the recognition unit (6) is further designed to use the existing deviations as a reliability of the probability of occurrence of the future trajectories or to determine the reliability based on the deviation, wherein a motion planning module (5) is provided which is configured to generate an ego trajectory for the ego vehicle, based on one or more future trajectories, each with the associated probability of occurrence of the road users, as well as on the determined reliability of the probability of occurrence of the future trajectories, wherein a sensor-OOD module (7) is present and the sensor-OOD module (7) is configured to receive the sensor data and, at least in the presence of an out-of-distribution (OOD) pattern, to detect a semantic or non-semantic shift of the sensor data compared to the training data and the sensor-OOD module (7) has artificially generated attack data, wherein the sensor-OOD module (7) is configured to detect out-of-distribution (OOD) patterns based on the artificially generated attack data. [2] Vehicle system (1,1a,1b) according to claim 1, characterized by , that the detection unit (6) is designed to store the out-of-distribution (OOD) patterns with the subsequent sensor data and / or the subsequent motion trajectories. [3] Vehicle system (1,1a,1b) according to claim 2, characterized by, that the trajectory prediction module (4) is trained to train the machine learning algorithm using the stored out-of-distribution (OOD) patterns and the subsequent sensor data and / or the subsequent motion trajectory and / or wherein the vehicle system (1,1a,1b) has an interface and is trained to transmit the out-of-distribution (OOD) patterns with the subsequent sensor data to an external server. [4] Vehicle system (1, 1a, 1b) according to any of the preceding claims, characterized by , that the vehicle system (1,1a,1b) has a remote control for external control of the ego vehicle and an interface, and wherein the detection unit (6) is configured to transmit, in the event of a deviation above a predetermined threshold, this deviation to a remote control center (8) via the interface, for monitoring or controlling the ego vehicle by means of the remote control. [5] Vehicle system (1, 1a, 1b) according to any of the preceding claims, characterized by , that the vehicle system (1,1a,1b) has a storage unit with simulated and / or real Out-Of-Distribution (OOD) attack patterns, and the detection unit (6) is trained to compare the current Out-Of-Distribution (OOD) patterns with the simulated and / or real Out-Of-Distribution (OOD) attack patterns. [6] Vehicle system (1,1a,1b) according to claim 5, characterized by , that the detection unit (6) is trained to determine the deviation between the Out-Of-Distribution (OOD) patterns and the In-Distribution (ID) patterns based on the deviation between the current Out-Of-Distribution (OOD) patterns with the simulated and / or real Out-Of-Distribution (OOD) attack patterns. [7] Vehicle system (1,1a,1b) according to any of the preceding claims, characterized by, that the motion planning module (5) is designed to generate an ego trajectory for the ego vehicle, based on one or more future trajectories, each with its associated probability of occurrence, as well as on the reliability of the probability of occurrence of the future trajectories, and on a detected semantic or non-semantic shift of the sensor data. [8] Vehicle system (1, 1a, 1b) according to any of the preceding claims, characterized by , that the recognition unit (6) is designed to represent the previous movement trajectory of the road users and / or sensor data and / or within a scene in an embedding space (11) and to recognize the existing out-of-distribution (OOD) patterns within the embedding space (11). [9] Vehicle system (1,1a,1b) according to claim 8, characterized by, that the detection unit (6) is designed to detect the out-of-distribution (OOD) patterns within the embedding space (11) by applying a Mahalanobis distance to the distribution of the in-distribution patterns. [10] Vehicle system (1, 1a, 1b) according to any of the preceding claims, characterized by , that the vehicle system (1,1a,1b) is designed to extract and store ground-truth movement trajectories of the road users from the sensor data, wherein the trajectory prediction module (4) is designed to perform a calibration of the machine learning procedure with respect to the probability of occurrence, if required, based on the ground-truth movement trajectory of the respective road users and the corresponding assigned future trajectories with probability of occurrence. [11] Vehicle system (1,1a,1b) according to claim 10, characterized by, that the trajectory prediction module (4) is designed to perform the calibration with respect to the probability of occurrence using parametric or non-parametric methods. [12] Vehicle system (1,1a,1b) according to claim 11, characterized by , that the trajectory prediction module (4) is trained to use the calibrated machine learning method only when the recognition unit (6) does not detect any out-of-distribution (OOD) patterns in the previous movement trajectory and / or movement data and / or within a scene associated with the respective road user.

Images