Method for controlling a vehicle, method for providing an annotated road traffic scene data set and method for training a neural network

Abstract

A method for controlling a vehicle comprises: detecting a current road traffic scene in which the vehicle is located, specifying a plurality of possible future behaviors of the vehicle, predicting a plurality of possible future road traffic scenes by means of a neural network taking into account the detected current road traffic scene and the specified behaviors of the vehicle, determining a measure of an effect of a behavior of the vehicle on other road users for each of the plurality of possible future road traffic scenes and selecting a behavior of the vehicle for which the determined measure of an effect of the behavior of the vehicle on other road users fulfills a specified condition.

Description

[0001] The present disclosure relates to a method for controlling a vehicle, a method for providing an annotated road traffic scene dataset, and a method for training a neural network.

[0002] Constant caution and mutual consideration are the first principles of road traffic regulations. While this is taken for granted by human drivers, it is particularly difficult to implement for automated driving functions, as it cannot be expressed in clear specifications. Mutual consideration requires, in particular, assessing and appropriately considering the consequences of one's own compliant actions on other road users. It is important to note that one's own compliant behavior does not necessarily mean that other road users will not be placed in difficult or safety-critical situations as a result of one's own actions. This can be understood by considering a scenario in which a road user enters a motorway and is on the acceleration lane, while the vehicle with automated driving functions is in the right lane of the motorway.An automated driving function might, for example, decide to accelerate further and not allow the other road user to merge, which is legal under certain conditions. This could force the other road user to brake sharply. Furthermore, it could result in the driver arriving at the end of the acceleration lane at a low speed, making further merging onto the highway very difficult or, due to speed differences with other road users, potentially dangerous. It is therefore desirable to implement social behavior in road traffic through technology.

[0003] According to various embodiments, a method for controlling a vehicle is provided, wherein the method comprises: capturing a current road traffic scene in which the vehicle is located, specifying several possible future behaviors of the vehicle, predicting several possible future road traffic scenes using a neural network taking into account the captured current road traffic scene and the specified behaviors of the vehicle, determining a measure of the impact of a vehicle behavior on other road users for each of the several possible future road traffic scenes, and selecting a vehicle behavior for which the determined measure of the impact of the vehicle behavior on other road users satisfies a specified condition.

[0004] This method allows social behavior in road traffic to be technically implemented, enabling mutual consideration even in the absence of a human driver, such as in autonomous vehicles. Accordingly, the measure of the impact of a vehicle's behavior on other road users can be described as social cost. This can be expressed as a numerical value, with the given condition being either a minimum or a maximum of these values.

[0005] The possible future behaviors of the vehicle can include any behavior of the vehicle, such as future trajectories, acceleration processes, braking processes, turn signals, etc.

[0006] The respective measures for the impact of the vehicle's behavior on other road users in each of the possible future road traffic scenarios can be combined (e.g., weighted and summed according to a respective probability of occurrence).

[0007] Furthermore, it can be provided that, based on a single behavior of the vehicle, several possible behaviors of other road users are predicted. Accordingly, the measure of the impact of the vehicle's behavior on other road users can be determined for each possible future behavior of those other road users. These measures for the predicted future behaviors of other road users can also be combined (e.g., weighted and summed according to a probability of occurrence) to obtain a value for each behavior of the vehicle (e.g., an automated vehicle).

[0008] To provide a procedure that is runtime-efficient, it may be intended that both the prediction of the several possible future road traffic scenarios and the determination of the measure of the impact of the vehicle's behavior on other road users for each of the several possible future road traffic scenarios are carried out using the (e.g., the same) neural network.

[0009] This approach can involve predicting multiple possible future traffic scenarios and determining the measure of the vehicle's behavior's impact on other road users for each scenario using a multi-task learning approach. Such a multi-task learning approach allows for offline calculation of the vehicle's impact on other road users. This has the advantage that underlying computational models can be arbitrarily complex without negatively impacting runtimes during online operation.

[0010] Furthermore, the procedure may include determining a measure of the vehicle's comfort, driving dynamics, safety, and compliance, whereby the selection of a vehicle behavior is carried out under the condition that the measure of the vehicle's impact on other road users and the measure of comfort, driving dynamics, safety, and compliance meet one or more predefined conditions. This allows for consideration of other needs of the vehicle's occupants besides comfort, such as reaching a destination efficiently and safely.

[0011] In one or more embodiments, determining the measure of the impact of the vehicle's behavior on other road users may include one, several, or all of the following steps: • Determine one or more criticality metrics pairwise between road users and aggregate across all pairs as well as all criticality metrics (aggregation can be done, for example, by weighted averaging, maximizing or minimizing), • Determining a measure of the dynamics of road users that the vehicle has acquired knowledge of (e.g., through sensor detection, vehicle-to-vehicle communication, vehicle-to-infrastructure communication, etc.), based on trajectories predicted for these road users (this measure is also referred to as dynamic costs, where the dynamic costs may include, for example, expected acceleration and / or jerk), • Weighting vehicle interactions such that direct and indirect vehicle interactions are weighted differently (direct vehicle interactions are interactions that are directly influenced by the vehicle, whereas indirect vehicle interactions are interactions that only arise indirectly under the assumption of a certain reaction from another road user to the vehicle), • Weighting to balance the impacts on road users (If the costs (measures) are equally distributed among road users, the balance is perfect and the costs are zero. The greater the cost differences, the worse the balance and the higher the cost of the balance. In this way, the procedure prevents individual road users from being particularly disadvantaged).

[0012] In another aspect, a method for providing an annotated road traffic scene dataset is provided, wherein the method includes: providing a road traffic scene dataset (e.g., the inD dataset from J. Bock et al.: “The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections”, 2020 IEEE Intelligent Vehicles Symposium (IV), October 20-23, 2020. Las Vegas, USA or the nuScenes dataset from H. Caesar et al.: “nuScenes: A multimodal dataset for autonomous driving”, CVPR 2020, Computer Vision Foundation), which represents several present or past or present and past road traffic scenes and their respective future developments, determining a respective measure for an impact of a given vehicle's behavior on other road users for the future development of each road traffic scene in the road traffic scene dataset and generating the annotated road traffic scene dataset by assigning the respective measures for an impact of the given vehicle's behavior on other road users to the corresponding road traffic scenes in the road traffic scene dataset.

[0013] An annotated road traffic scene dataset is generated according to this procedure based on a measure of the impact of a vehicle's behavior on other road users for several possible future road traffic scenes and can therefore serve as a basis for the previously described procedure for controlling a vehicle or for training the neural network used in this control procedure.

[0014] Determining the respective measures for the impact of the given vehicle's behavior on other road users can, as described above in the control procedure, involve one, several, or all of the following steps: determining one or more criticality metrics pairwise between road users and aggregating across all pairs and all criticality metrics; determining a measure for the dynamics of road users known to the vehicle, based on trajectories predicted for these road users; weighting vehicle interactions such that direct and indirect interactions are weighted differently; and weighting a balance of the impacts on the road users. The preceding supplementary explanations also apply here.

[0015] In another aspect, a method for training a neural network is provided, which is set up to predict, from a given road traffic scene and several given behaviors of a given vehicle located in the road traffic scene, several possible future road traffic scenes and a respective measure of the impact of a behavior of the given vehicle on other road users for each of the several possible future road traffic scenes, wherein the neural network is trained with an annotated road traffic scene dataset provided according to a previously described method.

[0016] This enables a multi-task learning approach where the respective measures (social costs) can be calculated offline. This allows the use of complex calculation models that do not affect runtime in online operation.

[0017] In one or more embodiments, a data processing system is provided which is configured to carry out a previously described procedure.

[0018] In one or more embodiments, a computer program is provided with instructions which, when executed by a processor, cause the processor to perform a previously described procedure.

[0019] In one or more embodiments, a computer-readable medium is provided which stores instructions which, when executed by a processor, cause the processor to perform a previously described procedure.

[0020] In the drawings, similar reference numerals generally refer to the same parts in the different views. The drawings are not necessarily to scale, with the emphasis generally placed on illustrating the principles of the invention. Various aspects are described in the following description with reference to the drawings below. Fig. Figure 1 shows a vehicle. Fig. Figure 2 shows a flowchart illustrating an example procedure for controlling a vehicle. Fig. Figure 3 illustrates a method for controlling a vehicle according to a first embodiment. Fig. Figure 4 illustrates exemplary steps for calculating social costs according to one embodiment. Fig. Figure 5 illustrates a method according to a second embodiment. Fig. Figure 6 illustrates a method according to a third embodiment.

[0021] The following detailed description refers to the accompanying drawings, which illustrate specific details and aspects of this disclosure in which the invention can be implemented. Other aspects may be used, and structural, logical, and electrical modifications may be made without altering the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0022] Several examples are described in more detail below.

[0023] Fig. Figure 1 shows a vehicle 101 according to an exemplary embodiment. The vehicle 101 can be designed as a motor vehicle, such as a car or truck.

[0024] As in Fig. As shown in Figure 1, the vehicle 101 can be equipped with a vehicle control unit (e.g., an Electronic Control Unit (ECU)) 102. The vehicle control unit 102 can include: data processing components, such as a processor (e.g., a CPU (central processing unit)) 103, and memory 104 for storing control software (a computer program) 107, according to which the vehicle control unit 102 operates, and data 109, which is processed by the processor 103. The processor 103 executes the control software 107.

[0025] The control software 107 can contain instructions which, when executed by the processor 103, cause the processor 103 to perform driver assistance functions or even to control the vehicle autonomously.

[0026] The control software 107 can be transferred to the vehicle 101, for example, from a computer system 105, via a network 106, or using a storage medium such as a memory card. This can occur, for example, as part of an update process in which the control software 107 is updated to newer versions or a current version.

[0027] The control software 107 can be implemented using a neural network 108 and can be trained by machine learning (ML). This means that the control software 107 can implement one or more ML models that are trained based on training data, in this example from the computer system 105. The computer system 105 can implement an ML training algorithm for training the one or more ML models.

[0028] The control software 107 can determine control actions for the vehicle 101 (such as steering, braking, etc.) from input data 109 available to it, which contains information about the vehicle 101's environment or from which it derives information about the environment (such as by detecting other road users, e.g., other vehicles). This input data 109 includes, for example, sensor data, such as from a camera, lidar sensor, and / or radar sensor of the vehicle 101, and / or information obtained through communication with other vehicles or a traffic information infrastructure.

[0029] A flowchart of a procedure 200 for controlling a vehicle, for example the one in Fig. Vehicle 101 shown in 1, according to one or more embodiments, is in Fig. 2 shown.

[0030] As in Fig. As shown in Figure 2, the procedure 200 can include: capturing 202 a current road traffic scene in which the vehicle 101 is located, specifying 204 several possible future behaviors, e.g., trajectories, of the vehicle 101, predicting 206 several possible future road traffic scenes using a neural network, taking into account the captured current road traffic scene and the specified behaviors of the vehicle 101, determining 208 a measure of the impact of a behavior of the vehicle 101 on other road users for each of the several possible future road traffic scenes, and selecting 210 a behavior of the vehicle 101 for which the determined measure of the impact of the behavior of the vehicle 101 on other road users fulfills a specified condition.

[0031] The based on the in Fig. The process shown in the flowchart 2 offers the possibility to technically record, influence and realize mutual consideration in road traffic to a desired extent.

[0032] This process is described in detail below using various exemplary embodiments.

[0033] Fig. Figure 3 schematically shows the sequence of a procedure 300 for controlling a vehicle, such as the one in Fig. Vehicle 101 shown in Figure 1, according to a first embodiment. Vehicle 101 is hereinafter referred to as Ego-Vehicle 101.

[0034] In procedure 300, based on a current environment model of the ego-vehicle 101, including the track history of the currently known road users (i.e., those visible to the ego-vehicle 101) (at 302), the current or past state of the ego-vehicle 101, and a currently intended behavior of the ego-vehicle 101 (e.g., the current trajectory candidate, i.e., the intended trajectory of the ego-vehicle 101) (at 304), the current traffic scene is then projected into the future using a neural network, i.e., predicted (at 306). A characteristic of scene prediction is that it is multimodal, i.e., that it allows for different possible developments of the current scene.

[0035] In the next step, the possible developments of the traffic scene, i.e., the trends, are used to determine a measure of the impact of the behavior of Ego-Vehicle 101 on other road users for each of the several possible future road traffic scenarios. This measure of the impact of Ego-Vehicle 101's behavior on other road users is subsequently referred to as the social cost.

[0036] Determining the social costs (308) can involve one, several, or all of the costs in Fig. The 4 steps shown are shown. • In step 402, one or more criticality metrics are determined pairwise between road users and aggregated across all pairs and all criticality metrics. Aggregation can be performed by weighted averaging, maximization, or minimization. • In step 404, a measure for the dynamics of road users, about which the ego-vehicle 101 has acquired knowledge (e.g., through sensor detection, vehicle-to-vehicle communication, vehicle-to-infrastructure communication, etc.), is determined based on trajectories predicted for the known road users. This measure is also referred to as dynamic costs. The dynamic costs can include, for example, the expected acceleration and / or the expected jerk. Furthermore, cost terms for criticalities with static elements of a given traffic scene can be considered. • In step 406, interactions of the ego-vehicle 101 are weighted such that direct and indirect interactions of the ego-vehicle 101 are weighted differently. Direct interactions of the ego-vehicle 101 are interactions that are directly influenced by the ego-vehicle. Indirect interactions of the ego-vehicle 101 are interactions that only arise indirectly, assuming a specific reaction from another road user to the ego-vehicle 101. • In step 408, the impacts on road users are weighted. If the costs are evenly distributed among all road users, the balance is perfect and the costs are zero. The greater the cost differences, the worse the balance and the higher the cost of maintaining it. In this way, the procedure prevents individual road users from being disproportionately disadvantaged.

[0037] Steps 402 to 408 thus provide (social) cost terms. When calculating social costs, the preceding cost terms can be calculated for each mode and / or each sampling point of the scene prediction.

[0038] The individual cost terms are weighted with individual weights w1, w2, w3, w4 and summed over both the sampling points and the modes. The individual weights w1, w2, w3, w4 result from a set of rules that are part of the system design. For example, the rules can be used to weight the modes according to their probability of occurrence. Furthermore, the weights w1, w2, w3, w4 can be weighted less heavily for time steps far in the future than for those that are imminent. This accounts for the increasing prediction error with a longer forecast horizon. In simpler implementations, the respective weights w1, w2, w3, w4 are defined as constants by a rule.

[0039] As further in Fig. As shown in Figure 3, in step 310 a measure (cost) for the comfort, driving dynamics, safety, and rule-compliant behavior of the ego-vehicle 101 is determined. These costs are weighted and added to the social costs calculated in step 308 (at 312). This yields a total measure (total cost) for each future road traffic scenario, depending on a possible future behavior of the ego-vehicle 101. This total measure serves as the basis for selecting a future behavior of the ego-vehicle; that is, the future behavior (e.g., trajectory) of the ego-vehicle 101 that fulfills a predefined condition is selected.

[0040] The selection of the future behavior of the ego vehicle 101 takes place in step 314. In an exemplary embodiment, the future behavior of the ego vehicle 101 with the lowest total costs can be selected.

[0041] In method 300 according to the first embodiment, the neural network is called multiple times for each behavior of the ego-vehicle 101, and the predicted possible temporal developments of the scene are evaluated with regard to social costs. Thus, in the method according to the first embodiment, the traffic scene prediction and the determination of the social costs are performed separately.

[0042] Fig. Figure 5 illustrates a method 500 for controlling a vehicle according to a second embodiment.

[0043] In the second embodiment of Method 500, multimodal traffic scene prediction is combined with social cost estimation in a multi-task learning approach. Method 500 comprises a preprocessing process 500-1 and a subsequent training process 500-2.

[0044] In the preprocessing process 500-1, a road traffic scene dataset 502 is first provided (e.g., the inD dataset from J. Bock et al.: “The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections”, 2020 IEEE Intelligent Vehicles Symposium (IV), October 20-23, 2020, Las Vegas, USA, or the nuScenes dataset from H. Caesar et al.: “nuScenes: A multimodal dataset for autonomous driving”, CVPR 2020, Computer Vision Foundation), which represents several present or past, or present and past, road traffic scenes and their respective future developments, i.e., road traffic scenes at different times t0...t k , where k is a natural number. For the future development of each road traffic scene in the road traffic scene dataset 502, corresponding measures (social costs C) are used. soc ) for an effect of a behavior T egoThe behavior of the ego vehicle 101 towards other road users was determined (at 504). The behavior T ego The ego vehicle 101, for example, can be extracted from the road traffic scene dataset 502. Finally, an annotated road traffic scene dataset 506 is created by assigning the social cost C. soc generated for the corresponding road traffic scenes of the road traffic scene data set 502.

[0045] The training process 500-2 is a multi-task learning process performed on the basis of the annotated road traffic scene dataset 506. More precisely, the training process 500-2 is a process for training a neural network 508, which is configured to generate, from a given road traffic scene and several given behaviors of the ego-vehicle 101 located in the road traffic scene, several possible future road traffic scenes and a respective measure of the impact of a behavior of the ego-vehicle 101 on other road users, i.e., social costs C. soc , for each of the several possible future road traffic scenes, where the neural network 508 is trained with the annotated road traffic scene dataset 506.

[0046] As in Fig. As shown in Figure 5, the neural network 508 comprises a backbone network 508-1, a first prediction head (first subnetwork) 508-2, and a second prediction head (second subnetwork) 508-3. The first prediction head 508-2 estimates the multimodal road traffic scene development, and the second prediction head 508-3 estimates the social costs associated with each road traffic scene.

[0047] In this approach, social costs are not calculated separately based on traffic scenario prediction, but are directly estimated using multi-task learning from the 508 neural network. While training for this implementation is more complex, it eliminates an additional, computationally intensive runtime step.

[0048] The advantage of this embodiment is therefore that the social costs can now be calculated offline, and thus the models for calculation, in particular the criticality metrics, can become almost arbitrarily complex without impairing real-time capability.

[0049] Fig. Figure 6 illustrates a method 600 for controlling a vehicle according to a third embodiment.

[0050] Method 600 is a method for training a neural network 604 using an annotated road traffic scene dataset 602, such as the previously described annotated road traffic scene dataset 506. The neural network 604 has a backbone network 604-1, a first prediction head 604-2, and a second prediction head 604-3. The first prediction head 604-2 estimates the multimodal road traffic scene development, and the second prediction head 604-3 estimates the social costs associated with each road traffic scene.

[0051] In the third embodiment of method 600, the training is designed such that the behavior of the ego vehicle is not incorporated into the computationally intensive backbone network 604-1, which is used to encode the other input data. This means that the backbone network 604-1 only needs to be called once per planning cycle. The behavior candidates to be evaluated, together with the latent features (encoded input data) of the backbone network 604-1, then serve as inputs for the first prediction head 604-2 and the second prediction head 604-3, which are subnetworks that perform the multimodal traffic scene prediction and the determination of social costs. These subnetworks are then called for each behavior candidate, i.e., multiple times per planning cycle.

[0052] The procedures described above can be performed by one or more computers with one or more data processing units. The term "data processing unit" can be understood as any type of entity that enables the processing of data or signals. The data or signals can, for example, be processed according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit can comprise or be composed of an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit, a programmable gate array (FPGA), or any combination thereof.Any other method for implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the process steps described in detail herein may be executed (e.g., implemented) by a data processing unit through one or more special functions performed by the data processing unit.

[0053] According to various explanations, the process is therefore implemented primarily using computers. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Zitierte Nicht-Patentliteratur

[0000] J. Bock et al.: „The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections“, 2020 IEEE Intelligent Vehicles Symposium (IV), October 20-23, 2020. Las Vegas, USA

[0012] H. Caesar et al.: „nuScenes: A multimodal dataset for autonomous driving“, CVPR 2020, Computer Vision Foundation [0012, 0044] J. Bock et al.: „The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections“, 2020 IEEE Intelligent Vehicles Symposium (IV), October 20-23, 2020

[0044]

Claims

[1] Method for steering a vehicle (101) wherein the method (200) comprises: Capturing (202) a current road traffic scene in which the vehicle (101) is located, Specifying (204) several possible future behaviors of the vehicle (101), Predictions (206) of several possible future road traffic scenes using a neural network taking into account the recorded current road traffic scene and the specified behaviors of the vehicle (101), Determine (208) a measure of the impact of a vehicle's (101) behavior on other road users for each of the several possible future road traffic scenarios and Selecting (210) a behavior of the vehicle (101) for which the determined measure of an effect of the behavior of the vehicle (101) on other road users satisfies a predefined condition. [2] Method according to claim 1, wherein both the prediction (206) of the multiple possible future road traffic scenes and the determination (208) of the measure of the effect of the behavior of the vehicle (101) on other road users are carried out for each of the multiple possible future road traffic scenes using the neural network. [3] Method according to claim 2, wherein the prediction (206) of the multiple possible future road traffic scenes and the determination (208) of the measure of the impact of the behavior of the vehicle (101) on other road users for each of the multiple possible future road traffic scenes are realized using a multi-task learning approach. [4] Method according to any one of claims 1 to 3, further comprising: Determining a measure of comfort, driving dynamics, safety and compliance of the vehicle (101), wherein the selection of a behavior of the vehicle (101) is carried out under the condition that the measure of an impact of the behavior of the vehicle (101) on other road users and the measure of comfort, driving dynamics, safety and compliance satisfy one or more predefined conditions. [5] Method according to any one of claims 1 to 4, wherein determining the measure of an impact of the behavior of the vehicle (101) on other road users comprises one, several or all of the following steps: Determine one or more criticality metrics pairwise between road users and aggregate across all pairs as well as all criticality metrics, Determining a measure of the dynamics of road users of which the vehicle (101) has acquired knowledge, based on trajectories predicted for these road users, Weighting of vehicle interactions (101) such that direct vehicle interactions (101) and indirect vehicle interactions (101) are weighted differently, and Weighing the impacts on road users to ensure a balance. [6] Method (500-1) for providing an annotated road traffic scene dataset (506), wherein the method (500-1) comprises: Providing a road traffic scene dataset (502) representing multiple present or past or present and past road traffic scenes and their respective future developments, Determine (504) a respective measure (C soc) for an effect of the behavior of a given vehicle on other road users for the future development of each road traffic scene of the road traffic scene dataset (502) and Generating the annotated road traffic scene dataset (506) by assigning the respective dimensions (C soc ) for an effect of the behavior of the specified vehicle on other road users to the corresponding road traffic scenes of the road traffic scene data set (502). [7] Method (500-1) according to claim 6, wherein determining a respective measure (C soc ) for the effect of the behavior of the specified vehicle on other road users exhibits one, several or all of the following steps: Determine one or more criticality metrics pairwise between road users and aggregate across all pairs as well as all criticality metrics, Determining a measure of the dynamics of road users that the vehicle has acquired knowledge of, based on trajectories predicted for these road users, Weighting of vehicle interactions such that direct and indirect vehicle interactions are weighted differently, and Weighing the impacts on road users to ensure a balance. [8] Method (500-2) for training a neural network (508) configured to predict, from a given road traffic scene and several given behaviors of a given vehicle located in the road traffic scene, several possible future road traffic scenes and a respective measure of the impact of a behavior of the given vehicle on other road users for each of the several possible future road traffic scenes, wherein the neural network (508) is trained with an annotated road traffic scene data set (506) provided according to a method according to claim 6 or 7. [9] Data processing system (102) configured to carry out a method according to any one of claims 1 to 8. [10] Computer program with instructions which, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 8. [11] Computer-readable medium storing instructions which, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 8.

Images