Computer-implemented method and system for planning the behavior of a vehicle in a traffic scene

Abstract

A computer-implemented method and system (100) for behavior planning of a vehicle (1) in a traffic scene is proposed, whereby the behavior planning pursues a predetermined driving goal. The system (100) comprises at least one perception level for aggregating scene-specific information (10) and for generating at least one scene representation (11) of the traffic scene, a neural network (110) that performs strategic behavior planning based on the scene representation (10, 11) generated by the perception level, and a downstream planning component (120) that performs detailed behavior planning based on the strategic behavior planning. According to the invention, the neural network (110) is trained to generate, as a result of strategic behavior planning, at least one geometric behavioral specification for the vehicle (1) in the given traffic scenario. To this end, the neural network (110) identifies at least one go area (3; 5) that the vehicle (1) may or should pass through to pursue the specified destination, and / or at least one no-go area (4) that the vehicle (1) should avoid when pursuing the specified destination. The downstream planning component (120) is configured to generate, as a result of detailed behavior planning, at least one trajectory (12) for the vehicle (1), taking into account the at least one geometric behavioral specification from the strategic behavior planning.

Description

State of the art

[0001] The invention relates to a computer-implemented method and a computer-implemented system for planning the behavior of a vehicle in a traffic scenario, wherein the behavior planning pursues a predetermined driving goal. The driving goal can be, for example, a destination, but also a destination route or a combination of destination and destination route, such as "Drive from A to B and use only country roads".

[0002] The task of autonomous driving is to control a self-driving vehicle based on aggregated, scene-specific information, particularly sensor data such as radar, lidar, and RGB camera signals, in such a way as to reach a destination as quickly, comfortably, and safely as possible. This includes, among other things, adhering to traffic regulations and avoiding collisions with infrastructure elements and / or other road users. This driving task can be subdivided into the subtasks of perception, prediction, planning, and control.

[0003] The task of perception is to extract relevant information from the aggregated scene-specific information, such as location and status information for static and dynamic objects in the traffic scene, especially other road users. Furthermore, lane markings and traffic signs can be identified at the perception level and compared with map data. In this way, an environment model is generated as a scene representation of the current traffic situation.

[0004] Prediction is used to estimate the future development of the traffic scene, especially the behavior of other participants in the traffic scene and of dynamic objects.

[0005] The planning process uses the environmental model and the prediction of the future development of the traffic scene to plan the future behavior of the ego vehicle.

[0006] Planning methods known from practice can essentially be divided into three categories: classical planning methods – these include, in particular, rule-based, sampling, tree-search, and optimization-based planning methods; learned planning methods; and hybrid planning methods, which are implemented as a combination of classical and learned methods. Among learned methods, the use of machine learning, especially deep learning (DL), has become the de facto standard in recent years, as it allows for the inclusion of diverse contextual information and, in particular, interactions between participants in a traffic scenario in the planning process.

[0007] Planning is usually structured hierarchically. A strategic behavioral planner makes abstract (high-level) decisions, which are then implemented by a downstream planning component within the framework of detailed planning. Strategic behavioral planning is generally carried out less frequently than the subordinate detailed planning. The downstream planning component then solves the planning tasks that arise from the strategic behavioral planning.

[0008] As a result of the planning, and especially the detailed planning, one or more trajectories are often generated that are suitable for the vehicle. Each trajectory includes position data, possibly together with vehicle state data, for a predefined number of consecutive time points. The state data typically describes the vehicle's state of motion, such as speed, acceleration, and / or orientation, at the respective time.

[0009] The planning result is then implemented using a controller by appropriately controlling the vehicle's actuators. Trajectory data proves particularly advantageous for this, as it can usually be directly incorporated into the control system.

[0010] The invention is based on a method and a system as described in the German unpublished patent application 102024203268.

[0011] Accordingly, at least one scene representation of the given traffic scene is generated based on aggregated, scene-specific information. This can involve aggregated sensor data, a scene representation in the form of latent features, or a derived environmental model of the given traffic scene. Based on this scene representation, strategic behavior planning is initially carried out using at least one neural network. Then, based on this strategic behavior planning, detailed behavior planning is performed using at least one subsequent planning component.

[0012] German patent application 102024203268 proposes using a text-based neural network, in particular a Large Language Model (LLM), for strategic behavioral planning and performing subsequent detailed planning with the help of a rule-based planning component. The qualitative behavioral suggestion of the LLM is thus quantitatively implemented by the downstream rule-based planning component in a reliable and user-friendly manner, as is appropriate to the capabilities of the rule-based planning component.

[0013] Using a text-based neural network for strategic behavior planning requires "translating" the scene representation into text queries, which can then be used as input for the network. The result of strategic behavior planning is the generation of text-based behavioral recommendations, which also need to be "translated" so they can be implemented by the downstream planning component. Disclosure of the invention

[0014] According to the invention, it is proposed that, within the framework of strategic behavior planning, at least one geometric behavior specification for the vehicle in the given traffic scenario is generated by identifying at least one "go" area that the vehicle may or should pass through to pursue the specified destination, and / or by identifying at least one "no-go" area that the vehicle should avoid when pursuing the specified destination. As a result of the detailed behavior planning, at least one trajectory for the vehicle is then generated, taking into account the at least one geometric behavior specification from the strategic behavior planning.

[0015] These measures enable the use of any unimodal or multimodal neural network for strategic behavioral planning. Unimodal or multimodal here refers to the modalities of the neural network's input data. Thus, input for the neural network can include not only text data, but also any perceptual data, such as sensor data from radar, lidar, video, ultrasound, audio sensors, and / or images, and / or data from an environmental model, as well as perception and prediction outputs. The measures according to the invention also allow the use of any downstream planning component for detailed planning or for the implementation of the results of the strategic behavioral planning.

[0016] According to the invention, it has been recognized that every type of scenario or given traffic scene can be analyzed with regard to go and no-go areas, and that therefore a strategic behavior plan in the form of geometric behavior specifications for the vehicle can be generated for every type of scenario or given traffic scene. Furthermore, it has been recognized according to the invention that geometric behavior specifications can also be very easily taken into account in the detailed planning when planning trajectories, which are based on position / time coordinates.

[0017] In a preferred embodiment of the invention, a unimodal or multimodal deep learning (DL) foundation model is used for strategic behavior planning. Such a foundation model is very large and has been pre-trained with extremely large datasets, usually in a self-supervised manner. Both the architecture of a foundation model and its training are generally task-independent. Most foundation models are based on transformer architectures. Overall, foundation models are characterized by a very good, global context understanding compared to task-specifically trained neural networks. Therefore, based on suitable input data, foundation models can comprehend entire scenes at runtime and reliably solve tasks within the overall context of such a scene.

[0018] These capabilities of a foundation model are used here for strategic behavior planning, which facilitates sound, situation-appropriate driving maneuver decisions. The shortcomings of foundation models regarding geometric understanding, safety-relevant aspects of the traffic scene, and the physical understanding of vehicle dynamics and vehicle specifics are compensated for by combining them with a downstream classical or deep learning-based planning component. This downstream component possesses a good understanding of the vehicle's physical capabilities, the geometry of the traffic scene, the necessary safety-relevant distances, and so on. It proves particularly advantageous if the downstream planning component also ensures compliance with the necessary traffic regulations. Overall, this leads to safe, consistent, and human-like driving behavior.

[0019] As previously mentioned, strategic behavior planning generates at least a geometric behavioral guideline for the vehicle in the given traffic scenario. This involves identifying at least one "go" zone, which the vehicle is permitted or should pass through to pursue its predetermined destination, and / or at least one "no-go" zone, which the vehicle should avoid while pursuing its predetermined destination.

[0020] In an advantageous embodiment of the invention, a sequence of hit points is generated as a geometric behavioral guideline within the framework of strategic behavior planning. Each hit point is defined by spatial coordinates and a time value and / or at least one state parameter for the vehicle. This parameter can, in particular, be the vehicle speed, acceleration, and / or orientation. Each hit point defined in this way represents a go-zone that the vehicle should pass through while pursuing its destination. Accordingly, the hit points form support points for the at least one trajectory that is generated within the framework of the subsequent detailed behavior planning.Since strategic behavioral planning is limited to high-level maneuver decisions, the number of hit points per time interval is significantly lower than the number of trajectory points generated during detailed planning for the same time interval.

[0021] In a further advantageous embodiment of the invention, a sequence of hit regions is provided as a geometric behavioral specification within the framework of strategic behavior planning. Each hit region is defined by a location in the form of a polygon and a time interval and / or an interval of at least one state parameter for the vehicle. Again, these parameters are in particular the vehicle speed, the vehicle acceleration, and / or the vehicle orientation. Each hit region defined in this way represents a go-area that the vehicle may or should pass through while pursuing its destination. The at least one trajectory generated within the framework of the subsequent detailed behavior planning should then pass through at least a portion of these hit regions or even lie exclusively within them.

[0022] In a particularly advantageous embodiment of the invention, at least one geometric behavioral specification is provided in the form of zones located in the traffic scene, to which semantic information about the possible behavior of the vehicle in the respective zone is assigned. This semantic information includes, in particular, conditions under which a zone may or may not be used. The possible behavior of the vehicle is described using at least one state parameter, in particular speed, acceleration, and / or orientation.

[0023] In this embodiment of the invention, the current traffic scene is analyzed using the neural network or foundation model to reduce the traffic scene to a combination of situation-dependent zones, i.e., geometric zones located within the traffic scene. In addition to the situation-dependent zones, the neural network or foundation model provides semantic information in the form of restrictions to which the vehicle is subject within these zones, or conditions that the vehicle must fulfill to reach the next desired state. The result of this type of strategic behavior planning is also referred to as geometric behaviors.

[0024] According to the invention, strategic behavior planning is based on the scene representation of the given traffic situation. Advantageously, a prediction of the future development of the given traffic situation can also be taken into account.

[0025] This applies equally to the detailed behavioral planning, which, according to the invention, is based on the strategic behavioral planning. Advantageously, the scene representation of the given traffic scene and / or a prediction of the future development of the given traffic scene are also taken into account.

[0026] As mentioned previously, both a classical and a machine learning-based planning component can be used for subsequent detailed behavior planning. Crucially, this planning component must consider the physical characteristics and capabilities of the vehicle, as well as the scene geometry (i.e., the given distances and angles), during detailed behavior planning. This ensures that the abstract planning decisions of the strategic behavior planning are translated into physically feasible and safe trajectories.

[0027] These trajectories can therefore be generated using rule-based, optimization-based, sampling-based, tree-search-based, or machine learning (ML) methods. In these cases, the at least one geometric behavioral requirement of the strategic behavioral planning is advantageously considered as a selection criterion or as an optimization criterion when generating the at least one trajectory. drawing

[0028] The computer-implemented method according to the invention for planning the behavior of a vehicle and the corresponding computer-implemented system are explained in more detail below with reference to exemplary embodiments and advantageous further developments in conjunction with the figures. Fig. Figure 1 illustrates a first embodiment of a behavior planning system according to the invention and Fig. 2 illustrates an advantageous further training of the in Fig. 1 of the depicted system 100 for behavior planning. Description of exemplary implementations

[0029] The block diagram of Fig. Figure 1 illustrates the interaction of the individual components of a computer-implemented system 100 according to the invention for planning the behavior of a vehicle 1 in a traffic scenario, wherein the behavior planning pursues a predetermined driving goal. The vehicle 1 is also referred to below as the ego vehicle.

[0030] System 100 comprises a perception level (not shown in detail here) for aggregating scene-specific information 10 and generating at least one scene representation 11 of the given traffic scene. The starting point for behavioral planning is always the state of a traffic scene at a planning point in time, and in particular the state of all static and dynamic objects and participants in the traffic scene at that time. The state of the traffic scene is described by scene-specific information that is aggregated from various information sources at the planning point or over a certain period of time before and up to the planning point. These information sources can be vehicle-integrated sensors, such as lidar sensors, radar sensors, and / or RGB cameras installed on the ego vehicle, or external sensors, such as...Lidar sensors, radar sensors, and / or RGB cameras installed in or on infrastructure elements or other road users. Further information sources include stored map information, possibly along with traffic regulations, as well as queryable weather and road condition information, traffic situation information, etc. The information from the various information sources is aggregated and processed by the perception level to generate at least one scene representation. The aggregated scene-specific information itself already constitutes a scene representation. However, with the help of machine learning components, this information can also be further processed, for example, into a scene representation in latent space. Based on this, an environment model can also be generated as a scene representation, for example, in the form of bird's-eye view images of the traffic scene, object lists, and / or occupancy grids.Furthermore, the results of a prediction of the future development of the traffic scene can be taken into account when generating such an environmental model.

[0031] According to the invention, the system 100 comprises a neural network 110 for strategic behavior planning. The input for the neural network 110 consists of at least one scene representation generated by the perception level. Here, the neural network 110 is provided with both the aggregated scene-specific information 10 and an environment model 11 generated from it. For the sake of completeness, it should be noted that the input of the neural network 110 can also be preprocessed and / or fused by another machine learning component in order to transform the scene-specific information into the input representation required for the neural network 110.

[0032] In the preferred embodiment of the invention described here, the neural network 110 is a deep learning base model, which will be referred to as the Foundation Model in the following. The Foundation Model 110 was pre-trained with a very large, task-independent dataset and then re-trained for strategic planning in automated driving, a process known as fine-tuning. This fine-tuning can be achieved through supervised learning. For this purpose, training data representing the desired output of the strategic planning is used. Alternatively, such a Foundation Model can also be re-trained through reinforcement learning in a specially designed simulation that also simulates the subsequent detailed planning.In this case, a loss function is optimized that, based on the simulated trajectories, considers both the detailed behavioral planning at the Motion Planning level and the strategic behavioral planning of the Foundation Model. This allows the Foundation Model to independently learn the required output of the strategic behavioral planning. The Foundation Model 110 can process input data from one or more modalities. It is particularly advantageous if the Foundation Model can utilize at least some of the different modalities of the aggregated scene-specific information.

[0033] According to the invention, the neural network, here the Foundation Model 110, is trained such that, as a result of strategic behavior planning, it generates at least one geometric behavioral guideline for the ego-vehicle 1 in the given traffic scene. For this purpose, at least one "Go" zone 3 or 5 is identified, which the ego-vehicle 1 may or should pass through to pursue the specified destination. Alternatively or additionally, at least one "No-Go" zone 4 is identified, which the ego-vehicle 1 should avoid when pursuing the specified destination.

[0034] This is illustrated by the schematic representations of a traffic scene in section 9. The left half of section 9 depicts a traffic scene with an ego-vehicle 1 moving towards an obstacle 2 in the right lane of a two-lane road. Foundation Model 110 analyzed this traffic scene and identified and located several Go zones 3 within it, shown here with hatching. A No-Go zone was also identified in the immediate vicinity of the obstacle 4. The output of Foundation Model 110 shown in the left half of section 9 corresponds to the concept of Geometric Behaviors described earlier.

[0035] The right half of partial view 9 illustrates a different type of geometric behavior specification for the traffic scene described above, involving ego-vehicle 1 and obstacle 2. Foundation Model 110 has generated a list of hit points 5, which can be interpreted as go areas, since they should be traversed by ego-vehicle 1 while pursuing its destination and consequently form support points for the planned trajectory of ego-vehicle 1. Here, a hit point is a tuple (x, y, t, v) of Cartesian coordinates x, y, an associated time t, and velocity v. The concept of hit points could also be extended to a concept of hit regions. The Cartesian point (x, y) of the hit point is replaced by a polygon P for hit regions. Time intervals T and velocity intervals V are also derived from the individual velocity and time values.

[0036] Furthermore, according to the invention, the system 100 comprises at least one planning component 120 downstream of the neural network or foundation model 110, which performs detailed behavior planning based on the strategic behavior planning of the neural network 110. This downstream planning component is configured to generate at least one trajectory 12 for the vehicle, taking into account the at least one geometric behavior specification of the strategic behavior planning.

[0037] It is essential that in the downstream low-level motion planning stage, specific boundary conditions relating to the vehicle – dynamics and dimensions – and traffic regulations are taken into account during detailed planning. The input of the corresponding planning component is not limited to the output of the strategic behavioral planning. Without any restriction of generality, all input data of the Foundation Model 110 can also be used by the downstream planning component 120.

[0038] When using a sampling-based planning component 120, the geometric behavioral specifications of the strategic behavioral planning can be taken into account through cost conditions.

[0039] When using optimization-based planning components, such as model predictive control, black box optimization, etc., the geometric behavioral requirements of the high-level planning can be taken into account through appropriate boundary conditions.

[0040] In control-based planning components, compliance with the geometric behavioral specifications of the strategic behavioral planning is ensured by open- and closed-loop control elements.

[0041] When using tree-search-based planning components, the geometric behavioral requirements of the high-level planning are met as best as possible by selecting the appropriate branches when unrolling the tree. ML-based planning components were taught, through appropriate loss functions during training, to adhere to the behavioral specifications of the high-level planning as closely as possible. It should be explicitly noted at this point that several of the aforementioned planning components can also be combined at the low-level Motion Planning level.

[0042] In general, it can be stated that the geometric behavioral specifications according to the invention are very well suited for evaluating trajectories. Trajectories generated by a planning component can thus be easily evaluated with regard to their distance from the identified go and no-go zones. If the strategic behavioral planning also provides semantic information about individual zones in the traffic scene, then a predefined set of rules can also be used to evaluate the trajectories, which prioritizes the trajectories, for example, with regard to safety and / or speed.

[0043] At the In Fig. System 200, as depicted in section 2, is a further training of the [system / system / etc.]. Fig. The system shown in Figure 1 comprises 100 components. Therefore, identical components are designated with the same reference symbols. For an explanation of these components, please refer to the description in Figure 1. Fig.1 referred to. In addition to the neural network 110 and the downstream planning component 120, the system 200 includes a further neural network 210, which extracts planning-relevant information 211 from the aggregated scene-specific information and makes it available to the downstream planning component 120.

[0044] In this further development of the invention, the high-level strategic planning level comprises an additional neural network 210 as a further high-level planning component alongside the foundation model 110. Thus, the foundation model 110 could output only the spatial component of the hit regions / hit points, and the additional high-level planning component 210 could determine the temporal component of the hit regions / hit points based on this output. However, a configuration in which the additional high-level planning component 210 is implemented as a classic planning component is also conceivable. For example, it could also provide location information for a geometric behavioral requirement, while the foundation model 110 contributes corresponding time / velocity information.

[0045] Planning component 210 could also provide additional relevant planning output that is, for example, more accurate than the output of the Foundation Model. This could be achieved, for instance, through a task-specific architecture and appropriate training of the neural network 210.

[0046] Possible implementations of such a neural network 210 and possible planning output could be, for example: • A safely navigable space that is fully compliant with regulations, output in grid or polygon format. This output format is easily represented in terms of cost. Implementation: CNN, which receives an environment model in encoded form. • A speed appropriate to the scene. Implementation: MLP, which receives an environment model in encoded form. This output format is also easy to represent in terms of cost.

[0047] In conclusion, it can be stated that the measures according to the invention contribute to better, situation-appropriate maneuver decisions within the framework of behavior planning for an at least partially automated vehicle, leading to safer, more consistent, and human-like driving behavior. This is achieved by utilizing the high contextual understanding of a uni- or multimodal neural network, in particular a foundation model, for high-level maneuver planning, while the concrete feasibility and physical implementation of this strategic behavior planning are ensured by underlying machine learning-based or classical planning / control elements. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 102024203268

[0012]

Claims

[1] Computer-implemented method for planning the behavior of a vehicle (1) in a traffic scene, wherein the behavior planning pursues a predetermined driving goal, - by generating at least one scene representation (11) of the given traffic scene based on aggregated scene-specific information (10), - by carrying out strategic behavior planning based on the scene representation (10, 11) using at least one neural network (110), and - by carrying out detailed behavioral planning based on the strategic behavioral planning with the help of at least one downstream planning component (120), characterized by , that within the framework of strategic behavioral planning at least a geometric behavioral specification for the vehicle in the given traffic scene is generated by - at least one Go area (3; 5) is identified which the vehicle (1) may or should pass through in order to pursue the specified destination, and / or - at least one no-go area (4) is identified which the vehicle (1) should avoid when pursuing the specified driving destination, and that as a result of the detailed behavior planning at least one trajectory (12) for the vehicle (1) is generated, taking into account at least one geometric behavior requirement of the strategic behavior planning. [2] Method according to claim 1, characterized by , that a unimodal or multimodal deep learning (DL) foundation model is used as the neural network (110) for strategic behavior planning, wherein the foundation model is very large and has been pre-trained with extremely large datasets, in particular self-supervised. [3] Method according to one of claims 1 or 2, characterized by, that at least one geometric behavior specification is provided in the form of a sequence of hit points (5), wherein each hit point (5) is determined by location coordinates and a time specification and / or at least one state parameter for the vehicle, in particular speed, acceleration and / or orientation. [4] Method according to one of claims 1 or 2, characterized by , that at least one geometric behavior specification is provided in the form of a sequence of hit regions, wherein each hit region is determined by a location specification in the form of a polygon and a time interval and / or an interval of at least one state parameter for the vehicle, in particular speed, acceleration and / or orientation. [5] Method according to one of claims 1 or 2, characterized by, that at least one geometric behavior specification is provided in the form of zones (3, 4) which are located in the traffic scene and to which semantic information on the possible behavior of the vehicle (1) in the respective zone (3, 4) is assigned, wherein the possible behavior of the vehicle (1) is described using at least one state parameter, in particular speed, acceleration and / or orientation. [6] Method according to any one of claims 1 to 5, characterized by , that strategic behavioral planning takes into account a prediction of the future development of the given traffic situation. [7] Method according to any one of claims 1 to 6, characterized by , that detailed behavioral planning takes into account scene representation and / or a prediction of the future development of the given traffic scene. [8] Method according to any one of claims 1 to 7, characterized by, that the at least one trajectory (12) is generated as a result of the detailed behavioral planning in a rule-based, optimization-based, sampling-based, tree-search-based or machine learning (ML)-based manner, and that the at least one geometric behavioral requirement is taken into account as a selection criterion or as an optimization criterion when generating the at least one trajectory (12). [9] Computer-implemented system (100) for planning the behavior of a vehicle (1) in a traffic scene, wherein the behavior planning pursues a predetermined driving goal, comprising at least: - a perception level for aggregating scene-specific information (10) and for generating at least one scene representation (11) of the traffic scene, - a neural network (110) that performs strategic behavior planning based on the scene representation (10, 11) generated by the perception level, and - a downstream planning component (120) that performs detailed behavioral planning based on the strategic behavioral planning, characterized by , that the at least one neural network (110) is trained such that, as a result of the strategic behavior planning, it generates at least one geometric behavior specification for the vehicle (1) in the given traffic scene, by - at least one Go area (3; 5) is identified which the vehicle (1) may or should pass through in order to pursue the specified destination, and / or - at least one no-go area (4) is identified which the vehicle (1) should avoid when pursuing the specified driving destination, and that the at least one downstream planning component (120) is configured to generate at least one trajectory (12) for the vehicle (1) as a result of the detailed behavior planning, taking into account the at least one geometric behavior specification of the strategic behavior planning. [10] System according to claim 9, characterized by , that at least one neural network (110) in the form of a DL basic model is provided for strategic behavioral planning. [11] System according to one of claims 9 or 10, characterized by , that the downstream planning component (120) is rule-based, optimization-based, sampling-based, tree-search-based or machine learning (ML)-based and generates at least one trajectory as a result of the detailed behavioral planning. [12] System (200) according to any one of claims 9 to 11, characterized by , that in addition to a DL basic model (110) and the downstream planning component (120) at least one further planning component, in particular a further neural network (220), is provided which extracts planning-relevant information (211) from the aggregated scene-specific information (10, 11) and makes it available to the downstream planning component (120).

Images