Computer-implemented method and system for behavioral planning of vehicle
By combining a rule-based planner with LLM and leveraging the generalization capability of LLM to provide behavioral recommendations for the rule-based planner, the planning uncertainty problem in complex scenarios in existing technologies is resolved, achieving safe and comfortable driving in complex traffic scenarios.
Patent Information
- Application Number
- CN202510430896.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-04-10
- Filing Date
- 2025-04-08
- Publication Date
- 2025-10-17
AI Technical Summary
Existing rule-based planners cannot provide reliable and safe planning results in complex traffic scenarios or those that deviate from the predetermined planning scenarios. Although large language models (LLMs) have generalization capabilities, they are insufficient in quantitative issues and may generate unreasonable planning results, leading to safety risks.
Combining a rule-based planner with LLM, the paper generates text queries to constrain the LLM's behavioral suggestions to predetermined planning scenarios. The generalization capability of LLM is used to provide behavioral suggestions for the rule-based planner. The qualitative suggestions generated by LLM are quantitatively implemented by the rule-based planner to ensure the reliability and security of the planning results.
Generating reliable, safe, and explainable planning results in complex scenarios improves the generalization ability of rule-based planners, enabling them to handle complex traffic scenarios such as a ball rolling onto a lane and lane changes in interactive scenarios, ensuring safe and comfortable driving of the ego vehicle.
Smart Images
Figure CN120792830A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present invention relates to a computer-implemented method and system for behavior planning of a vehicle in a traffic scenario. Here, a rule-based planner is used, whose behavior planning imitates at least one planning scenario from a pre-given set of planning scenarios specific to the planner. The behavior planning of the rule-based planner is based on a scenario representation of the traffic scenario, which is generated based on aggregated scenario-specific information. BACKGROUND
[0002] The task of autonomous driving is to control the ego vehicle based on aggregated scenario-specific information, in particular based on sensor data such as radar, lidar, RGB camera, so that the destination is reached as quickly, comfortably and safely as possible. Here, in particular, traffic rules should be observed and collisions with infrastructure elements and / or other participants of the traffic scenario should be avoided. This driving task can be subdivided into the subtasks of perception, prediction, planning and control / regulation.
[0003] The task of perception is to extract relevant information from the aggregated scenario-specific information, such as position and state information about static and dynamic objects of the traffic scenario, in particular about other traffic participants. In addition, lane markings can be identified on the perception level and traffic signs, etc. can be recognized and compared with map data. In this way, an environment model is generated as a scenario representation of the current traffic scenario.
[0004] By means of prediction, the future development of the traffic scenario is estimated, in particular the behavior of other participants and dynamic objects of the traffic scenario.
[0005] Planning utilizes the environment model and the prediction of the future development of the traffic scenario in order to plan the future behavior of the ego vehicle. Usually, for this purpose one or more trajectories are generated for the ego vehicle, which can be considered. Each trajectory comprises position data, if necessary together with state data of the ego vehicle at a pre-given number of successive time points. The state data usually describes the motion state of the ego vehicle at the respective time point, for example speed, acceleration and / or orientation. The result of the planning is then implemented by correspondingly manipulating the actuator system of the ego vehicle. Planning in the form of trajectory data proves to be advantageous, since trajectory data can mostly be directly adjusted.
[0006] A series of rule-based planners are known from practice, such as the Intelligent Driver Model (IDM) or the Predictive Driver Model (PDM). Both planners provide quantitative planning in the form of a trajectory along a pre-given lane line, in which a pre-given minimum distance to a vehicle driving ahead is observed, and the target speed is adjusted when driving freely. Rule-based planners are very robust with regard to planning for these simple planning scenarios, and are usually optimized with regard to implementable optimization criteria, so that the planning is also interpretable.
[0007] However, the known rule-based planners cannot generalize beyond such simple planning scenarios, since they are limited to a planner-specific set of pre-given planning scenarios. For scenarios that deviate from the pre-given planning scenarios and / or require a more extensive planning capability, the planners cannot provide reliable, safe planning results. Examples of this are scenarios in which the lane centerline is not recognized or must be left due to a detected obstacle, or scenarios in which a targeted interaction with other traffic participants is required. Furthermore, rule-based planners cannot compensate for errors on the perception level, such as errors in object recognition or errors in the map used.
[0008] It is furthermore known to use large language models (LLMs) for behavior planning of autonomous vehicles. Thereby, the publication "GPT-Driver: Learning to Drive with GPT" by Jiageng Mao et al. https: / / arxiv.org / pdf / 2310.01415.pdf describes the possibility to generate a textual query, i.e. a so-called prompt, as input for an LLM, which describes a traffic scenario and requests a behavior planning, so that the LLM provides trajectory data for the ego vehicle as a response.
[0009] LLMs such as the general pre-trained transformer (GPT) of openAI or Llama of Meta consist of tens of billions of parameters and are trained on data sets in the order of the internet. In correspondence therewith, the LLMs have a prominent ability to "understand" the depicted situation and to correctly grasp causality, and thus also a very good generalization capability.
[0010] However, the planning results of the LLMs are not as good as expected. One reason for this is that generating a trajectory for the ego vehicle is a quantitative problem, while LLMs usually have a deficit in solving quantitative problems. Furthermore, it cannot be ruled out that the LLM produces planning results that do not correspond to reality, which can lead to enormous safety risks. Furthermore, it is problematic that LLMs are relatively slow, so that their behavior planning is usually not real-time capable. SUMMARY
[0011] According to the invention, a planning approach is proposed which extends the planning capabilities of rule-based planners such that rule-based planners can also be used for behavior planning in more complex scenarios and also provide reliable planning results for these scenarios.
[0012] According to the invention, this is achieved by using a text-based neural network, in particular an LLM, for selecting at least one planning scenario which the rule-based planner imitates.
[0013] To this end, at least one text query describing a traffic scenario is generated for the neural network based on the scenario representation. With this prompt, a behavior suggestion supported by the planner-specific set of pre-given planning scenarios is requested, which is thus limited by the set of pre-given planning scenarios. Based on the prompt, the neural network then generates a corresponding text-based behavior suggestion. The behavior suggestion is assigned to at least one of the pre-given planning scenarios, which is generally also possible due to the limitation of the prompt query. In this way, at least one of the pre-given planning scenarios is selected so that the behavior planning of the rule-based planner is based on the at least one planning scenario.
[0014] According to the invention, it has been recognized that, when combining a classical rule-based planner with an LLM in an appropriate manner, the special generalization capabilities of the LLM can be utilized in order to effectively use the rule-based planner even in more complex scenarios and to generate reliable, safe planning results. Thereby, if the LLM suggests a behavior for the ego vehicle in view of possible developments of the traffic scenario and in particular the interactions of the traffic participants, and this qualitative suggestion is then quantitatively implemented safely and comfortably by the rule-based planner, complex scenarios can also be "understood" and addressed in a targeted manner by combining the two model types.
[0015] In the case of such actions, the safety of the behavior planned according to the invention can even be guaranteed as long as it can be assumed that the underlying perception is error-free, since the corresponding trajectory is generated by the rule-based planner. Furthermore, the planning results generated according to the invention are executable and interpretable, since the behavior suggestion of the LLM exists in linguistic form and the rule-based planner is able to implement interpretability on the trajectory level, in particular when the planning of the rule-based planner is optimized in view of executable optimization criteria such as driving comfort and / or driving progress along the route.
[0016] An important aspect of the method according to the invention is that the behavior suggestion generated by the LLM is limited by a set of pre-given planning scenarios specific to the planner. In a preferred embodiment of the invention, this set comprises at least one of the following planning scenarios:
[0017] - driving along a nominal line on the carriageway,
[0018] - adjusting a predefined minimum distance to a vehicle driving in front,
[0019] - adjusting a target speed,
[0020] - changing lane,
[0021] - avoiding an obstacle.
[0022] At this point, it should be noted in detail that the above list of planning scenarios is by no means final. One or more rule-based planners used within the scope of the present application can also be designed for other planning scenarios.
[0023] If the LLM issues a behavior recommendation for a more complex maneuver in a given traffic scenario, i.e. for example recommends a passing maneuver, this can also be assigned to a plurality of planning scenarios, which are then used in succession or even overlappingly to implement the behavior recommendation. In the case of a passing maneuver, this can be for example the planning scenarios "braking", "changing lane", "following a carriageway with lane clearance" and / or "avoiding an obstacle". Alternatively, however, the LLM can also directly recommend these planning scenarios. In this way, the LLM makes a qualitative decision for introducing a passing maneuver at a specific point in time. The rule-based planners then take over the quantitative detail planning, which provide corresponding trajectory data.
[0024] The planning approach according to the present application provides safe planning results even in so-called "long tail" scenarios, which require a good understanding of the traffic scenario, and clearly outperforms the planning of a rule-based planner used alone. As an example of a "long tail" scenario, here the traffic scenario of a ball rolling onto the carriageway should be mentioned. A rule-based planner would recognize in the case of the ball only a small obstacle that can be rolled over without danger. Correspondingly, the rule-based planner would not plan a special measure for the continued driving of the ego vehicle. In contrast, the LLM recognizes in the case of a ball rolling onto the carriageway a special dangerous situation, since a child can run onto the carriageway. Since this situation requires special attention, the LLM would provide a behavior recommendation in the sense of "braking" and / or "avoiding", which is then implemented by the rule-based planner according to the planning approach according to the present application.
[0025] According to the present application, the text query to the neural network is generated on the basis of a scenario representation of the traffic scenario. This is preferably carried out in a rule-based manner. In particular, it is proposed to use for the text query a uniform, fixedly predefined structure and a predefined format, which comprises at least one of the following semantic blocks:
[0026] - characterizing the task of the neural network as maneuver planning for a ego vehicle in a traffic scenario with other participants,
[0027] - specifying a rule-based planner or a planner-specific set of pre-given planning scenarios,
[0028] - describing the traffic scenario by a scenario representation,
[0029] - describing a predicted future development of the traffic scenario,
[0030] - describing the ego vehicle, in particular past behavior, if necessary past trajectory and current state,
[0031] - a specification about a sought navigation destination of the ego vehicle
[0032] - specifying the response of the neural network as a behavior recommendation for the ego vehicle, the behavior recommendation being supported by the planner-specific set of pre-given planning scenarios.
[0033] The text-based behavior recommendation of the neural network can also be analyzed in a rule-based manner and assigned to at least one of the pre-given planning scenarios of the rule-based planner.
[0034] In a preferred embodiment of the invention, the neural network-based behavior recommendation generates a target line for the vehicle motion. This describes the future behavior or maneuver of the vehicle, but not its concrete implementation. The target line serves as an input for a rule-based planner, which selects an appropriate planning scenario and thereby generates a behavior plan in the form of trajectory data, which describes the spatial and temporal motion of the vehicle along the target line. The rule-based planner thereby provides a concrete implementation for the qualitative behavior recommendation of the neural network. BRIEF DESCRIPTION OF DRAWINGS
[0035] The application will be explained in more detail below with reference to the only Figure 1 Embodiments of the application and advantageous improvements are explained in more detail below.
[0036] The application will be explained in more detail below with reference to the only Figure 1 A block diagram of a computer-implemented system 100 for behavior planning of a vehicle in a traffic scenario according to the application is shown. DETAILED DESCRIPTION
[0037] The application will be explained in more detail below with reference to the only Figure 1 The block diagram of a computer-implemented system 100 for behavior planning of a vehicle in a traffic scenario according to the application illustrates the interaction of the individual components of the computer-implemented system 100.
[0038] The starting point of the behavior planning is always the state of the traffic scenario 1 at the planning point in time and, in particular, the state of all static and dynamic objects and participants of the traffic scenario at the planning point in time. The state of the traffic scenario is described by scenario-specific information aggregated from different information sources at or over a certain period of time before and up to the planning point in time. The information sources can be vehicle-own sensors, such as laser radar sensors, radar sensors and / or RGB cameras installed at the ego vehicle, or also vehicle-external sensors, such as laser radar sensors, radar sensors and / or RGB cameras installed in or at infrastructure elements or other traffic participants. As information sources, in addition, stored map information are considered, if necessary together with traffic rules and queryable weather and road condition information, traffic situation information, etc. The information of the different information sources is aggregated and processed by the perception layer 10 in order to generate an environment model as a scenario representation 2.
[0039] The scenario representation 2 is used as an input for a rule-based planner 50, the functional scope of which is limited to a finite number of pre-given planning scenarios. In correspondence therewith, the planner 50 is only able to perform a behavior planning on the basis of the scenario representation 2, which corresponds to one of these pre-given planning scenarios. For example, the following planning scenarios can be provided to the planner 50:
[0040] - driving along a nominal line on the lane,
[0041] - adjusting a pre-given minimum distance to a vehicle driving ahead,
[0042] - adjusting a target speed,
[0043] - changing a lane,
[0044] - avoiding an obstacle.
[0045] However, according to the invention, the scenario representation 2 is also used as an input for a prompt generator 20, with which a text query 3, a so-called prompt, is generated for the LLM 30. In response to such a prompt 3, the LLM 30 should provide a behavior recommendation 4 for the vehicle in the respective traffic scenario. However, according to the invention, this behavior recommendation 4 should be limited to behavior recommendations that can be mapped by the planning scenarios supported by the rule-based planner 50.
[0046] In the embodiments described here, the prompt 3 is generated in a rule-based manner and has a uniform, fixed pre-given structure and a pre-given format. The prompt comprises a plurality of semantic blocks, so that the information of the individual semantic blocks can be categorized according to their position in the prompt. For example, it is thereby possible to set up separate semantic blocks for the following information, respectively:
[0047] - the task of the LLM is characterized by a maneuver planning for the ego vehicle in a traffic scenario with other participants,
[0048] - a rule-based planner or a planner-specific set of pre-given planning scenarios is specified,
[0049] - the traffic scenario is described by a scenario representation,
[0050] - if necessary, a predicted future development of the traffic scenario is described,
[0051] - the ego vehicle, in particular past behavior, if necessary past trajectory and current state, is described,
[0052] - a specification of a sought navigation destination of the ego vehicle,
[0053] - the response of the LLM is specified as a behavior recommendation for the ego vehicle, which is supported by a planner-specific set of pre-given planning scenarios.
[0054] The LLM 30 provides a behavior recommendation 4 as a response to the prompt 3. The behavior recommendation can be described, for example, by a selection of a lane, a desired lane clearance, or by a nominal line and a maximum speed. By means of the evaluation module 40, the response of the LLM 30 is evaluated and assigned to at least one of the pre-given planning scenarios of the rule-based planner 50 with which the behavior recommendation 4 of the LLM 30 can be implemented. The planner 50 then uses this planning scenario in order to generate a trajectory for the ego vehicle that is optimal in terms of comfort and safety, which trajectory follows the behavior recommendation 4 of the LLM 30.
[0055] The LLM 30 is thus used here only for a qualitative behavior planning in the form of a behavior recommendation. This makes it possible to select at least one planning scenario that is supported by the rule-based planner 50. On the basis of the planning scenario thus selected, the rule-based planner 50 then performs a quantitative behavior planning 6 for the vehicle in the traffic scenario analyzed by the LLM 30.
[0056] At this point, it should be noted that the individual components of the system 100 can be constructed as integral parts of the vehicle, but the individual functions can also be implemented outside the vehicle, such as the prompt generator 20, the LLM 30, the evaluation module 40 and / or the rule-based planner 50. If a corresponding data connection is available and suitable connectivity is guaranteed, these components can also be implemented, for example, in the cloud or edge cloud.
[0057] Finally, it can be determined that the system 100 according to the present application has significantly better generalization capabilities than a simple rule-based planner. Typically, a simple rule-based planner can only follow the current lane, whereas the system according to the present application can also initiate a lane change and thus can also follow a different lane. Furthermore, the LLM behavior planner can also "understand" difficult scenarios, such as a current lane being blocked by a vehicle that has broken down. In this case, the behavior planning according to the present application can also utilize the rule-based generated trajectory planning to exit the lane and to bypass the accident site on the opposite lane along the corresponding nominal line suggested by the LLM, which leads the vehicle to slowly bypass the accident site without the risk of a collision with oncoming vehicles.
[0058] Even in the case of a faulty response of the LLM, the rule-based planner prevents the planning of unsafe behavior. If the LLM on the behavior level, for example, selects a blocked lane, the planning of the rule-based planner sets a stop as a safe and comfortable behavior along this faulty selected lane. Thus, the rule-based planner guarantees the safety of the behavior planning according to the present application.
[0059] The behavior planning scheme according to the present application is particularly also suitable for behavior planning in interactive scenarios, such as changing lanes in congested traffic. Here, the ego vehicle has to signal its intention to the traffic in the adjacent lane so that this traffic can react, for example, by braking, in order to create a gap that is sufficient for the lane change. A rule-based planner alone can only poorly map this type of interaction. In principle, the LLM is able to understand such complex interactions. Therefore, the LLM can purposefully select a behavior in the proposed system that communicates this intention to the adjacent traffic, for example, by deviating from the ego lane without leaving this lane. The rule-based planner will comfortably and safely follow this behavior without "understanding" this interaction. Once the adjacent traffic creates a gap, the LLM selects the lane change as the appropriate behavior, which the rule-based planner can then implement.
Claims
1. A computer-implemented method for planning the behavior of a vehicle in a traffic scenario (1), - wherein at least one scene representation (2) of the traffic scene (1) is generated based on the aggregated scene-specific information, and wherein a rule-based planner (50) is used, said rule-based planner (50) performing a behavior planning (6) based on a scenario representation (2), said behavior planning emulating at least one planning scenario from a planner-specific set of predefined planning scenarios, It is characterized by: using a text-based neural network (30) for selecting at least one planning scenario, - in such a way that, based on the scenario representation (2), at least one textual query (prompt) (3) describing the traffic scenario is generated for the neural network (30), with which action recommendations (4) supported by a planner-specific set of predefined planning scenarios are requested, and - This is done by assigning a text-based behavior suggestion (4) generated by a neural network (30) based on a text query (3) to at least one of the predefined planning scenarios in order to base a behavior plan (6) of a rule-based planner (50) on this planning scenario.
2. The method according to claim 1, characterized in that A large language model (LLM) (30) is used to select at least one planning scenario.
3. The method according to claim 1 or 2, characterized in that The predetermined planning scenario set includes at least one of the following planning scenarios: a. Drive along the rated line on the lane, b. Setting a predetermined minimum distance to the vehicle ahead, c. Adjust the target speed, d. Changing lanes, e. Avoid obstacles.
4. The method according to any one of claims 1 to 3, characterized in that The text query (3) for the neural network is generated in a rule-based manner, wherein a uniform, fixedly predefined structure of predefined semantic blocks in a predefined format is used for the text query (3).
5. The method according to any one of claims 1 to 4, characterized in that The text-based action recommendations (4) of the neural network are analyzed in a rule-based manner and assigned to at least one of the predefined planning scenarios of a rule-based planner (50).
6. The method according to any one of claims 1 to 5, characterized in that Based on the behavior suggestions (4) of the neural network (30), at least one target line for the movement of the vehicle is generated as input to a rule-based planner (50), and the rule-based planner (50) provides at least one behavior plan (6) in the form of trajectory data describing the spatial and temporal movement of the vehicle along the at least one target line.
7. A computer-implemented system (100) for planning the behavior (6) of a vehicle in a traffic scenario (1), comprising at least: a perception layer (10) for aggregating scene-specific information and generating at least one scene representation (2) of the traffic scene (1), - at least one rule-based planner (50) that performs behavior planning (6) based on a scenario representation (2) generated by a perception layer (10), the behavior planning corresponding to at least one planning scenario from a set of planner-specific pre-given planning scenarios, a text-based neural network (30) for selecting at least one planning scenario from among the predefined planning scenarios as a basis for a behavior plan (6) of a rule-based planner (50), a prompt generator (20) for generating a text query (3) for the neural network (30) based on the scenario representation (2) generated by the perception layer (10), wherein the text query (3) is used to request a behavior suggestion (4) within the scope of a set of predefined planning scenarios that are specific to the planner, An evaluation module (40) for assigning a behavior recommendation (4) generated by the neural network (30) to at least one of the predefined planning scenarios.
8. The system according to claim 7, characterized in that The neural network (30) is implemented in the form of a large language model (LLM).