Computer-implemented method and system for behavior planning of a vehicle

Abstract

A concept for behavior planning of a vehicle in a traffic scene is proposed that extends the planning capabilities of a rule-based planner so that the rule-based planner can also be used for behavior planning in more complex scenarios. A rule-based planner (50) is used which, on the basis of a scene representation (2) of the traffic scene (1), carries out a behavioral planning (6) which simulates at least one planning scenario from a planner-specific set of predetermined planning scenarios, According to the invention, a text-based neural network (30) is used to select the at least one planning scenario. For this purpose, at least one text query (3) describing the traffic scene is generated for the neural network (30) based on the scene representation (2), with which a behavioral recommendation (4) is requested that is supported by the planner-specific set of predefined planning scenarios. Based on the text query (3), the neural network (30) generates a text-based behavioral recommendation (4) that is assigned to at least one of the predefined planning scenarios. The behavioral planning (6) of the rule-based planner (50) is then based on the planning scenario thus selected.

Description

Prior ArtThe invention relates to a computer implemented method and system for planning the behavior of a vehicle in a traffic scene. In this case, a rule-based scheduler is used, the behavior planning of which simulates at least one planning scenario from a scheduler-specific set of predefined planning scenarios. The behavior planning of the rule-based scheduler is based on a scene representation of the traffic scene generated based on aggregated scene-specific information.The task of autonomous driving is to control an ego vehicle on the basis of aggregated scene-specific information, in particular on the basis of sensor data, such as radar, lidar, RGB camera, for example, in such a way that a target is reached as quickly, comfortably and reliably as possible. In this case, inter alia, traffic rules are to be observed and collisions with infrastructure elements and / or further participants of the traffic scene are to be avoided. This driving task can be subdivided into the sub-tasks perception (perception), prediction (prediction), planning (planning) and control / regulation (control).The task of perception is to extract relevant information from the aggregated scene-specific information, such as location and state information regarding static and dynamic objects of the traffic scene, in particular regarding other road users. Furthermore, roadway markings can be identified at the level of perception and traffic signs or the like can be recognized and matched to map data. In this way, a surroundings model is generated as a scene representation of the current traffic scene.With the aid of prediction, the future development of the traffic scene is estimated, in particular the behavior of further participants in the traffic scene and of dynamic objects.The planning uses the environment model and the prediction of the future development of the traffic scene in order to plan the future behavior of the ego vehicle. Frequently, for this purpose, one or more possible trajectories for the ego vehicle are generated. Each trajectory comprises position data, optionally together with state data of the ego vehicle, for a predefined number of successive points in time. The state data generally describe the state of movement of the ego vehicle, such as speed, acceleration and / or orientation, at the respective point in time. The result of the planning is then implemented by a corresponding control of the actuator system of the ego vehicle. Planning in the form of trajectory data proves to be advantageous since trajectory data can usually be adjusted directly.A number of rule-based schedulers are known from practice, such as the intelligent driver model (IDM) or the predictive driver model (PDM). Both designers provide quantitative planning in the form of trajectories along a predefined lane line, a predefined minimum distance from the vehicle driving ahead being observed and a target speed being set when the vehicle is travelling freely. The planning of rule-based planners for these simple planning scenarios is very robust and is usually optimized with regard to comprehensible optimization criteria, so that it can also be interpreted.However, the known rule-based schedulers are not able to generalize beyond such simple scheduling scenarios, as they are limited to a scheduler-specific set of predefined scheduling scenarios. They do not provide reliable, safe planning results for scenarios that deviate from the predetermined planning scenarios and / or require continued planning competences. Examples of this are scenarios in which no lane center line has been detected or the lane center line has to be exited because an obstacle has been detected, or scenarios which require a targeted interaction with other road users. In addition, rule-based schedulers are unable to compensate for perception-level errors, such as object detection errors or errors in the map used.It is also known to use a Large Language Model (LLM) for the behavior planning of autonomous vehicles. Thus, the publication by Jiageng Mao et al "GPT-Driver: Learning to Drive with GPT" https: / / arxiv.org / pdf / 2310.01415.pdf describes a possibility of generating text requests, so-called prompt, as input for an LLM, which describe a traffic scene and request a behavior plan, so that the LLM delivers trajectory data for the ego vehicle as response.LLMs, such as General Predicted Transformer (GPT) from openAl or Lama from Meta, consist of several billion parameters and are trained on records on the order of the Internet. Accordingly, they have outstanding abilities to "understand" outlined situations and to correctly grasp the cause and effect, and consequently also have very good generalization ability.Nevertheless, the scheduling results of LLMs remained beyond expectation. One reason for this is that generating trajectories for an ego vehicle is a quantitative problem and LLMs often have deficiencies in solving quantitative problems. Furthermore, it cannot be excluded that LLMs produce scheduling results that do not meet truth, which may result in significant safety risks. It is also problematic that LLMs are relatively slow, which is why their behavior planning is generally not real-time capable.Disclosure of the InventionAccording to the invention, a planning concept is proposed which extends the planning competence of a rule-based scheduler, so that the rule-based scheduler can also be used for behavior planning in more complex scenarios and also provides reliable planning results for these scenarios.This is achieved according to the invention in that a text-based neural network, in particular an LLM, is used for the selection of the at least one planning scenario that the rule-based scheduler simulates. For this purpose, at least one text request describing the traffic scene is generated for the neural network on the basis of the scene representation. This prompt requests a behavioral recommendation that is supported by the scheduler-specific set of predefined planning scenarios, which is therefore limited by the scheduler-specific set of predefined planning scenarios. Based on the prompt, the neural network then generates a corresponding text-based behavioral recommendation. This is assigned to at least one of the predefined planning scenarios, which is also generally possible on account of the restriction of the prompt request. In this way, at least one of the predefined planning scenarios is selected in order to use it as a basis for the behavior planning of the rule-based scheduler.According to the invention, it has been recognized that, given a suitable combination of a classic rule-based scheduler with an LLM, the special generalization capabilities of an LLM can be used in order to effectively use the rule-based scheduler even in more complex scenarios and to generate reliable, secure scheduling results. Thus, by combining both model types, complex scenarios can also be detected, "understood" and solved in a targeted manner if the LLM proposes a behavior for the ego vehicle taking into account the possible developments of the traffic scene and in particular the interaction of the road users, and this qualitative proposal is then reliably and comfortably implemented quantitatively by the rule-based scheduler.In this procedure, the safety of the behavior planned according to the invention can even be guaranteed, provided that the underlying perception can be assumed to be error-free, since the corresponding trajectory is generated by a rule-based planner. In addition, the planning results generated according to the invention can be understood and interpreted because the behavioral suggestions of the LLM are present in speech form and the rule-based planner enables an interpretability at trajectory level, in particular if the planning of the rule-based planner is optimized with respect to a comprehensible optimization criterion, such as driving comfort and / or travel progress along the route.An essential aspect of the method according to the invention is that the behavioral recommendations generated by the LLM are limited by the scheduler-specific set of predefined planning scenarios. In a preferred embodiment of the invention, this set comprises at least one of the following planning scenarios:driving along a target line in a driving lane,setting a predetermined minimum distance to a vehicle driving ahead,adjusting a target speed,lane change,avoiding an obstacle.At this point, it should be expressly pointed out that the above list of planning scenarios is not to be understood as exhaustive at all. The rule-based scheduler or schedulers used within the scope of the invention can also be designed for further planning scenarios.If the LLM emits a behavioral recommendation for a more complex maneuver in a given traffic scene, i.e., proposes a passing maneuver, for example, then this could also be assigned to a plurality of planning scenarios, which are then applied successively or overlappingly in order to implement the behavioral recommendation. In the case of an overtaking maneuver, this could be, for example, the planning scenarios "braking", "lane change" or "following lane with lane departure" and / or "avoiding an obstacle". Alternatively, the LLM could also suggest these planning scenarios directly. In this way, the LLM makes the qualitative decision to initiate a passing maneuver at a particular time. Quantitative detail planning is then taken over by the rule-based scheduler which supplies corresponding trajectory data.The planning concept according to the invention also provides safe planning results in so-called long-tail scenarios, which require a good understanding of the traffic scene, and is clearly superior to the planning of a rule-based planner alone. As an example of a "long-tail" scenario, a traffic scene is mentioned here, in which a ball rolls onto the roadway. The rule-based planner would only recognize a small obstacle in the ball, which can be rolled over without risk. Accordingly, the rule-based scheduler would not plan any special measures for continuing the journey of the ego vehicle. In contrast, the LLM recognizes a special dangerous situation in the ball rolling on the roadway, since a child could run on the roadway. Since this situation requires special attention, the LLM will provide a behavioral recommendation in the sense of "braking" and / or "swerving", which the rule-based scheduler then implements according to the planning concept according to the invention.According to the invention, the text request for the neural network is generated on the basis of a scene representation of the traffic scene. This is preferably rule-based. In particular, it is proposed to use a uniform, permanently predefined structure and a predefined format for the text requests, which structure comprises at least a part of the following semantic blocks:characterizing the task of the neural network as maneuver planning for an ego vehicle in a traffic scene with further participants,specification of the rule-based scheduler or the scheduler-specific set of predefined scheduling scenarios,describing the traffic scene by the scene representation,describing the predicted future development of the traffic scene,describing the ego vehicle, in particular the past behavior, optionally the trajectory covered and the current state,information on the intended navigation destination of the ego vehicle,specification of the response of the neural network as a behavioral recommendation for the ego vehicle, which is supported by the scheduler-specific set of predefined planning scenarios.The text-based behavioral recommendation of the neural network can also be analyzed on a rule-based basis and assigned to at least one of the predefined planning scenarios of the rule-based scheduler.In a preferred embodiment of the invention, at least one setpoint line for the movement of the vehicle is generated on the basis of the recommendation for the behavior of the neural network. Although this describes a future behavior or maneuver of the vehicle, it does not describe its specific implementation. The setpoint line serves as input for the rule-based scheduler, which selects a suitable planning scenario and thus generates a behavior planning in the form of trajectory data, which describes the spatial and temporal movement of the vehicle along the setpoint line. The rule-based scheduler thus provides a concrete implementation for the qualitative recommendation for the behavior of the neural network.DRAWINGExemplary embodiments and advantageous refinements of the invention are explained in more detail below in conjunction with the single figure.The single FIGURE shows a block diagram of a computer-implemented system 100 according to the invention for planning the behavior of a vehicle in a traffic scene.DESCRIPTION OF EMBODIMENTSThe block diagram of the single FIGURE illustrates the interaction of the individual components of a computer-implemented system 100 according to the invention for planning the behavior of a vehicle in a traffic scene.The starting point of the behavior planning is always the state of a traffic scene 1 at a planning time and in particular the state of all static and dynamic objects and participants of the traffic scene at the planning time. The state of the traffic scene is described by scene-specific information, which is aggregated from different information sources at the planning time or also over a certain period of time before and until the planning time. The information sources can be vehicle-onboard sensors, such as lidar sensors installed on the ego vehicle, radar sensors and / or RGB cameras, or else non-vehicle sensors, such as lidar sensors, radar sensors and / or RGB cameras, which are installed in or on infrastructure elements or other road users. Further suitable information sources are stored map information, optionally together with traffic rules, as well as queryable weather and road condition information, traffic situation information, etc. The information of the different information sources is aggregated and processed by a perception level 10 in order to generate a surroundings model as scene representation 2.The scene representation 2 serves as input for a rule-based scheduler 50, the scope of functions of which is limited to a limited number of predefined scheduling scenarios. Accordingly, the scheduler 50 may only perform behavior planning corresponding to any of these predetermined planning scenarios based on the scene representation 2. The scheduler 50 could be provided with, for example, the following scheduling scenarios:driving along a target line in a driving lane,setting a predetermined minimum distance to a vehicle driving ahead,adjusting a target speed,lane change,avoiding an obstacle.According to the invention, the scene representation 2 is also used as input for a prompt generator 20, with which text requests 3, so-called prompts, are generated for an LLM 30. The LLM 30 is intended to provide a behavioral recommendation 4 for the vehicle in the respective traffic scene in response to such a prompt 3. According to the invention, however, this behavioral recommendation 4 should be limited to behavioral recommendations that can be mapped by the planning scenarios supported by the rule-based scheduler 50.The prompt 3 is generated rule-based in the exemplary embodiment described here and has a uniform, permanently predefined structure and a predefined format. They comprise a plurality of semantic blocks, so that the information of the individual semantic blocks can be classified on the basis of their position in the prompt. Separate semantic blocks can thus be provided, for example, for the following information:characterizing the task of the LLM as maneuver planning for an ego vehicle in a traffic scene with further participants,specification of the rule-based scheduler or the scheduler-specific set of predefined scheduling scenarios,describing the traffic scene by the scene representation,optionally. Description of the predicted future development of the traffic scene,describing the ego vehicle, in particular the past behavior, optionally the trajectory covered and the current state,information on the intended navigation destination of the ego vehicle,specification of the response of the LLM as a behavioral recommendation for the ego vehicle, which is supported by the scheduler-specific set of predefined planning scenarios.The LLM 30 provides a behavioral recommendation 4 in response to the prompt 3. With the aid of an evaluation module 40, the response of the LLM 30 is evaluated and assigned to at least one of the predefined planning scenarios of the rule-based scheduler 50, with which the behavioral recommendation 4 of the LLM 30 can be implemented. The scheduler 50 then uses this planning scenario to generate a trajectory for the ego vehicle that is optimal with regard to comfort and safety, which trajectory follows the behavioral recommendation 4 of the LLM 30.The LLM 30 is thus used here merely for qualitative behavior planning in the form of a behavior recommendation. This enables the selection of at least one planning scenario supported by the rule-based scheduler 50. Based on the planning scenario thus selected, the rule-based scheduler 50 then performs quantitative behavior planning 6 for the vehicle in the traffic scene analyzed by the LLM 30.It should be pointed out at this point that the individual components of the system 100 can be formed as components of the vehicle, but individual functions can also be realized outside the vehicle, such as e.g. the prompt generator 20, the LLM 30, the evaluation module 40 and / or the rule-based scheduler 50.Finally, it can be stated that the system 100 according to the invention has a clearly better generalization capability than a simple rule-based scheduler. Thus, as a rule, these can only follow the current lane, while the system according to the invention can also initiate a lane change and can thus also follow different lanes. In addition, the LLM behavior planner can also "understand" difficult scenarios, such as blocking the current lane by vehicles that are broken down. In this case, the behavior planning according to the invention could also plan a departure from the driving lane and a reversal of the accident site on the oncoming driving lane along a corresponding setpoint line proposed by the LLM with rule-based generated trajectories which guide the vehicle slowly around the accident site without risking a collision with the oncoming traffic.The rule-based scheduler prevents the scheduling of an insecure behavior even in case of erroneous responses of the LLM. For example, if the behavior level LLM selects a lane that is blocked, then the rule-based scheduler planning provides a safe and comfortable behavior along that erroneously selected lane to stop. The rule-based scheduler thus guarantees for the security of the behavior planning according to the invention.The concept of behavior planning according to the invention is also suitable in particular for behavior planning in interactive scenarios, such as a lane change in dense traffic. In this case, the ego vehicle must clarify the traffic in the adjacent lane its own intention, so that it can react to it, for example by braking, in order to generate a gap sufficient for the lane change. This type of interaction can only poorly depict rule-based schedulers alone. An LLM is in principle capable of understanding such complex interactions. Consequently, in the proposed system it could specifically select a behavior that gives the adjacent traffic this intention, for example by deviating from its own lane without leaving it. The rule-based scheduler would follow this behavior comfortably and safely without "understanding" the interaction. Once the adjacent traffic creates a gap, the LLM would select a lane change as appropriate behavior that the rule-based scheduler then converted.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Cited Non-Patent LiteratureJiageng Mao et al "GPT-Driver: Learning to Drive with GPT" https: / / arxiv.org / pdf / 2310.01415.pdf

[0008]

Claims

Computer-implemented method for planning the behavior of a vehicle in a traffic scene (1), - in which at least one scene representation (2) of the traffic scene (1) is generated on the basis of aggregated scene-specific information, and - in which a rule-based scheduler (50) is used, which carries out a behavior planning (6) on the basis of the scene representation (2), which simulates at least one planning scenario from a scheduler-specific set of predefined planning scenarios, characterized in that a text-based neural network (30) is used for the selection of the at least one planning scenario, - in that at least one text request (Prompt)(3) describing the traffic scene is generated for the neural network (30) on the basis of the scene representation (2), with which text request a behavior recommendation (4) is requested, the text-based behavior recommendation (4) generated by the neural network (30) on the basis of the text request (3) is assigned to at least one of the predefined planning scenarios by the scheduler-specific set of predefined planning scenarios, in order to form the basis for this planning scenario for the behavior planning (6) of the rule-based scheduler (50).Method according to Claim 1, characterized in that a Large Language Model (LLM)(30) is used for the selection of the at least one planning scenario.Method according to either of Claims 1 and 2, characterized in that the set of predefined planning scenarios comprises at least one of the following planning scenarios: a. driving along a setpoint line in a lane, b. setting a predefined minimum distance from a vehicle driving ahead, c. setting a target speed, d. lane change, e. avoiding an obstacle.Method according to one of Claims 1 to 3, characterized in that the text request (3) for the neural network is generated on a rule-based basis, wherein a uniform, permanently predefined structure with predefined semantic blocks in a predefined format is used for the text request (3).Method according to one of Claims 1 to 4, characterized in that the text-based behavioral recommendation (4) of the neural network is analyzed on a rule-based basis and is assigned to at least one of the predefined planning scenarios of the rule-based scheduler (50).Method according to one of Claims 1 to 5, characterized in that, on the basis of the recommendation (4) for the behavior of the neural network (30), at least one setpoint line for the movement of the vehicle is generated as input for the rule-based scheduler (50), and in that the rule-based scheduler (50) supplies at least one behavior plan (6) in the form of trajectory data which describe the movement of the vehicle in space and time along the at least one setpoint line.Computer-implemented system (100) for planning behavior (6) of a vehicle in a traffic scene (1), at least comprising: - a perception level (10) for aggregating scene-specific information and for generating at least one scene representation (2) of the traffic scene (1), - at least one rule-based scheduler (50) which, on the basis of the scene representation (2) generated by the perception level (10), carries out a behavior planning (6) which corresponds to at least one planning scenario from a scheduler-specific set of predefined planning scenarios, - a text-based neural network (30) for selecting at least one of the predefined planning scenarios as the basis for the behavior planning (6) of the rule-based scheduler (50), a prompt generator (20) for generating text requests (3) for the neural network (30) on the basis of the scene representation (2) generated by the perception level (10), wherein a respective behavioral recommendation (4) is requested with the text requests (3) within the scope of the scheduler-specific set of predefined planning scenarios, an evaluation module (40) for assigning the behavioral recommendations (4) generated by the neural network (30) to at least one of the predefined planning scenarios.System according to Claim 7, characterized in that the neural network (30) is realized in the form of a large language model (LLM).

Images