Computer-implemented method for controlling a vehicle and vehicle

Abstract

The invention relates to a computer-implemented method (100) for controlling a vehicle, which has at least one sensor for acquiring at least one current measurement data time series of at least one environmental area of ​​the vehicle, comprising at least the following steps: Determining (102) whether at least one pedestrian is detected in at least one current measurement data time series using at least one perception agent; If at least one pedestrian is detected (104): Acquiring (106) at least one signal of the at least one detected pedestrian in the at least one measurement data time series using the at least one perception agent; Interpreting (108) the at least one signal using an LLM-based intention interpretation agent; Generating (110) and providing (112) at least one control command for the vehicle based on the at least one interpreted signal using at least one maneuver planning agent;and generating (114) and providing (116) at least one output signal for an external vehicle-human interface by means of at least one communication agent, wherein the at least one output signal displays information for the at least one detected pedestrian that is appropriate to the at least one control command. The method (100) can increase the traffic safety of vehicle maneuvers.

Description

[0001] The invention relates to a computer-implemented method for controlling a vehicle and a vehicle.

[0002] With the transition to highly automated and fully automated driving systems (Level 4 and above), interaction patterns between vehicles and other road users are changing. While conventional vehicles with human drivers rely heavily on eye contact, implicit signals, and human intuition, autonomous vehicles lack the ability to reliably interpret these subtle forms of nonverbal communication. Existing systems use cameras and rule-based models to detect pedestrians and predict their movement. However, they can only grasp gestures, intentions, and ambiguous situations, such as "is the pedestrian really waiting?" or "are they demanding right-of-way?", in a rudimentary way. At the same time, vehicles lack the ability to clearly and situationally communicate their own driving intentions to pedestrians.

[0003] The current state of the art has the following limitations: Limited gesture interpretation: Classical models only recognize simple body postures or directions of movement, but not complex intentions, nonverbal signals, or social contexts. No semantic analysis: Pedestrian actions are not contextualized (e.g., "raising a hand to stop," "hesitating," "signaling a gesture of thanks"). Lack of bidirectional communication: Vehicles can only convey clear intentions to pedestrians to a limited extent; existing HMI systems are static. Uncertain interactions in densely populated urban areas: Level 4 vehicles, in particular, require a robust, universally applicable communication logic.

[0004] The object of the invention is to provide a computer-implemented method for controlling a vehicle and a vehicle with which the traffic safety of vehicle maneuvers can be increased.

[0005] The problem is solved by the features of the independent claims. Advantageous further developments are the subject of the dependent claims and the following description.

[0006] According to a first aspect, a computer-implemented method for controlling a vehicle, which has at least one sensor for acquiring at least one current measurement data time series of at least one environmental area of ​​the vehicle, is described, comprising at least the following steps: Determining whether at least one pedestrian is detected in at least one current measurement data time series, using at least one perception agent; If at least one pedestrian is detected: Acquiring at least one signal of the at least one detected pedestrian in the at least one measurement data time series using the at least one perception agent; Interpreting the at least one signal using a Large Language Model (LLM)-based intention interpretation agent; Generating and providing at least one control command for the vehicle based on the at least one interpreted signal using at least one maneuver planning agent;and generating and providing at least one output signal for an external vehicle-human interface by means of at least one communication agent, wherein the at least one output signal displays information suitable for the at least one detected pedestrian in relation to the at least one control command.

[0007] The invention comprises a multi-agent system that combines LLM-based semantic understanding with multimodal sensor processing to dynamically interpret pedestrian gestures and behavior and derive adaptive vehicle responses. Furthermore, the vehicle communicates its intentions back to the pedestrians via an external, highly dynamic display.

[0008] The system consists of several cooperating agents: Perception agent: can, for example, capture image, radar, and lidar data and extract pedestrian gestures and micro-movements. LLM-based intent interpretation agent: interprets detected signals, particularly semantically: examples include "hand signal to stop," "hesitant crossing," "demanding right-of-way," and / or "distracted behavior (smartphone use)." Maneuver planning agent: derives driving maneuvers, particularly adaptive ones, from the understanding, especially semantic understanding, e.g., proactive braking, cooperative yielding, waiting states, and / or unambiguous reverse signaling. HMI communication agent: can, for example, control an external display as an external vehicle-human interface, e.g., front / side displays and / or LED matrix, which can clearly display vehicle intentions, textually or symbolically. Examples: “Please go”, “I’m stopping”, “I’m going first” and / or arrow symbols indicating direction of movement.

[0009] The agents can continuously exchange contextual information and mutually optimize their decisions.

[0010] Furthermore, bidirectional, and in particular semantic, communication can take place: For example, the system can not only recognize pedestrian gestures, especially semantically, but also send back vehicle intention signals, especially semantic ones. This allows for a closed communication loop.

[0011] Furthermore, an LLM-based interpretation of real-world body signals can be performed. Signals from the detected pedestrian, such as gestures, micro-movements, gaze directions, and / or nonverbal patterns, can be interpreted like elements of speech.

[0012] Furthermore, maneuver planning, especially adaptive planning, can be carried out: decisions can then be based not only on deep learning predictions, but also on semantic "intention-to-action" processing.

[0013] Specifically for Level 4 vehicles, this approach can compensate for the lack of human interaction in urban environments and increase safety, trust, and transparency.

[0014] The computer-implemented method can include a computer-implemented sensor fusion module: For example, processing of camera, lidar, radar, and / or thermal data can be performed for precise pedestrian detection and, preferably, gesture extraction. LLM reasoning layer: LLM models (on-device or hybrid) can interpret movement patterns as semantic tokens. Examples: The movement pattern "arm raised" can be interpreted as the semantic token "stop signal"; the movement pattern "lateral hesitation" can be interpreted as the semantic token "uncertain intent"; the movement pattern "shoulder turn toward vehicle" can be interpreted as the semantic token "expectation of a signal." A multi-agent orchestrator can coordinate perception, interpretation, planning, and communication agents.A decision engine for driving maneuvers can translate semantic meaning into concrete control commands, particularly as speed adjustment, a stop timer, and / or cooperative yielding. An external communication channel, serving as an external vehicle-human interface, can, for example, feature an LED- or e-ink-based display, optionally with arrows, signals, or textual cues. This can potentially be supplemented by short acoustic signals. A safety layer can include fail-safe logic to prioritize safe maneuvers in cases of uncertainty in gesture interpretation.

[0015] This provides a computer-implemented method for controlling a vehicle that interprets gestures, behavior, and situational context in a deeply semantically accurate way, while simultaneously communicating dynamically with pedestrians. This enhances the traffic safety of vehicle maneuvers. Furthermore, it establishes a novel communication standard between autonomous vehicles and pedestrians, improving safety, transparency, and acceptance, particularly in complex urban environments.

[0016] An agent is understood to be a program that can act independently and autonomously to a certain extent.

[0017] According to some embodiments, it is conceivable that a signal from at least one detected pedestrian may include at least one gesture and / or at least one micro-movement of the at least one detected pedestrian.

[0018] According to some embodiments, it is conceivable that in the step: Interpreting the at least one signal, a semantic interpretation of the at least one signal can be carried out.

[0019] According to some embodiments, it is conceivable that the at least one current measurement data time series can include camera data, lidar data, radar data and / or thermal data.

[0020] According to some embodiments, it is conceivable that at least one multi-agent orchestrator can coordinate the at least one perception agent, the at least one LLM-based intention interpretation agent, the at least one maneuver planning agent and / or the at least one communication agent.

[0021] According to a second aspect, a computer program product is described, comprising instructions that, when the program is executed by a computer, cause it to perform the steps of the procedure according to the preceding description.

[0022] The advantages, effects, and further developments of the computer program product result from the advantages, effects, and further developments of the method described above. Therefore, reference is made to the preceding description in this regard. A computer program product can be understood, for example, as a data carrier on which a computer program element is stored, containing instructions executable by a computer. Alternatively or additionally, a computer program product can also be understood, for example, as a permanent or volatile data storage medium, such as flash memory or main memory, that contains the computer program element. However, this does not exclude other types of data storage media that contain the computer program element.

[0023] According to a third aspect, a vehicle is described comprising at least one sensor for capturing at least one current measurement data time series of at least one environmental area of ​​the vehicle, at least one external vehicle-human interface and at least one computer that is designed to perform the steps according to the procedure according to the preceding description.

[0024] The advantages, effects, and further developments of the vehicle result from the advantages, effects, and further developments of the device described above. To avoid repetition, reference is therefore made to the preceding description in this regard.

[0025] The invention is described below with reference to an exemplary embodiment and the accompanying drawing. It shows: Fig. 1. A flowchart of the process.

[0026] The computer-implemented method for controlling a vehicle is according to Fig. 1 hereinafter referred to in its entirety by reference numeral 100.

[0027] The computer-implemented method 100 can be executed in conjunction with a vehicle. The vehicle can have at least one sensor for acquiring at least one current measurement data time series of at least one area surrounding the vehicle, at least one external vehicle-human interface, and at least one computer. The computer can be configured to execute the steps described below according to method 100.

[0028] In a first step, it is determined whether at least one pedestrian is detected in at least one current measurement data time series. For this purpose, the measurement data time series from the at least one sensor can first be received. The at least one current measurement data time series can include camera data, lidar data, radar data, and / or thermal data.

[0029] The investigation according to step 102 can be carried out using at least one perception agent.

[0030] If at least one pedestrian is detected according to step 104, step 106 can be executed. In step 106, at least one signal from the at least one detected pedestrian is captured in the at least one measurement data time series. The signal from the at least one detected pedestrian can include at least one gesture and / or at least one micro-movement. The capture of the at least one signal can be performed using at least one perception agent.

[0031] In a further step, at least one signal is interpreted. The interpretation can be performed at the semantic level and carried out using an LLM-based intention interpretation agent.

[0032] According to a further step 110, at least one control command for the vehicle can be generated based on the at least one interpreted signal. The generated signal is also made available accordingly, according to step 112. Steps 110 and 112 are performed using at least one maneuver planning agent.

[0033] In a further step 114, at least one output signal is generated for an external vehicle-human interface. This output signal can also be provided according to step 116. The output signal displays information relevant to the control command for the detected pedestrian. This means, for example, that instructions can be issued to the pedestrian that correspond to the driving maneuver, such as the instruction to stop because the vehicle will continue driving, or the pedestrian can be informed which driving maneuver the vehicle will perform so that they can react accordingly.

[0034] Steps 114 and 116 are performed using at least one communication agent.

[0035] At least one multi-agent orchestrator can coordinate at least one perception agent, at least one LLM-based intention interpretation agent, at least one maneuver planning agent, and / or at least one communication agent. At least two of these agents can be coordinated by the at least one multi-agent orchestrator.

[0036] The example described above does not in any way limit the invention. Rather, the invention can be modified in numerous ways. All features of the invention described above can be essential to the invention, either alone or in combination.

Claims

[1] Computer-implemented method (100) for controlling a vehicle having at least one sensor for recording at least one current measurement data time series of at least one environmental area of ​​the vehicle, comprising at least the following steps: a. Determine (102) whether at least one pedestrian is detected in at least one current measurement data time series, using at least one perception agent; b. If at least one pedestrian is detected (104): Capture (106) at least one signal of the at least one detected pedestrian in the at least one measurement data time series using the at least one perception agent; c. Interpreting (108) the at least one signal using an LLM-based intention interpretation agent; d. Generating (110) and providing (112) at least one control command for the vehicle based on the at least one interpreted signal by means of at least one maneuver planning agent; and e. Generating (114) and providing (116) at least one output signal for an external vehicle-human interface by means of at least one communication agent, wherein the at least one output signal displays information suitable for the at least one detected pedestrian in relation to the at least one control command. [2] Computer-implemented method (100) according to claim 1, characterized by , that a signal from at least one detected pedestrian shows at least one gesture and / or at least one micro-movement of the at least one detected pedestrian. [3] Computer-implemented method (100) according to claim 1 or 2, characterized by, that in step: Interpreting (108) the at least one signal, a semantic interpretation of the at least one signal is performed. [4] Computer-implemented method (100) according to any one of the preceding claims, characterized by that it includes at least one current measurement data time series, camera data, lidar data, radar data and / or thermal data. [5] Computer-implemented method (100) according to any one of the preceding claims, characterized by , that at least one multi-agent orchestrator coordinates the at least one perception agent, the at least one LLM-based intention interpretation agent, the at least one maneuver planning agent and / or the at least one communication agent. [6] Computer program product comprising instructions which, when the program is executed by a computer, cause it to perform the steps of the method (100) according to any one of claims 1 to 5. [7] Vehicle comprising at least one sensor for capturing at least one current measurement data time series of at least one environmental area of ​​the vehicle, at least one external vehicle-human interface and at least one computer configured to perform the steps according to the method (100) according to any one of claims 1 to 5.

Images