Computer-implemented method for generating prompts for a Vision Language Action Model for controlling a vehicle and vehicle

Abstract

The invention relates to a computer-implemented method (100) for generating prompts for a Vision Language Action model for controlling a vehicle, comprising at least the following steps: determining (102) safety guidelines from a provided list of safety guidelines; prioritizing (104) the determined safety guidelines according to safety relevance and probability of occurrence; generating (106) prompting rules based on the prioritized safety guidelines; and generating (108) a prompt for the Vision Language Action model for controlling the vehicle. Taking into account further optimization goals, the method (100) provides a high level of safety for controlling a vehicle, particularly autonomously.

Description

[0001] The invention relates to a computer-implemented method for generating prompts for a vision language action model for controlling a vehicle. The invention further relates to a vehicle.

[0002] Vision Language Action Models (VLAMs) enable the integration of world knowledge for generalized behavior in road traffic, but can cause unsafe behavior through targeted prompt injection, hallucination, or out-of-context situations. The realized behavior depends heavily on the system prompt used. System prompts for VLAMs must reconcile various design goals: safety, comfort, efficiency, etc. This is a multidimensional optimization problem to which the prompts must be tuned.

[0003] The object of the invention is to provide a computer-implemented method for generating prompts for a Vision Language Action Model for controlling a vehicle and a vehicle that, taking into account further optimization goals, provides a high level of safety when controlling a vehicle, especially autonomously.

[0004] The problem is solved by the features of the independent claims. Advantageous further developments are the subject of the dependent claims and the following description.

[0005] According to a first aspect, a computer-implemented method for generating prompts for a Vision Language Action Model for controlling a vehicle is described, comprising at least the following steps: determining safety guidelines from a provided list of safety guidelines; prioritizing the determined safety guidelines according to safety relevance and probability of occurrence; generating prompting rules based on the prioritized safety guidelines; and generating a prompt for the Vision Language Action Model for controlling the vehicle.

[0006] The procedure can include several methods: - A list of safety principles (e.g., do not leave the roadway, avoid collisions, protect VRUs,...) - Prioritizing these Safety Principles using Severity & Occurrence (analogous to Functional Safety) - Systematic prompting rules according to safety prioritization, which apply the following prompt prioritization techniques: Importance first; Repetition; Negative prompting; - A system analogous to the ASIL levels specifies the order in which the Safety Principles appear in the system prompt, how often they are repeated, and from when negative prompting is used.

[0007] According to some embodiments, it is conceivable that the prompting rules can be prioritized according to at least one of the following prompt prioritization techniques: Importance first, Repetition and / or Negative Prompting.

[0008] According to some embodiments, it is conceivable that a system analogous to an Automotive Safety Integrity Level could specify for the safety guidelines the order in which the safety guidelines can appear in the system prompt, how often they can be repeated and / or from when negative prompting can be used for them.

[0009] According to some examples, it is conceivable that the safety guidelines could at least be designed to keep vehicles in the roadway, to avoid collisions and / or to protect vulnerable road users.

[0010] According to a second aspect, a computer program product is described, comprising instructions that, when the program is executed by a computer, cause it to perform the steps of the procedure according to the preceding description.

[0011] The advantages, effects, and further developments of the computer program product result from the advantages, effects, and further developments of the method described above. Therefore, reference is made to the preceding description in this regard. A computer program product can be understood, for example, as a data carrier on which a computer program element is stored, containing instructions executable by a computer. Alternatively or additionally, a computer program product can also be understood, for example, as a permanent or volatile data storage medium, such as flash memory or main memory, that contains the computer program element. However, this does not exclude other types of data storage media that contain the computer program element.

[0012] According to a third aspect, a vehicle is described, comprising at least a computer for controlling the vehicle, which is at least capable of performing the steps according to the procedure described above.

[0013] The advantages, effects, and further developments of the vehicle result from the advantages, effects, and further developments of the device described above. To avoid repetition, reference is therefore made to the preceding description in this regard.

[0014] The invention is described below with reference to an exemplary embodiment and the accompanying drawing. It shows: Fig. 1. A flowchart of the computer-implemented procedure.

[0015] The computer-implemented procedure is carried out according to Fig. 1 hereinafter referred to in its entirety by reference numeral 100.

[0016] The computer-implemented method 100 is designed to generate prompts for a Vision Language Action model for controlling a vehicle.

[0017] As a first step, safety guidelines are determined from a provided list of safety guidelines. This list may have been pre-created and stored in the vehicle's electronic memory or on an external server. The safety guidelines may, at a minimum, aim to keep the vehicle within its lane, avoid collisions, and / or protect vulnerable road users.

[0018] In a further step (104), the identified security guidelines can be prioritized according to their security relevance and probability of occurrence. The priority assessment can be carried out according to Table 1: Table 1 Safety Principal Severity Occurance ASIL Jaywalker max low D Ped-Crossing max high D Cut-IN medium high C Road EdgeCrossing high medium C Parking Vehicle low high B Low Speedcollision lowest low A high speed collision high low B SpeedBumpVelocity lowest low QM

[0019] An Automotive Safety Integrity Level can also be provided.

[0020] In a further step (106), prompting rules are generated based on the prioritized security policies. The prompting rules can be prioritized according to at least one of the following prompt prioritization techniques:

[0021] Importance first, repetition, and / or negative prompting. A system analogous to an Automotive Safety Integrity Level (ASIL) for safety guidelines can specify the order in which the safety guidelines appear in the system prompt, how frequently they are repeated, and / or when negative prompting can be used for them. Table 2 shows a possible mapping of prompt methods to an ASIL rating. Table 2 ASIL Prioritätsgruppe AnzahlRepetitions NegativePrompting QM lowest single none A low double none B medium double single C highest triple double D highest quadruple triple

[0022] In a further step 108, at least one prompt for the Vision Language Action model can be generated to control the vehicle.

[0023] The example described above does not in any way limit the invention. Rather, the invention can be modified in numerous ways. All features of the invention described above can be essential to the invention, either alone or in combination.

Claims

[1] Computer-implemented method (100) for generating prompts for a Vision Language Action model for controlling a vehicle, comprising at least the following steps: a. Determine (102) security policies from a provided list of security policies; b. Prioritize (104) the identified security policies according to their security relevance and probability of occurrence; c. Generating (106) prompting rules based on the prioritized security policies; and d. Generating (108) a prompt for the Vision Language Action model to control the vehicle. [2] Computer-implemented method (100) according to claim 1, characterized by that the prompting rules are prioritized according to at least one of the following prompt prioritization techniques: Importance first, Repetition and / or Negative Prompting. [3] Computer-implemented method (100) according to claim 1 or 2, characterized by, that a system analogous to an Automotive Safety Integrity Level specifies for the safety guidelines the order in which the safety guidelines appear in the system prompt, how often they are repeated and / or from when negative prompting is used for them. [4] Computer-implemented method (100) according to any one of the preceding claims, characterized by , that the safety guidelines are at least designed to keep drivers in the lane, avoid collisions and / or protect vulnerable road users. [5] Computer program product comprising instructions which, when the program is executed by a computer, cause it to perform the steps of the method (100) according to any one of claims 1 to 4. [6] Vehicle comprising at least one computer for controlling the vehicle, which is configured at least to perform the steps according to the method (100) according to any one of claims 1 to 4.

Images