Method and system for the automated execution of a vehicle parking maneuver

Abstract

The invention relates to a method for the automated execution of a parking maneuver of a vehicle, comprising the following steps - Receiving a voice input (210) with a natural language parking instruction from a vehicle user; - Converting speech input (210) into a text representation (230) using a speech-to-text component (230); - Semantic analysis of the text representation (250) by a parsing component (270) and generation of a structured input representation (290); - Capturing the vehicle environment by means of a sensor unit (340) comprising at least one camera and a LiDAR sensor, and recording sensor data (320); - Generating a semantically annotated, three-dimensional scene model (370) by fusion of the sensor data (320); - Processing the structured input representation (290) and the scene model (370) by a multimodal language model (MLLM) (400), wherein the MLLM (400) generates a vehicle trajectory (450) corresponding to the language input (210); - Converting the generated vehicle trajectory (450) into vehicle dynamics control parameters (550); - Execution of the parking process by a control module (700) via vehicle-side control units (770).

Description

[0001] The present invention relates to a method and a system for the automated execution of a parking operation of a vehicle, in particular for applications in the field of modern driver assistance systems (Advanced Driver Assistance Systems, ADAS) as well as for future highly automated or autonomous driving functions (HAF) or so-called Automated Driving Systems (ADS).

[0002] In recent years, numerous driver assistance systems have become established in the field of automotive engineering, designed to make parking easier for the driver. Classic parking assistance systems such as acoustic parking aids, optical reversing cameras, or semi-automated parking functions are now standard equipment in many vehicles. More advanced systems already allow automatic steering while the driver simultaneously controls speed and braking (so-called semi-automated parking systems).

[0003] Despite these advances, significant limitations remain regarding the flexibility, customizability, and intuitive usability of these systems. Common solutions are typically based on predefined maneuvers or fixed sets of rules. Parking space selection is achieved either through predefined sensor logic or simple user interaction via buttons or menu options. A semantic description of the desired parking destination (e.g., "Park in front of the red vehicle located in a shaded area") or natural language interaction is generally not provided for in these systems or is only technically feasible to a limited extent.

[0004] Furthermore, existing systems are typically modular in design: they consist of several separate components for environmental perception, decision-making, and vehicle control. These components usually interact via clearly defined interfaces, which, while increasing traceability, also increases system complexity and limits responsiveness in dynamic scenarios. Moreover, implementing such systems often requires significant development effort with software specifically designed for the respective vehicle architecture.

[0005] Modern automated driving systems, particularly in the area of ​​parking assistance, increasingly rely on artificial intelligence (AI) methods. These systems often employ artificial neural networks, which are trained through machine learning from large datasets. Such networks enable the recognition of complex patterns in sensor data and the use of this information to make driving decisions – for example, in object detection, trajectory planning, or the control of vehicle maneuvers.

[0006] Despite the capabilities of modern AI methods, conventional neural networks have significant limitations regarding their generalizability. Since they are typically trained on specialized training data for specific parking maneuvers, they lack the ability to flexibly respond to new or differing situations that were not explicitly part of the training process. This leads to such systems quickly reaching their functional limits in real-world application scenarios, particularly in urban environments, thus compromising the reliable operation of the parking function for the user.

[0007] A key shortcoming is the insufficient ability to process semantic and spatial context. While simple scenes with clear object distances and structured parking areas can still be processed reliably, conventional systems falter when confronted with semantically complex environments. This refers to scenes in which a multitude of different objects with specific meanings and functions occur (e.g., parked vehicles, bicycles, delivery zones, driveways, construction site barriers, no-parking signs, parking spaces with charging infrastructure, reserved areas for emergency vehicles, or special use zones with structural or legal access restrictions), and these objects must be interpreted differently in the respective context (e.g.,a delivery vehicle with its hazard warning lights flashing as a temporary obstacle), and in which verbal instructions are semantically linked to concrete objects in the scene (e.g., "Park in front of the white SUV, but not in the driveway").

[0008] Furthermore, in the case of imprecise, impractical, or legally impermissible parking instructions, a query or interaction with the user is required. For example, the system might receive the input "Park in front of the taxi." This refers to a parking space in an area designated as a taxi stand. In this case, a semantic interpretation of the instruction and a context-sensitive rejection with a simultaneous suggestion for improvement are necessary, for example, in the form of an alternative parking space behind a suitable vehicle or at a permissible distance from the target area.

[0009] In such scenarios, a purely geometric or classificatory understanding is insufficient. Rather, a combination of semantic information (What is it?) with spatial location (Where is it?) and functional context (What does it mean right now?) is necessary to correctly interpret naturally expressed parking instructions and translate them into appropriate vehicle actions.

[0010] Conventional neural networks are not designed to capture and process the meaning of objects in relation to their specific spatial position, as they typically rely on clearly structured data and predictable decision-making processes. Dynamic, voice-controlled interaction, in which the vehicle interprets a natural language instruction in a real-world environment and independently develops a safe and sensible parking solution, is either impossible or severely limited with these systems.

[0011] German patent DE 10 2022 213 191 A1 relates to a method for assisting a user when parking or maneuvering a vehicle. First, sensor data about the vehicle's surroundings is acquired by an in-vehicle sensor system. Based on this data, a virtual view of the surroundings is displayed. Simultaneously, the user can interact with the system via voice command: An acoustic voice control command is recognized and evaluated, whereupon either the virtual view is adjusted or the vehicle controls are modified accordingly. The latter includes, in particular, the control of steering and drive motors to provide targeted support during parking or driving maneuvers.

[0012] German patent DE 10 2017 219 065 A1 describes a method for configuring at least highly automated vehicle control. First, user input is captured at a human-machine interface. This input is a command for configuring the automated vehicle control. Additionally, environmental information characterizing the vehicle's current local environment is captured. Based on an evaluation of the user input in relation to this environmental information, a target for automated vehicle control is derived. Subsequently, a driving strategy specific to the local environment is defined according to this target. Finally, control signals are generated and output to drive the vehicle automatically according to the defined driving strategy.

[0013] CN 120 003 527 A relates to a method for automated driving based on an end-to-end and a multimodal large-scale model. For this purpose, image information, vehicle history information, navigation information, and text information are first acquired from several sensors of a target vehicle. These different types of information are each converted into coded tokens using assigned encoders. The coded tokens are fed into a trained multimodal large-language model, which then generates decoded tokens. These are subsequently processed in parallel by a trajectory decoder and a text decoder. This process outputs, on the one hand, a planned driving trajectory including control information for the vehicle, and on the other hand, a textual interpretation of the current driving strategy. Based on the planned trajectory and the control information, the vehicle is guided automatically.

[0014] CN 119 272 899 A concerns a data processing method in the field of automated driving, specifically for evaluating automated driving behavior using a large-language model. For this purpose, the vehicle state and environmental information of an automated vehicle are first recorded. From this information, a natural language feature is generated that represents the vehicle state and environmental information in natural language. This natural language feature is then fed into a large-language model. The model subsequently outputs a description of the current automated driving scene, generates a future prediction for the described driving scene, and, based on this, produces an evaluation of the automated vehicle's driving behavior.

[0015] DE 10 2021 113 052 A1 relates to a method for generating environmental information about the surroundings of a vehicle equipped with a driver assistance system including at least one environmental sensor and an optical camera. For this purpose, image information of the vehicle's surroundings is first acquired using the camera, and a point cloud of the surroundings is created using the environmental sensor. Subsequently, a semantic image of the surroundings is generated from the image information by feeding it into a neural network, in particular a convolutional neural network. This semantic image has a reduced resolution compared to the original image information. The environmental points of the point cloud are directly mapped into the semantic image. Finally, the environmental information is generated by assigning the semantic information from the semantic image to the respective environmental points.

[0016] DE 10 2023 208 155 A1 relates to a computer-implemented method for data processing in a vehicle system using a voice-based user interface. For this purpose, voice input is first captured by a microphone connected to or integrated into the vehicle system and converted into a text message by the vehicle system. The text message is then transmitted to a processing unit. On the processing unit, a response is generated using a machine learning algorithm. For this purpose, a task list containing at least one task is derived from the text message. Based on these tasks, information is retrieved and / or commands are generated for the vehicle system.

[0017] German patent DE 10 2013 022 596 B3 relates to a method and a system for voice activation of a software agent from standby mode. For this purpose, audio data is continuously buffered in an audio buffer, ensuring that the buffer always contains recent audio data. Simultaneously, the audio data is fed to a secondary speech recognition system, which can be designed to be particularly energy-efficient. If the secondary speech recognition system detects an activation word, a primary speech recognition process is initiated, which converts the content stored in the audio buffer into text. The conversion begins at the start of a sentence, which is identified in the audio buffer by a pause in speech. The generated text is then fed into a dialogue system.

[0018] German patent DE 10 2023 004 804 B3 relates to a method for predicting the motion of objects in the vicinity of a vehicle that is at least partially automated. For this purpose, environmental data acquired by sensors is determined and fused. The fused environmental data is fed to a scene interpretation module, which generates queries based on the recognized scene and transmits these to a Large Language Model. Based on the responses generated by the Large Language Model, further interpretations of the scene described by the fused environmental data are performed.

[0019] The object of the invention is to provide a method and system that is able to semantically interpret natural language input from a vehicle user, link it with a three-dimensional spatial perception of the vehicle environment, and independently derive a vehicle trajectory corresponding to the parking request in order to carry out the parking process automatically, without having to resort to separately trained task-specific sub-models or rule-based subsystems for trajectory planning and trajectory control.

[0020] This problem is solved according to the invention with respect to a method by the features of claim 1, with respect to a system by the features of claim 6, and with respect to a computer program product by the features of claim 10. The further claims relate to preferred embodiments of the invention.

[0021] The present invention enables the automated execution of parking maneuvers based on natural language input combined with semantic-spatial 3D scene understanding. By employing a multimodal language model (MLLM), the system according to the invention is able to recognize verbally named target objects, locate them in the real-world environment, and derive a suitable vehicle trajectory without using rule-based subsystems or task-specific trained models. The modular architecture allows for flexible integration into existing vehicle platforms and supports intuitive, dialogue-based human-machine interaction. This creates a new form of user-centered vehicle control that handles complex parking situations fully automatically with minimal user intervention.

[0022] According to a first aspect, the invention provides a method for the automated execution of a parking maneuver of a vehicle. The method comprises the following steps: - Receiving a voice input with natural language parking instructions from a vehicle user via a voice input module; - Converting speech input into a text representation with a machine-readable text format using a speech-to-text component; - Semantic analysis of the text representation by a parsing component and generation of a structured input representation, wherein the input representation contains the relevant action information, object descriptions and spatial relationships, which are passed as a prompt to a multimodal language model (MLLM); - Capturing the vehicle's surroundings using a sensor unit that includes at least one camera and LiDAR sensors, and recording sensor data; - Generating a semantically annotated, three-dimensional scene model by fusing the sensor data in a sensor data processing unit, wherein the sensor data processing unit provides the 3D scene model as a point cloud representation; - Processing the structured input representation and the scene model by a multimodal language model (MLLM), wherein the structured input representation and the semantically annotated 3D scene model are combined into a multimodal input and fed together to the MLLM, wherein the MLLM generates a vehicle trajectory corresponding to the language input, and wherein the MLLM is trained to iteratively evaluate and optimize the generated vehicle trajectory within the framework of an internal model self-reflection, wherein the evaluation is based on internal model evaluations and / or simulated feedback steps, and a modified vehicle trajectory is generated if the trajectory is negatively evaluated; - Converting the generated vehicle trajectory into vehicle dynamics control parameters using a trajectory module; - Conversion of the control parameters into control commands by a control module and execution of the parking process via vehicle-side control units, whereby the control module executes the vehicle trajectory using existing vehicle interfaces (e.g. CAN bus) without relying on separately pre-trained, rule-based subsystems for trajectory planning.

[0023] In a further development, it is planned that the sensor data processing unit will also provide the 3D scene model as an image representation.

[0024] In an advantageous embodiment, it is provided that the camera data is segmented by a segmentation model (e.g. SAM) and the lidar data is assigned to the image segments by lidar-to-mask mapping.

[0025] Advantageously, the user receives feedback on the parking process via a voice output or a graphical visualization.

[0026] In a further training course, it is planned that the user can correct or cancel the parking process by means of a new voice input, whereby the voice input module then generates a new structured input representation and the processing in the MLLM is executed again.

[0027] According to a second aspect, the invention provides a system for the automated execution of a vehicle parking maneuver. The system comprises a speech input module for receiving a parking instruction in natural language, with a speech-to-text component and a parsing component for converting and analyzing the speech input into a structured input representation. The control module executes the vehicle trajectory using existing vehicle interfaces (e.g., CAN bus) without relying on separately pre-trained, rule-based subsystems for trajectory planning.An environmental perception module with a sensor unit for acquiring environmental data, comprising at least one camera and a lidar sensor, and a sensor data processing unit for fusing the sensor data and generating a semantically annotated, three-dimensional scene model, wherein the sensor data processing unit is configured to provide the 3D scene model as a point cloud representation; a multimodal language model (MLLM) configured to process the structured input representation and the 3D scene model to generate a vehicle trajectory that corresponds to the user instruction in the voice input; a trajectory module for converting the trajectory into vehicle dynamics control parameters;and a control module for converting the control parameters into control commands and for transmitting them to the vehicle-side control units, wherein the system is configured to combine the structured input representation and the semantically annotated 3D scene model into a multimodal input and provide it to the MLLM for further processing; wherein the MLLM is configured to iteratively evaluate and optimize the generated vehicle trajectory within the framework of an internal model self-reflection, wherein the evaluation is based on internal model evaluations and / or simulated feedback steps, and a modified vehicle trajectory is generated if the trajectory is negatively evaluated; and wherein the control module is configured to execute the generated trajectory without recourse to task-specific trained submodels or rule-based planning logics.

[0028] In an advantageous embodiment, sensor fusion is achieved through lidar-to-image mapping or a bird's-eye view method.

[0029] In another embodiment, an output module is designed to create a speech output and / or a visualization.

[0030] In a further training, it is provided that the user can correct or cancel the parking process by means of a new voice input, whereby the voice input module is trained to then generate a new structured input representation, and whereby the MLLM is trained to execute the processing again.

[0031] According to a third aspect, the invention provides a computer program product with an executable program code that is configured to perform the method according to the first aspect when executed.

[0032] The invention will now be explained in more detail with reference to an embodiment shown in the drawing.

[0033] It shows: Fig. 1 a block diagram to illustrate an embodiment of a system according to the invention; Fig. 2 a flowchart to explain the individual process steps of a process according to the invention; Fig. 3 a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

[0034] Additional features, aspects and advantages of the invention or its embodiments become apparent from the detailed description in conjunction with the claims.

[0035] A typical parking maneuver occurs in inner-city traffic situations, for example, when searching for a suitable parking space on a densely parked street. The vehicle moves along a row of parked cars at a reduced speed. The driver observes the surroundings, identifies potential gaps between already parked vehicles, and visually assesses whether the available space is sufficient to park their own vehicle.

[0036] Once a suitable parking space has been identified – for example, a gap between a red van and a white SUV – the driver decides whether to park immediately. This requires taking into account the vehicle's position relative to the parking space, the steering angle, the distances to neighboring objects, and any restrictions imposed by lane markings, driveways, or visibility.

[0037] In many cases, there is only a limited time window available for parking, for example, due to following vehicles or confined spaces. Furthermore, the environment can be dynamic, for example, due to pedestrians, cyclists, or maneuvering vehicles. The driver must therefore make a decision within a very short time and execute the parking maneuver precisely, often requiring complex spatial assessments and repeated corrections.

[0038] This scenario places high demands on perception, orientation, and driving skills, especially in confined spaces, with limited visibility, or under time pressure. At the same time, it is an everyday part of driving in many urban and suburban contexts.

[0039] Fig. Figure 1 shows a system 100 according to the invention for the automated execution of a vehicle parking maneuver, particularly for applications in the field of advanced driver assistance systems (ADAS) and for future highly automated or autonomous driving functions (HAF) or so-called automated driving systems (ADS). The system 100 is based on a combination of sensors, speech processing, a multimodal speech model (MLLM), scene analysis, and control components. The system 100 consists of several modules, each responsible for individual process steps. These include, in particular, a speech input module 200, an environment perception module 300, a multimodal speech model (MLLM) 400, a trajectory module 500, a control module 700, and an optional output module 800. Each of these modules can be equipped with a storage unit and / or a processing unit that technically supports the respective functionality.

[0040] A module within the meaning of the present invention is defined as a specialized unit of software and / or hardware components that performs a precisely defined sub-function within the overall system. Each module processes specific inputs, executes corresponding computational steps, and delivers outputs that are passed on to other modules via clearly defined interfaces. In this way, coordinated and effective cooperation within the system 100 according to the invention is ensured for the automated execution of parking operations.

[0041] Two or more of these modules can be integrated, either alternatively or additionally, into a shared computing unit, such as a central control unit in the vehicle, a distributed edge system, or a connected server infrastructure. This enables scalable and resource-efficient implementation of voice-based parking control.

[0042] A processor within the meaning of the present invention can be a microcontroller, a main processor (CPU), a specially programmable processor, or a virtualized computing unit. The processor is configured to perform the processing steps required for the method according to the invention. These include, in particular, the conversion of spoken language into machine-readable text (speech-to-text), the generation of a semantic-spatial scene model based on multimodal sensor data, the inference in the MLLM to determine a suitable trajectory, and the conversion into vehicle dynamics control commands.

[0043] For computationally intensive subprocesses, such as processing high-dimensional sensor data (camera, lidar), semantic segmentation, or trajectory inference using neural models, the processor can include additional specialized computing units. These include, in particular, graphics processing units (GPUs), tensor processing units (TPUs), or other AI accelerators optimized for parallel matrix and vector operations. These enable improved execution of AI-based system components, especially in real time.

[0044] The computing units can be deployed locally in the vehicle, on a connected computing system, or distributed across a cloud environment. In particular, the MLLM 400 can be implemented locally or in a hybrid configuration (with outsourcing to a cloud infrastructure) to flexibly meet the requirements for latency, data protection, and computing capacity.

[0045] A high-performance cloud infrastructure is used, particularly for the training and continuous improvement of the MLLM 400. This provides scalable computing capacity required for training complex deep learning models, especially for processing multimodal sensor data.

[0046] A storage unit within the meaning of the invention can be volatile memory (e.g., RAM) or persistent memory (e.g., SSD, HDD, or cloud storage). Volatile memory is used for the temporary storage of sensor data, intermediate model results, or trajectory designs. Persistent memory can be used to store trained language models, scene annotations, trajectory schemes, or user profiles. In advantageous embodiments, model-based intermediate results or preprocessing results can be stored and efficiently retrieved in recurring driving situations.

[0047] System 100 includes communication interfaces that enable bidirectional, low-latency, and reliable data transmission between the individual modules and, if necessary, with external computing or data services such as a connected cloud infrastructure. Communication can be wired or wireless. These communication interfaces are primarily used for exchanging voice data, sensor data (e.g., from cameras or lidar), trajectory information, and control commands between the modules, as well as for connecting to external data or computing services.

[0048] Wired technologies include, in particular, Automotive Ethernet and CAN bus, which enable reliable and low-latency communication within the vehicle architecture. Wireless communication methods include, among others, mobile communication standards such as 4G LTE, 5G and, in the future, 6G, as well as WLAN technologies such as Wi-Fi. ®as well as short-range connections such as Bluetooth ® and Ultra-Wideband (UWB) technology is used.

[0049] To ensure the integrity and confidentiality of sensitive data, such as voice data, sensor readings, and other user input, the System 100 can be equipped with cryptographic methods. These include transport encryption using TLS and the encrypted storage of model-based decision structures. The storage of generated trajectories can also be encrypted if required, especially when stored in conjunction with user-specific information or reusable trajectory patterns.

[0050] System 100 comprises a speech input module 200, which is designed to capture spoken instructions from a user in the form of speech input 210, convert them into a machine-readable format, and prepare them for subsequent processing by the multimodal language model (MLLM) 400. The speech input module 200 thus forms the interface between the human user or operator and the system 100 according to the invention, which then enables automated trajectory planning and driving control via the control module 700.

[0051] The voice input 210 can be implemented via a microphone unit permanently integrated into the vehicle, for example as part of the infotainment system or a driver assistance module. Alternatively or additionally, the voice input module 200 can also be implemented wholly or partially via a mobile device, in particular a smartphone. In this embodiment, the voice input 210 is captured via the microphone of the smartphone, with the transmission of the captured data to the input module 200 being wireless, for example via Bluetooth. ® , Wi-Fi ® or a mobile communication connection. This configuration enables flexible interaction with the system 100 according to the invention, e.g. from the vicinity of the vehicle or as part of a preconfiguration outside the vehicle interior.

[0052] The speech input module 200 includes a speech-to-text component 230, which is designed to convert user-inputted speech 210 into a machine-readable text format. This speech-to-text component 230 can be implemented as a software application running on a suitable computing unit. Execution can take place locally on an in-vehicle control unit, on an external mobile device—especially a smartphone—or on a cloud-based computing instance.

[0053] As an exemplary embodiment of the speech-to-text component 230, a modern neural language model such as the so-called Whisper model can be used. This is a language model for automatic speech recognition, implemented as an end-to-end system. In such a system, the acoustic input data from the speech input 210 is directly converted into text without requiring separate components for feature extraction, acoustic modeling, or decoding. The conversion takes place entirely within a common neural network trained as an encoder-decoder structure. It is capable of reliably transcribing spoken language even under conditions with acoustic interference, different accents, or multilingual input. The language model directly converts the speech input data from the speech input 210 into structured text and is therefore particularly suitable for embedded applications with natural language input.

[0054] The output of the speech-to-text component 230 is in the form of a text representation 250, which, for the purposes of the present invention, is defined as structured, machine-readable text. This text representation maps the semantic content of the original speech input 210 and can, for example, be in the form of plain text, a tokenized sequence, or formatted input text. The text representation 250 serves as an input signal for the subsequent semantic analysis, in which the content is further structured and prepared for processing by the multimodal language model (MLLM) 400.

[0055] Following the conversion of the speech input 210 into a text representation 250, structural and semantic preprocessing is performed by a parsing component 270. This component is designed to analyze the speech input 210 for relevant semantic content and generate a structured input representation 290 for the multimodal language model (MLLM) 400. This includes, in particular, the identification of objects named in the speech input 210 (e.g., "red delivery van"), spatial relationships (e.g., "in front of", "behind", "between", "forward", "sideways", "backwards"), and explicit or implicit instructions (e.g., "park", "parking", "unparking").

[0056] The parsing component 270 decomposes the text representation 250 into semantically meaningful units, links these contextually, and transforms them into a structured input representation 290, for example, in the form of a customized prompt for the MLLM 400. In a preferred embodiment, the parsing component 270 can also take user-specific preferences into account, for example, regarding preferred parking directions, desired comfort distances, or areas to be excluded, such as near trees or driveways.

[0057] A prompt within the scope of the present invention is a structured input request to the multimodal language model (MLLM) 400 and contains semantically relevant information in a machine-understandable form. The prompt is based on the structured input representation 290 generated by the speech input module 200, which is obtained through semantic analysis of the natural language input 210. Such a prompt can be in the form of a text token, a sequential data object, or a vector-based representation format. It serves to transmit the content context, the goal of the parking operation, and any user specifications (e.g., comfort distance, desired parking position) to the MLLM 400.

[0058] The prompt thus represents the linguistic-semantic interface between user input and model inference and, together with the 3D scene model 370, forms the basis for the generation of a context-sensitive vehicle trajectory 450.

[0059] The parsing component 270 is specifically designed as a software-based processing instance with integrated artificial intelligence (AI) algorithms. In particular, machine learning (ML) and natural language processing (NLP) methods are used to semantically analyze the text representation 250 and generate a structured input representation 290.

[0060] The parsing component 270 can comprise one or more neural language models trained or refined to recognize semantic units, action verbs, object references, and spatial relationships. Semantic analysis can be performed using recurrent neural networks, such as LSTMs (Long Short-Term Memory networks), or transformer-based architectures (e.g., GPT). These models are capable of extracting meaningful elements from textual input, such as target objects ("red van"), spatial relationships ("in front of..."), and action intentions ("park," "park").

[0061] Neural networks are a central component of modern machine learning, and especially deep learning. They are inspired by the workings of the human brain, where information is processed through the interconnection of neurons. Neural networks are particularly good at learning nonlinear relationships in sensor data and are therefore very powerful in analyzing complex data. By training them on large datasets, neural networks can detect erroneous patterns that might be missed by other techniques.

[0062] A neural network consists of neurons arranged in multiple layers and interconnected in various ways. A neuron can receive information at its input from the outside or from another neuron, process it in a specific manner, and then pass it on in a modified form to another neuron at its output, or output it as a final result. Hidden neurons are located between the input and output neurons. Depending on the type of network, there can be several layers of hidden neurons. They are responsible for the transmission and processing of information. Output neurons ultimately deliver a result and transmit it to the outside world. The arrangement and interconnection of the neurons result in different types of neural networks, such as feedforward networks (FFNs), recurrent neural networks (RNNs), and convolutional neural networks (CNNs).Each network is optimized for specific tasks and data types. Networks can be optimized either through unsupervised learning, where patterns are recognized without previously labeled data, or through supervised learning, where the model is trained with labeled examples.

[0063] The Convolutional Neural Network (CNN) is a specific type of artificial neural network. It has multiple convolutional layers and is well-suited for machine learning and artificial intelligence (AI) applications in the field of pattern recognition. The individual layers of the CNN are the convolutional layer, the pooling layer, and the fully connected layer. The convolutional layer is the actual convolutional layer and is capable of recognizing and extracting individual features from the input data. In pattern and image recognition, these can be features such as lines, edges, or specific shapes. The input data is processed in the form of tensors, such as a matrix or vectors. The pooling layer, also called the subsampling layer, condenses and reduces the resolution of the recognized features using appropriate filtering functions. The reduced data volume increases the processing speed.Because the CNN is divided into several local, partially connected layers, it requires significantly less memory than fully connected neural networks. The training time of the Convolutional Neural Network is also considerably shorter. Thanks to the use of modern graphics processing units (GPUs), CNNs can be trained very efficiently.

[0064] Deep neural networks (DNNs) are a special type of artificial neural network (ANN) characterized by their multi-layered structure. These networks consist of many layers of neurons arranged in successive layers. Each layer of neurons performs specific calculations and passes the results on to the next layer.

[0065] The term "deep" refers to the fact that deep neural networks have multiple hidden layers. The idea behind deep networks is to capture and learn complex relationships within the data. When training a deep neural network, the weights of the connections between the neurons are adjusted so that the desired output is produced for a given input data. Deep neural networks are particularly powerful and are therefore preferably used in the context of the present invention.

[0066] In an extended embodiment, the parsing component 270 can additionally feature context-adaptive processing, in which user input is interpreted in light of previous interactions or stored user preferences. This can be achieved by incorporating fine-grained embedding representations, entity-tracking mechanisms, or semantic storage structures.

[0067] The resulting structured input representation 290 can be in various formats, for example, as a token structure in a prompt or as a graph-based representation, and is then passed to the downstream multimodal language model (MLLM) 400 for further processing. The speech input module 200, which comprises the speech-to-text component 230 and the parsing component 270, is solely responsible for the language-based preprocessing of the speech input 210. Independent trajectory planning or decision logic is not implemented in the speech input module 200.

[0068] The transfer to the MLLM 400 takes place via standardized, low-latency communication interfaces, whereby data transmission can be secured by cryptographic methods, in particular Transport Layer Security (TLS). Due to its modular design, the speech input module 200 is open to various technical implementations. In particular, speech input 210 can also be performed via an external mobile device, which enables flexible interaction with the system according to the invention even outside the vehicle interior.

[0069] The environmental perception module 300 is designed to capture and process the immediate vehicle environment based on various sensor data and to provide it in a structured, model-interpretable representation. The environmental data generated by the environmental perception module 300 serves, in particular, as input for the multimodal language model (MLLM) 400, which links the spatial and semantic information with a linguistically defined driving task, for example, within the context of a voice-controlled parking maneuver.

[0070] To capture the environment, sensor data 320 from multiple sources are fused in the environmental sensing module 300. For this purpose, the environmental sensing module 300 comprises a sensor unit 340, which includes, in particular, lidar sensors, radar systems, and camera-based sensor systems. These provide complementary data on the geometry, movement, and visual properties of the vehicle's surroundings.

[0071] The 340 sensor unit can contain one or more camera sensors, which can be configured as monocular, stereo, or fisheye cameras. Monocular cameras provide two-dimensional image data with high color resolution and are particularly useful for object detection, classification, and semantic segmentation. Stereo cameras additionally enable depth estimation based on image parallax, allowing for the reconstruction of spatial structures. Fisheye cameras offer an extended field of view, which is especially advantageous at close range. The camera sensors are preferably mounted at the front, sides, and rear of the vehicle to provide a complete all-around view.

[0072] The 340 sensor unit can also include lidar sensors that generate a three-dimensional point cloud of the vehicle's surroundings by emitting and recording light pulses over time (time-of-flight measurement). Each lidar measurement provides a spatial coordinate point with distance and angle information relative to the vehicle's position. The point cloud represents a geometrically accurate representation of the environment and forms the basis for spatial scene modeling. In conjunction with segmentation data from camera images, each lidar point can be further enriched with semantic information (so-called lidar-to-mask mapping).

[0073] The 340 sensor unit can additionally include radar sensors, which are used particularly for detecting objects at medium and long distances and for measuring relative velocities. Radar sensors operate on the basis of electromagnetic waves and use time-of-flight and Doppler measurements to determine the distance, direction of movement, and speed of objects. Compared to camera-based or lidar-based detection, radar sensors are less sensitive to adverse environmental conditions such as rain, fog, or darkness. They provide measurement data with comparatively lower spatial resolution but complement the sensor data from camera and lidar systems with reliable speed and distance information, especially for moving objects such as vehicles or pedestrians.

[0074] The sensor data 320 acquired in the environmental perception module 300 from camera, lidar, and radar sensors are fused by a sensor data processing unit 350 to create a coherent, multidimensional environmental representation. This fusion can take place at various levels: - Sensor level: Here, raw data from different sensors (e.g., depth information from the lidar point cloud with color information from the camera) are directly combined, e.g., by mapping lidar points onto image coordinates. - Feature level: Extracted features such as contours, edges, object classes or motion vectors are correlated and integrated across sensors, e.g. through AI-based feature fusion in neural networks. - Decision level: Several pre-processed object information pieces, such as radar tracking of a moving vehicle and visual recognition as an SUV, are combined into a consistent, semantically enriched object representation.

[0075] Sensor data fusion preferably takes place in a common global coordinate system defined relative to the vehicle's position. This enables precise spatial mapping of all detected objects and structures.

[0076] In image processing, a segmentation model is used, such as a model from the SAM family (Segment Anything Model). This segmentation model is designed to divide visual scenes into segmented image units, precisely masking individual objects or object parts, such as a car door, a bumper, or a bicycle. To enable semantically consistent object recognition, small-scale segment masks are aggregated into larger, logically related object units using heuristic rules.

[0077] To enhance spatial understanding, a link is established between image segmentation and lidar data. For this purpose, the image masks generated by the segmentation model are linked to the corresponding lidar point clouds using a technique called lidar-to-mask mapping. In this way, each detected image area (e.g., "red delivery van") can be enriched with specific geometric information (e.g., distance, angle, dimension).

[0078] The fused data form the basis for generating the semantically annotated 3D scene model 370, which encompasses both the semantic and geometric properties of the environment. The 3D scene model 370, generated by the sensor data processing unit 350, contains a structured representation of the vehicle's surroundings in three-dimensional spatial coordinates. It includes geometric information about the detected objects (e.g., position, orientation, dimensions), their semantic classification (e.g., vehicle, tree, pedestrian), and their relationships to one another (e.g., in front of, beside, between). The scene model can be in the form of an annotated point cloud, a voxel grid, or a projective view (e.g., a bird's-eye view with semantic layers). This 3D scene model 370 is provided to the multimodal language model (MLLM) 400 for interpreting the environment in the context of spoken instructions.

[0079] The Environmental Perception Module 300 is designed to operate in real-time or near real-time with high resolution and is flexibly scalable to different sensor architectures. In advantageous configurations, it can also communicate with external computing units or cloud infrastructures to offload computationally intensive processing steps.

[0080] The multimodal language model (MLLM) 400 is designed as a central inference module. It is configured to interpret the driving tasks defined by the voice input 210 in the context of the current environmental situation of the vehicle as detected by the sensor unit 240, and to generate a vehicle trajectory corresponding to the driving task. The MLLM 400 thus takes over the semantic-spatial interpretation of the user's voice input 210 with regard to the vehicle environment detected by the environmental sensing module 300.

[0081] A Multimodal Large Language Model (MLLM) 400 represents an advanced AI technology designed to process both linguistic and visual / sensory input. Unlike traditional large language models (LLMs), which primarily analyze speech patterns, an MLLM can interpret image, video, and sensor data and place it into semantic contexts. This enables context-sensitive analysis of the vehicle's environment and realistic modeling of objects such as vehicles, pedestrians, or traffic signs.

[0082] Language models, especially Large Language Models (LLMs), are based on neural networks optimized for processing and generating natural language. They are trained on large amounts of text to recognize statistical patterns and semantic relationships.

[0083] LLMs use deep neural networks, specifically transformer architectures such as GPT (Generative Pre-trained Transformer). These deep neural networks consist of multiple layers of artificial neurons connected by weights.

[0084] LLMs are trained on billions of words from various sources (books, articles, websites). Through this training, they learn probability distributions for words and sentences, i.e., which word is most likely to follow another.

[0085] The key to the success of modern language models is the self-attention mechanism. This enables the language model to prioritize relevant parts of a text and establish connections across large text passages. Furthermore, processing occurs in multiple layers (encoders and decoders), with each layer capturing more abstract language features.

[0086] Before processing, the text is broken down into smaller units called tokens. These can be individual letters, words, or even syllables. The model then works not with the raw text, but with these sequences of tokens.

[0087] During text generation, the language model iteratively predicts the most likely next token based on previous inputs. It doesn't simply choose the most probable word deterministically, but rather introduces a degree of variability through temperature and sampling mechanisms. Temperature is a parameter that determines how deterministic or random the language model's output is. This parameter influences the probability distribution of the next possible token. Sampling determines how the model selects the next word (token) from this probability distribution.

[0088] Language models like GPT-4 or MLLMs combine these mechanisms with other modalities (images, videos, sensor data) to go beyond pure text processing. In this context, the term modality refers to the different types of information or data sources that an MLLM can process. Classical linguistic models (LLMs) work with only a single modality—namely, text. A multimodal model (MLLM), on the other hand, can understand and process other modalities besides text, such as images, videos, audio, or sensor data.

[0089] Text is a linguistic modality, images represent a visual modality for the recognition and analysis of objects and scenes, videos are a dynamic visual modality for motion recognition and behavior analysis, audio is an acoustic modality for speech recognition and mood analysis, and sensor data is a spatial-physical modality containing data on the vehicle's driving dynamics. Multimodal language models can combine several of these modalities and integrate them into a common semantic space.

[0090] The MLLM 400 receives two inputs: firstly, the structured input representation 290, for example, in the form of a system-compatible prompt generated by the speech input module 200; and secondly, the semantically enriched 3D scene model 370 provided by the sensor data processing unit 350. Both information sources are combined to form a multimodal input 410, giving the MLLM 400 an integrated view of linguistic, semantic, and geometric aspects of its environment.

[0091] The MLLM 400 is designed to create a semantic-spatial scene model 430 based on this combined input 410. In this scene model 430, named objects from the speech input (e.g., "red delivery van") are matched with and grounded to concrete objects in the 3D scene model 370. This includes both the classification and the localization of relevant objects relative to the vehicle, taking into account spatial relationships (e.g., "in front of," "next to," "between") and contextual conditions.

[0092] Based on the created scene model 430, the MLLM 400 performs an analysis of the driving task and generates a vehicle trajectory 450 adapted to the user's requirements. The vehicle trajectory is not calculated by a rule-based subsystem or a task-specific trained network, but is generated entirely within the MLLM 400, i.e., in an end-to-end processing step from the spoken task description (speech input 210) and the acquired sensor data 320 for environmental perception. The generated vehicle trajectory 450 is then passed to the control module 700 for execution.

[0093] In a preferred embodiment, the MLLM 400 includes an optional self-reflection mechanism configured to check the generated vehicle trajectory 450 for plausibility, drivability, and safety, and to optimize it if necessary. This self-reflection mechanism can be implemented based on internal model evaluations or simulated feedback steps and enables iterative improvement of the planning output.

[0094] In a preferred embodiment, the multimodal language model MLLM 400 is based on a neural network with a transformer-based architecture, which has been pre-trained on a very large scale using multimodal datasets consisting of text, image, sensor, and spatial context information. The training typically takes place within a self-supervised learning process, in which semantic and spatial correlations across different modalities are learned to enable the MLLM 400 to recognize complex relationships between speech, image segments, and 3D point information.

[0095] Within the scope of the present invention, however, the MLLM 400 is not trained specifically for a given task, i.e., not specifically for parking maneuvers or driving dynamics. Rather, a generically pre-trained model is used in a zero-shot or few-shot configuration, in which verbally described driving tasks are directly translated into model-internal representations and context-dependently implemented into suitable vehicle trajectories. This allows for flexible, adaptively interpretive use without requiring a specialized submodel for each application.

[0096] The trajectory module 500 is designed to translate the vehicle trajectory 450 generated by the multimodal language model (MLLM) 400 into controllable driving commands for the vehicle's actuators. The trajectory module 500 thus functions as a technical translation unit between the abstract, model-internally generated vehicle trajectory 450 and the concretely executable control parameters 550 of a vehicle.

[0097] In an advantageous embodiment, the vehicle trajectory 450 output by the MLLM 400 is presented as a time-discrete sequence of vehicle states, for example, in the form of positions, curve radii, or speed profiles. The trajectory module 500 converts this information into specific vehicle dynamics control parameters 550 or control variables such as steering angle, lateral and longitudinal accelerations, target speeds, or braking force values. This conversion preferably takes place in real time or near real time to ensure continuous vehicle control.

[0098] In a preferred embodiment, the trajectory module 500 additionally includes a safety and plausibility check. This check is designed to analyze the generated trajectory 450 with regard to drivability, collision risks, and comfort criteria. Among other things, it can be checked whether minimum distances to static and dynamic objects are maintained, whether the vehicle's dynamic limits are observed, particularly with regard to permissible accelerations, turning radii, or steering angle changes, and whether abrupt or uncomfortable driving maneuvers are avoided.

[0099] The verified and, if necessary, corrected driving commands are then transmitted as control parameter 550 to the control module 700, which initiates their physical execution in the vehicle environment. The trajectory module 500 thus represents an interface between the speech-based and AI-based planning level and the real-time capable control units 770 of the vehicle.

[0100] The control module 700 is designed to convert the vehicle dynamics control parameters 550 generated and verified by the trajectory module 500 into control commands 750 and to transmit these to the vehicle's control units 770, thereby initiating their physical execution. The control module 700 thus forms the interface between the software-based driving decision logic and the actual actuators of the vehicle.

[0101] The control commands 750 executed by the control module 700 relate in particular to the control of the steering, the braking system, and the powertrain. The control module 700 accesses existing vehicle interfaces such as the CAN bus (Controller Area Network), the FlexRay bus, or other communication protocols established in the automotive industry. Access is via standardized command formats, as already used in the context of existing advanced driver assistance systems (ADAS).

[0102] In a preferred embodiment, the control module 700 is designed such that no modifications to the existing vehicle architecture are required. Instead, the integration of the functions according to the invention is achieved as a so-called "balcony solution," meaning that the control module 700 is functionally positioned above the existing control level and uses the existing vehicle interfaces and control units merely as actuators. This enables simple retrofitting or integration into existing platforms without requiring extensive modifications to safety-critical systems.

[0103] The 700 control module can also work with feedback from vehicle-internal sensors, for example, for position control, steering angle confirmation, or monitoring of drive behavior. This feedback can be used in particular to monitor trajectory execution or to validate the vehicle's response to the 750 control commands.

[0104] In an advantageous embodiment, the control module 700 comprises a prioritization or transfer layer through which the generated control commands 750 can be seamlessly coordinated with existing driving functions, for example in situations where manual intervention by the driver takes place or vehicle-internal safety mechanisms take precedence.

[0105] In an extended embodiment, the control module 700 can incorporate additional safeguards to prevent erroneous control commands or misinterpretations. For example, critical driving or control commands can be safeguarded by the emergency braking and collision avoidance assistants integrated as standard in the vehicle. Furthermore, the MLLM 400 can be secured by an internal plausibility check loop. This involves targeted self-queries to verify the contextual consistency and compatibility of the generated suggestions with the overall context of the parking scenario. Such procedures can be particularly helpful in detecting and correcting potential misinterpretations or "hallucinations" of the MLLM 400 at an early stage.

[0106] The output module 800 is designed to provide the user with feedback on the current status, execution, or planned action of the system 100 according to the invention. The output module 800 serves in particular to facilitate transparent, explainable, and interactive communication between the system 100 and the user. It can be integrated, together with the speech input module 200, into a common processing unit or operating component, for example, in an in-vehicle infotainment system or a mobile device application.

[0107] In a preferred embodiment, the output module 800 generates a speech output 830 to provide system states, confirmations, or queries to the user in acoustic form. The speech output 830 can be generated locally by an embedded speech synthesis system, for example, using a resource-efficient model such as eSpeak, or alternatively, it can be generated in the cloud. eSpeak is a cross-platform text-to-speech system for synthetic speech output that is particularly suitable for resource-efficient applications such as embedded vehicle modules. The speech output 830 makes it possible to inform the user about the progress of the parking process, for example, through natural language feedback such as "Okay, I'm parking in front of the red Mercedes," or "There are currently only disabled parking spaces in the shade, but I can park in front of the black Honda."

[0108] Additionally, the output module 800 can create a graphical visualization 850 that visually displays the planned trajectory, detected target objects, or the current system status to the user on the infotainment system display. This increases the user's comprehensibility of the automated action.

[0109] In an extended embodiment, the output module 800 can also be used to interact with the user, for example, to allow the user to influence the parking process through voice correction or a cancel command. For instance, the user can respond to feedback with a voice command such as "No, please park in front of the white SUV" or "Cancel operation." In this case, the voice input module 200 is reactivated to process the modified command.

[0110] The output module 800 is thus functionally closely linked to the speech input module 200 and the MLLM 400, and together with these modules forms the interface to user interaction in the sense of a multimodal operating concept with a natural interaction logic.

[0111] A typical application scenario for the present invention arises when parking a vehicle in an inner-city residential area with densely parked vehicles along the roadside. The vehicle user drives along a row of parked vehicles and notices a potentially suitable gap between a red delivery van and a white SUV. The driver wants the vehicle to park itself automatically, without having to steer manually, because the parking space is very tight.

[0112] To do this, he speaks the command "Park in front of the red van" via the vehicle's integrated voice interface or a connected smartphone. The voice input 210 is captured by the voice input module 200, whereby the speech-to-text component 230 first converts the spoken instruction into machine-readable text as a text representation 250. Subsequently, the parsing component 270 semantically analyzes the text representation 250 and creates a structured input representation 290 from it. This contains relevant action information, object descriptions, and spatial relationships, which are passed as a prompt to the multimodal language model MLLM 400.

[0113] In parallel, the environmental perception module 300 captures the vehicle's current surroundings using cameras, lidar and radar sensors of the sensor unit 340. The sensor data 320 are fused in the sensor data processing unit 350 to form a semantically annotated, three-dimensional scene model 370, which includes detected objects, spatial arrangements and free spaces.

[0114] The MLLM 400 receives a multimodal input 410, composed of the structured input representation 290 and the 3D scene model 370. It links the objects named in the speech input 210 with the actually recognized elements in a scene model 430 ("grounding") and analyzes the driving task on this basis. Taking into account vehicle dimensions, comfort parameters, and available maneuvering areas, a suitable vehicle trajectory 450 is generated entirely within the MLLM 400 in a continuous end-to-end process without separate rule-based subsystems.

[0115] This vehicle trajectory 450 is converted by the trajectory module 500 into specific vehicle dynamics control parameters 550, such as steering angle, target speed, and braking forces. Additionally, a safety and plausibility check is performed to ensure drivability, obstacle clearance, and comfort criteria. The verified control parameters 550 are then transferred to the control module 700, which converts them into control commands 750 and transmits them via existing vehicle interfaces (e.g., CAN bus) to the control units 770 for steering, braking, and drive.

[0116] The vehicle then automatically performs the parking maneuver – including all necessary steering, braking, and propulsion operations. During this time, the user can optionally receive feedback on the system status via the output module 800. This feedback is provided, for example, through a voice output 830 ("Parking operation in progress" or "I am beginning to park in front of the red vehicle") or through a graphical visualization 850 in the infotainment system.

[0117] Furthermore, the system 100 according to the invention is able to react flexibly to changing situations – e.g., suddenly appearing obstacles or displaced objects – and, if necessary, adjust the planning or initiate a new voice dialogue. The user can also correct or cancel the process by means of further voice input.

[0118] This scenario illustrates the capabilities of the system according to the invention, particularly with regard to the interplay of intuitive voice input, semantic-spatial environmental perception, and fully automatic driving control. It demonstrates that the user can initiate a complex maneuver with minimal effort, without requiring detailed input regarding position, dimensions, or steering strategy.

[0119] The present invention offers a multitude of technical and application-specific advantages through its modular architecture. The separation of the system components into functionally clearly defined modules—such as the speech input module 200, the environmental perception module 300, the MLLM 400, the trajectory module 500, the control module 700, and the output module 800—creates a structured, transparent, and adaptable system architecture. Each module is developed, tested, and validated independently. This facilitates both the targeted further development of individual components and troubleshooting and maintenance during operation. Furthermore, modules can be replaced or supplemented with alternative implementations without altering the overall architecture.

[0120] Furthermore, the architecture is designed for interoperability with existing vehicle platforms. By using standardized interfaces – for example, for voice input, sensor data acquisition, or vehicle integration via CAN or FlexRay bus – the System 100 can be coupled with existing control units. Integration can be implemented as a so-called "balcony solution" above the existing vehicle systems, without deeply interfering with safety-critical subsystems.

[0121] The modularity also allows for high scalability of System 100. Beyond the existing parking function, the architecture can easily be extended to include further automated driving maneuvers, such as automated maneuvering, precise entry into narrow courtyards, or exiting parking garages. By using a multimodal language model 400 with generalizing inference capabilities, System 100 is not limited to specific use cases but is adaptable to varying driving tasks and environmental conditions.

[0122] Another key advantage lies in the traceability of system decisions. The clear functional separation of speech processing, semantic-spatial scene analysis, and control enables transparent analysis of behavior in safety-critical situations. This supports both internal functional verification and external certifiability, for example, within the framework of approval procedures for highly automated driving functions (HAD) or automated driving systems (ADS).

[0123] Overall, the architecture according to the invention enables a powerful, adaptive and at the same time comprehensible realization of automated parking processes based on natural language input and intelligent scene interpretation.

[0124] In Fig. Figure 2 shows the procedure steps for the automated execution of a parking operation of a vehicle.

[0125] In step S10, a voice input 210 with a natural language parking instruction from a vehicle user is received via a voice input module 200.

[0126] In step S20, the speech input 210 is converted into a text representation 250 with a machine-readable text format using a speech-to-text component 230.

[0127] In step S30, the text representation 250 is semantically analyzed by a parsing component 270 and a structured input representation 290 is generated from it.

[0128] In step S40, the vehicle environment is captured by means of a sensor unit 340, which includes at least one camera and a LiDAR sensor, whereby sensor data 320 are recorded.

[0129] In step S50, a semantically annotated, three-dimensional scene model 370 is generated by fusion of the sensor data 320 in a sensor data processing unit 350.

[0130] In step S60, the structured input representation 290 and the scene model 370 are processed by a multimodal language model MLLM 400, whereby the MLLM 400 generates a vehicle trajectory 450 corresponding to the language input 210.

[0131] In step S70, the generated vehicle trajectory 450 is converted into vehicle dynamic control parameters 550 by a trajectory module 500.

[0132] In step S80, the control parameters 550 are converted into control commands 750 and the parking process is carried out by a control module 700 via vehicle-side control units 770.

[0133] Fig. Figure 3 schematically represents a computer program product 900 comprising an executable program code 950 configured to perform the method according to the first aspect of the present invention.

[0134] The invention enables a natural and situation-adaptive human-machine interaction, allowing the user to automate complex parking scenarios using only voice commands, such as "Park in front of the red van in the shade", without having to specify exact coordinates or specific driving maneuver parameters. Reference sign 100 System 200 voice input module 210 Voice input 230 Speech-to-text component 250 Text representation of speech input 270 Parsing component 290 Input representation 300 Environmental Perception Module 320 sensor data points (camera, lidar, radar) 340 sensor unit (camera, lidar, radar) 350 sensor data processing unit 370 3D scene model 400 Multimodal Language Model (MLLM) 410 Multimodal input 430 Semantic-spatial scene model (Grounding) 450 Generated vehicle trajectory 500 Trajectory Module 550 Vehicle dynamics control parameters 700 control module 750 control commands for physical implementation 770 control units (steering, brake, drive) 800 output module 830 Speech output 850 Graphical Visualization 900 computer program product 950 program code

Claims

Method for the automated execution of a vehicle parking operation, comprising the following steps: - Receiving (S10) a speech input (210) with natural language parking instructions from a vehicle user via a speech input module (200); - Converting (S20) the speech input (210) into a text representation (250) with a machine-readable text format using a speech-to-text component (230); - Semantic analysis (S30) of the text representation (250) by a parsing component (270) and generating a structured input representation (290), wherein the input representation (290) contains the relevant action information, object descriptions, and spatial relationships, which are passed as a prompt to a multimodal language model (MLLM) (400); - Capturing (S40) the vehicle environment using a sensor unit (340) comprising at least one camera and a LiDAR sensor, and recording sensor data (320);- Generating (S50) a semantically annotated, three-dimensional scene model (370) by fusion of the sensor data (320) in a sensor data processing unit (350), wherein the sensor data processing unit (350) provides the 3D scene model (370) as a point cloud representation;- Processing (S60) the structured input representation (290) and the scene model (370) by a multimodal language model (MLLM) (400), wherein the structured input representation (290) and the semantically annotated 3D scene model (370) are combined into a multimodal input (410) and fed together to the MLLM (400), wherein the MLLM (400) generates a vehicle trajectory (450) corresponding to the language input (210), and wherein the MLLM (400) is configured to iteratively evaluate and optimize the generated vehicle trajectory within the framework of an internal model self-reflection, wherein the evaluation is based on internal model evaluations and / or simulated feedback steps, and a modified vehicle trajectory is generated if the trajectory is negatively evaluated; - Converting (S70) the generated vehicle trajectory (450) into vehicle dynamics control parameters (550) by a trajectory module (500);- Conversion (S80) of the control parameters (550) into control commands (750) by a control module (700) and execution of the parking operation via vehicle-side control units (770), wherein the control module (700) executes the vehicle trajectory (450) using existing vehicle interfaces (e.g. CAN bus) without resorting to separately pre-trained, rule-based subsystems for trajectory planning.; Method according to claim 1, wherein the sensor data processing unit (350) additionally provides the 3D scene model (370) as an image representation. Method according to claim 1 or 2, wherein the segmentation of the camera data is carried out by a segmentation model (e.g. SAM) and the lidar data are assigned to the image segments by lidar-to-mask mapping. Method according to one of the preceding claims, wherein the user receives feedback on the parking process via a voice output (830) or a graphical visualization (850). Method according to one of the preceding claims, wherein the user can correct or cancel the parking process by means of a new voice input (210), wherein the voice input module (200) then generates a new structured input representation (290) and the processing in the MLLM (400) is performed again. System (100) for the automated execution of a parking operation of a vehicle, comprising a speech input module (200) for receiving a speech input (210) with a parking instruction in natural language with a speech-to-text component (230) and a parsing component (270) for converting and analyzing the speech input into a structured input representation (290), wherein the input representation (290) contains the relevant action information, object descriptions and spatial relationships, which are intended to be passed as a prompt to a multimodal language model (MLLM) (400);an environment perception module (300) with a sensor unit (340) for acquiring environment data (320), comprising at least one camera and a lidar sensor, and a sensor data processing unit (350) for fusing the sensor data (320) and generating a semantically annotated, three-dimensional scene model (370), wherein the sensor data processing unit (350) is configured to provide the 3D scene model (370) as a point cloud representation; a multimodal language model (MLLM) (400) configured to process the structured input representation (290) and the 3D scene model (370) to generate a vehicle trajectory (450) that corresponds to the user instruction in the language input (210); a trajectory module (500) for converting the trajectory (450) into vehicle dynamics control parameters (550);and a control module (700) for converting the control parameters (550) into control commands (750) and for transmitting them to the vehicle-side control units (770), wherein the system (100) is configured to combine the structured input representation (290) and the semantically annotated 3D scene model (370) into a multimodal input (410) and to provide it to the MLLM (400) for further processing; wherein the MLLM (400) is configured to iteratively evaluate and optimize the generated vehicle trajectory within the framework of an internal model self-reflection, wherein the evaluation is based on internal model evaluations and / or simulated feedback steps, and a modified vehicle trajectory is generated if the trajectory is negatively evaluated; and wherein the control module (700) is configured to execute the generated trajectory (450) without recourse to task-specific trained submodels or rule-based planning logics. System (100) according to claim 6, wherein the sensor fusion is performed by lidar-to-image mapping or by a bird's-eye view method. System (100) according to claim 6 or 7, wherein an output module (800) is configured to produce a speech output (830) and / or a visualization (850). System (100) according to one of claims 6 to 8, wherein the user can correct or cancel the parking process by means of a new voice input (210), wherein the voice input module (200) is configured to generate a new structured input representation (290) thereafter, and wherein the MLLM (400) is configured to execute the processing again. Computer program product (900) comprising an executable program code (950) configured to execute the method according to any one of claims 1 to 5.

Images