LLM-based querying of functionalities in the distributed vehicle network

Abstract

The invention relates to a method for processing and using information about an electronic structure of a motor vehicle (1), wherein a database (2) is used (S1) which stores technical data about hardware properties and / or software properties of the electronic structure of the motor vehicle (1), wherein, upon a user request (S2) in natural language at a user interface (4), a Large Language Model is caused (S3) to semantically process (S4) the user request in connection with the data of the database (2) read out by means of a Retrieval Augmented Generation Module and, based on the semantic processing, to output a human-readable answer to the user request (S5).

Description

[0001] The invention relates to a method for processing and using information about an electronic structure of a motor vehicle, as well as a system for processing and using information about an electronic structure of a motor vehicle.

[0002] A modern electronic architecture of a motor vehicle, especially a passenger car, typically comprises distributed systems with numerous ECUs (electronic control units), sensors, actuators, software-based functions, and APIs. The ECUs typically communicate via various bus systems such as CAN, FlexRay, Ethernet, or others, sharing signals in diverse ways, for example, cyclically, event-driven, on demand, via server-client communication, microservices, etc. The associated communication matrix is ​​highly complex and often lacks sufficient documentation. Individual APIs, on the other hand, are more frequently accompanied by some documentation. When the software implementation of the electronic architecture changes, the corresponding documentation is often outdated. Furthermore, developers often lack direct access to the API code.This makes development complex, time-consuming, and often requires obtaining information through personal contact.

[0003] US 2004 / 015809 A1 concerns a program generation system, comprising: a specification database configured to contain middleware information based on a middleware specification, and a middleware code generation module operatively coupled to the specification database and configured to generate middleware integration code based on the middleware information.

[0004] The object of the invention is to facilitate for a user the use of information from a database containing technical data on hardware properties and / or software properties of an electronic structure, i.e., electronic architecture, in particular with data buses and control units, of a motor vehicle.

[0005] The invention is defined by the features of the independent claims. Advantageous further developments and embodiments are the subject of the dependent claims.

[0006] A first aspect of the invention relates to a method for processing and using information about an electronic structure of a motor vehicle, wherein a database is used that stores technical data about hardware properties and / or software properties of the electronic structure of the motor vehicle, wherein, in response to a user request in natural language at a user interface, a Large Language Model is caused to semantically process the user request in connection with the data from the database read by means of a Retrieval Augmented Generation Module and, based on the semantic processing, to output a human-readable answer to the user request.

[0007] The user request can be made via voice input or directly in text form. Example user requests include a request for API usage in a specific vehicle type and software version, and "How can I retrieve the outside temperature in vehicle type XX with software version YY? How does the API call work?".

[0008] The human-readable language of the response can include natural language, program code, or a table containing human-readable information such as numbers, symbols, or similar elements. While natural language can be output audibly as speech or displayed as text on a screen, program code, unlike binary code, is human-readable text but can also be processed by a computer. For example, a script can be output that can be directly read by an interpreter, or program code to be compiled, such as for the C programming language. Other examples of programming languages ​​in which program code is preferred are Python and Rust. Additional programming languages ​​can be supported, depending on which ones the Large Language Model (LLM) has been pre-trained for.This approach particularly leverages the fact that Large Language Models can explain program code in natural language, based on the program code itself and its comments. This prevents errors in responses caused by documentation not being updated to reflect the latest software update. Large Language Models can also output pseudocode, which, while not necessarily directly readable and executable by a computer, provides users with clear guidance on how to translate it into program code in a familiar programming language.

[0009] The user interface allows a user to submit a request to the Large Language Model in natural language, which the model then interprets semantically. In addition to the request, the Large Language Model uses database data retrieved by a Retrieval Augmented Generation module. Unlike the Large Language Model's internal, non-interpretable parameters, the request information is used as input. Specifically, the database data is treated as equivalent, meaning it is used within the same context. The Large Language Model thus considers the database data as evidence, rather than providing a purely statistical response to the request.

[0010] In other words, the Large Language Model considers the database data as evidence in the response it generates, rather than generalizing from previously trained data. This means the Large Language Model's response is based directly on the database data without any loss of information. The Retrieval Augmented Generation module is specifically capable of generating an abstracted and structured representation of the database, providing the Large Language Model with context for the user query.

[0011] The database contains technical knowledge about hardware and / or software characteristics of an electronic structure of the motor vehicle, also called the E / E architecture, and thus about elements such as ECUs (abbreviation for "electronic control unit"), sensors, actuators, functions represented in software, buses and / or APIs (abbreviation for "application program interface"), and can therefore be easily and continuously updated.The advantage of using a Retrieval Augmented Generation module, abbreviated 'RAG' module, lies in the fact that the most recent state of the database can be used in the RAG module without requiring a new training process for the Large Language Model. This is achieved by re-reading and parsing the data, splitting it into junk, and creating embeddings based on these junk to obtain an abstracted knowledge base that can be read by a retriever and used by the Large Language Model. Specifically, this is accomplished by processing parts of the knowledge base relevant to the user request within a shared context with the user's query in the Large Language Model. Therefore, the Large Language Model should be chosen with a sufficiently large context window.

[0012] Large Language Models (LLMs) are a methodology from the field of so-called "natural language processing" because linguistic information is represented in a very large model, typically a neural network, which can generalize and draw conclusions, particularly at the semantic level. A Retrieval Augmented Generation (RAG) module serves to combine information retrieval methods with LLM methods.

[0013] The advantages of this method are faster and more personalized access to information for developers, resulting in shorter development times and higher software quality for the vehicle.

[0014] According to an advantageous embodiment, the human-readable language comprises natural language that represents a response to the user request, and wherein the human-readable language comprises program code if the user request relates to a programming task.

[0015] According to a further advantageous embodiment, the Large Language Model also generates a test case that can be executed along with the program code. Preferably, the program code and / or the test case includes an API call for the electronic structure of the motor vehicle. Advantageously, the Large Language Model can automatically generate documentation for the created program code.

[0016] According to a further advantageous embodiment, the human-readable language comprises program code that has the property of addressing a hardware component and / or a software component of the electronic structure of the vehicle, and wherein the program code is executed automatically.

[0017] Advantageously, the system can execute the generated program code in a development environment that represents the vehicle in real life, as hardware-in-the-loop, as closed / open loop or in a simulation, and then verify whether the generated call works correctly.

[0018] According to another advantageous embodiment, automatic verification of the program code is carried out by executing the program code.

[0019] Preferably, if an error occurs during the automatic execution of the program code, at least one of the following reactions is performed: - a repetition of the execution of at least the Large Language Model, optionally with additional input information; - an output of information for the user about the error; - an output of information for a developer responsible for an API, software architecture and / or documentation, if the error is suspected to be there; - Automatic sending of an email or creation of a bug ticket in a repository with information about the bug. This creation preferably happens automatically using the Large Language Model, or another language model.

[0020] According to a further advantageous embodiment, the program code is executed in a development environment, wherein the hardware component and / or software component of the vehicle's electronic structure addressed by the program code is used in the integrated state, as components separated from the vehicle on a hardware-in-the-loop test bench, or represented in a simulation environment to verify the program code. The hardware-in-the-loop test bench can be an open-loop or a closed-loop test bench.

[0021] According to another advantageous embodiment, the Retrieval Augmented Generation module processes the database data as ground truth within a context window of the Large Language Model.

[0022] According to a further advantageous embodiment, the technical data of the database includes at least one of the following: a textual description of functions, APIs, sensors, actuators, and / or test cases; a software version number; program code of the functions, APIs, actuators, sensors, and / or test cases; release notes; a vehicle network matrix between ECUs, sensors, and actuators; a description of bus signals; value ranges of bus signals; and a transmission method on the data bus. The database data may contain code from software of the vehicle's electronic structure and must, in particular, be versioned and / or contain links to vehicle types, model years, software variants, etc.

[0023] According to a further advantageous embodiment, the Large Language Model detects and marks any discrepancy between documentation stored in the database and software stored in the database relating to the electronic structure of the motor vehicle, and / or informs the user and / or automatically corrects the documentation.

[0024] Another aspect of the invention relates to a system for processing and using information about an electronic structure of a motor vehicle, comprising a database that stores technical data about hardware properties and / or software properties of the electronic structure of the motor vehicle, and comprising a computing unit designed to prompt a Large Language Model, in response to a user request in natural language at a user interface of the system, to semantically process the user request in connection with the data from the database read by means of a Retrieval Augmented Generation Module and to output a human-readable answer to the user request based on the semantic processing.

[0025] Advantages and preferred further developments of the proposed system result from an analogous and substantive transfer of the above statements made in connection with the proposed procedure.

[0026] Further advantages, features and details will become apparent from the following description, in which - possibly with reference to the drawing - at least one embodiment is described in detail.

[0027] They show: Fig. 1: A method for processing and using information about an electronic structure of a motor vehicle according to an embodiment of the invention. Fig. 2: A test rig with a vehicle and a system according to an embodiment of the invention.

[0028] The representations in the figures are schematic and not to scale.

[0029] Fig. Figure 1 shows a method for processing and using information about the electronic structure of a motor vehicle. The electronic structure is characterized by the hardware and / or software properties of the electronics and computer technology used in the motor vehicle. A technical description of these properties is contained in a database 2, which is provided and used below (S1), but is not very practical for user review. However, the user can submit a user request (S2) in natural language to a user interface 4, in particular by voice input or typed text input. This user request (S2) is provided to a Large Language Model (LLM), which then processes the user request semantically (S3) in conjunction with the data from database 2, which is read by means of a Retrieval Augmented Generation (RAM) module (S4).For this purpose, after an update of database 2, it is ensured that all recent data from database 2 have been read using a parser, divided into chunks, and semantically interpreted, i.e., transformed into text embedding within an embedding space. This allows a retriever of the Retrieval Augmented Generation module to search the embedding space for relevant parts, with relevance being defined by the user query. Thus, the user query can be used in its semantic context to search the transformed database 2, and an evidence-based response to the user query can be output in human-readable language (S5).

[0030] Fig.Figure 2 shows a test setup with a real motor vehicle 1, which is embedded in a hardware-in-the-loop configuration via interfaces. A database 2 is available, which stores technical data about the hardware and software properties of the electronic structure of the motor vehicle 1. A computing unit 3 has access to the database 2 and features a user interface 4 that allows it to accept a user query in natural language. The following query is entered as an example: "How can I retrieve the outside and inside temperatures in vehicle type X with software version Y or higher? How does the API call from ECU2 work? What would a program code look like that sets the seat heating parameters depending on the outside and inside temperatures and calls the seat heating API?"A Large Language Model executed on compute unit 3, either on the local compute unit 3 or in the cloud, generates a RAG call and delivers the following example result: . "The indoor temperature is determined by a sensor with an accuracy of 0.1°C and is read by ECU1 and then sent to ECU2 via a CAN bus. The API call is: Get_inside_temp_from_ECU1:API20). The outdoor temperature is determined by a sensor with an accuracy of 0.2°C and is read by ECU1 and then sent to ECU2 via a CAN bus. The API call is:" The function `Get_outside_temp_from_ECU1:APl3()` is used to control the seat heating. To activate the seat heating, a level within the range [0..3] must be set, where "0" is off and "3" is the highest level. An API for this control is available on ECU2: `ECU2:API3()`. For example, the function `request_seatheating=3` sets the seat heating to level 3, thus activating the seat heating actuator. If the setting has been correctly applied, the seat heating confirms with the signal `set_seatheating_set_successfully = True`.

[0031] Another example of user input would be: "Write executable program code for vehicle type XX with software version YY that opens all windows for at least three seconds when the unlock button in the vehicle is pressed!"

[0032] The executable program code generated by the Large Language Model is uploaded as a software update to the corresponding vehicle control unit of vehicle 1 on the test bench and is thus ready for use. In addition to documentation and commentary of the program code, the Large Language Model also generates an automatic test case. This is an example of such a test case: - Close the vehicle windows; - Switch on the vehicle ignition; - Activate the unlock button for at least three seconds; - wait at least five seconds; - Test condition: Check if all windows are open.

[0033] The test setup uses a camera that detects all moving parts of vehicle 1, such as windows, doors, seatbelt pretensioners, wheels, engine parts, active aerodynamic elements, as well as lights, button illumination, displays, and other relevant information. Multiple cameras may be necessary inside and outside vehicle 1, or alternatively, cameras mounted on a movable robot arm to allow for different perspectives. Additional external sensors, such as temperature sensors, microphones, motion sensors, strain gauges, etc., can also be used to monitor the execution of the test case. A variety of conditions for quality control are conceivable. For example, the initial condition "all windows closed" can be followed by a subsequent test condition "all windows open." Since speech outputs are often not identical, a similarity measure, e.g.,The similarity measure is created from weighted criteria of the various conditions. A typical similarity measure for problems involving natural language processing is the so-called "cosine similarity." However, many measures are known in the prior art that can be used here. In particular, a corresponding similarity or quality measure is compared with a predefined threshold. The automatic tracking of the test case allows processing unit 3 to check for errors in the execution of the generated program code. The error can be contained in the generated program code or in a previously created API; that is, it is possible to find such an error or missing features in existing program code. If such an error is found, it can be corrected manually, or a new user request can be made via user interface 4.

[0034] Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 2004 / 015809 A1

[0003] Cited non-patent literature

[0000] Write executable program code for vehicle type XX with software version YY that opens all windows for at least three seconds when the unlock button in the vehicle is pressed!

[0031]

Claims

[1] Method for processing and using information about an electronic structure of a motor vehicle (1), wherein a database (2) is used (S1) which stores technical data about hardware characteristics and / or software characteristics of the electronic structure of the motor vehicle (1), wherein, in response to a user request (S2) in natural language at a user interface (4), a Large Language Model is caused (S3) to semantically process (S4) the user request in connection with the data from the database (2) read by means of a Retrieval Augmented Generation Module and to output a human-readable answer to the user request based on the semantic processing (S5). [2] The method of claim 1, wherein the human-readable language comprises natural language that represents a response to the user request, and wherein the human-readable language comprises program code if the user request relates to a programming task. [3] Method according to claim 2, wherein the Large Language Model generates a test case executable with the program code. [4] Method according to any of the preceding claims, wherein the human-readable language comprises program code having the property of addressing a hardware component and / or a software component of the electronic structure of the vehicle, and wherein the program code is executed automatically. [5] Method according to claim 4, wherein automatic verification of the program code is carried out by executing the program code. [6] Method according to claim 5, wherein the program code is executed in a development environment, wherein the hardware component and / or software component of the electronic structure of the motor vehicle (1) addressed by the program code in the state integrated into the motor vehicle (1), as components separated from the motor vehicle (1) on a hardware-in-the-loop test rig, or represented in a simulation environment, are used to verify the program code. [7] Method according to one of the preceding claims, wherein the Retrieval Augmented Generation module processes the data of the database (2) as ground truth within a context window of the Large Language Model together with the user input. [8] Method according to any one of the preceding claims, wherein the technical data of the database (2) comprises at least one of the following: a textual description of functions, APIs, sensors, actuators, and / or test cases; a software version number; program code of the functions, APIs, actuators, sensors, and / or test cases; release notes; a vehicle network matrix between ECUs, sensors, and actuators; a description of bus signals; value ranges of bus signals; a transmission mode; [9] Method according to one of the preceding claims, wherein the Large Language Model detects and marks a discrepancy between documentation stored in the database (2) and software of the electronic structure of the motor vehicle (1) stored in the database (2), and / or informs the user and / or automatically corrects the documentation. [10] System for processing and using information about an electronic structure of a motor vehicle (1), comprising a database (2) which stores technical data about hardware characteristics and / or software characteristics of the electronic structure of the motor vehicle (1), and comprising a computing unit (3) which is designed to prompt a Large Language Model to semantically process the user request in natural language at a user interface (4) of the system in connection with the data in the database (2) read by means of a Retrieval Augmented Generation Module and to output a human-readable answer to the user request based on the semantic processing.

Images