Intelligent assistant for a motor vehicle

Abstract

A method (200) for controlling a motor vehicle (105) comprises steps of capturing (9) a verbal utterance from a person (115) on board the motor vehicle (105); wherein the utterance includes a control instruction relating to an aspect of a driving state of the motor vehicle (105); wherein information relating to the driving state of the motor vehicle (105) is stored in a data middleware (205); determining, by means of a large language model (250), a database query to determine the aspect in a graph-based query language based on the utterance; determining the aspect of the driving state based on the database query using the data middleware (205); and executing (13) the control instruction taking into account the aspect of the driving state.

Description

[0001] The present invention relates to an information system on board a motor vehicle. In particular, the invention relates to the control of a voice-capable intelligent assistant on board a motor vehicle.

[0002] A function of a motor vehicle can be controlled by voice command. For this purpose, a spoken utterance from a person on board can be recorded and evaluated, and a function assigned to a recognized voice command can be triggered accordingly.

[0003] The meaning of an utterance can depend on the context in which it was made. For example, the voice command "Please open my window" might refer to the window of the vehicle closest to the speaker. Typical language processing is unable to determine such a context because it lacks the necessary information.

[0004] Speech processing can be performed using a Large Language Model (LLM), which already possesses a certain amount of world knowledge. However, information specific to the vehicle or the speaker is not available. Furthermore, training an LLM can be very complex, making it difficult to add new functions to a trained speech processor.

[0005] German patent application DE 10 2016 200 815 A1 discloses a method and a device for voice control of a motor vehicle. A server located outside the vehicle is configured to process a voice command and can access vehicle-specific data stored in a memory located outside the vehicle. The server is further configured to access vehicle-specific data in order to process the voice command. The motor vehicle has a voice input unit for a user, in particular a driver of the motor vehicle, to input a voice command, a communication unit for data communication with the server located outside the vehicle, and a data acquisition unit for acquiring vehicle-specific data.The method captures a voice command via the voice input unit, transmits a voice file representing the voice command to the server via the communication unit, captures the vehicle-specific data via the data acquisition unit, transmits the vehicle-specific data to the server via the communication unit, and processes the voice command.

[0006] Publication CN 1 17 609 470 A discloses a construction method for a question-and-answer system and an intelligent data management platform based on the large language model and the knowledge graph. This solves the problem of potentially overlooking important information when a traditional model processes discrete variables, thereby achieving more accurate and comprehensive data analysis. The user query is transformed into a database command using natural language processing, and the knowledge graph is generated by combining structured and unstructured data, thus improving the efficiency and accuracy of data querying.

[0007] Publication CN 1 18 364 087 A discloses a method and a system for improving the querying and answering of questions based on a database provided.The process comprises the following steps: capturing various document types containing text and diagram information; performing text segmentation processing and attribute extraction and storage of diagram information; converting the text information into a vector format, and encoding diagram attributes accordingly and storing them in a vector database; creating an index containing the document key information in the database; analyzing a problem proposed by a user using a large model, understanding and vectorizing the user's intent; matching vectorized questions with indexes in a database, finding and sorting related documents or information fragments, inputting highly relevant information into a large language model, generating preliminary answers, and further refining the answers.According to this method, the query strategy can be dynamically adapted to the user's requirements while ensuring data security; the method is implemented through a flexible database query language and efficient indexing technology, thereby improving the system's response speed and query efficiency.

[0008] German patent application DE 10 2021 120 246 A1 discloses a speech recognition system for a vehicle, wherein the speech recognition system has at least one receiving unit for receiving speech input from a user, wherein the speech recognition system has a determination unit which is configured to determine at least one vehicle-specific context information, and a processing unit which is configured to at least partially process the speech input which has one or more keywords, wherein the processing unit is further configured to interpret the speech input based on at least one of the keywords and the vehicle-specific context information and to execute an action corresponding to the speech input.

[0009] German patent application DE 10 2015 213 722 A1 discloses a method for operating a speech recognition system in a vehicle, as well as a speech recognition system. In the method according to the invention, at least one voice input from a user is captured, and data relating to the context of the voice input is recorded. A recognized text of the voice input is generated, and a semantic analysis of the recognized text is performed. Finally, a function is executed based on the semantic analysis.

[0010] One of the problems underlying the present invention is to provide an improved technology for controlling a motor vehicle based on a voice command. The invention solves this problem by means of the subject matter of the independent claims. Dependent claims describe preferred embodiments.

[0011] A procedure for steering a motor vehicle comprises the steps of capturing a verbal utterance from a person on board the motor vehicle; wherein the utterance includes a steering instruction relating to an aspect of a driving state of the motor vehicle; wherein information relating to the driving state of the motor vehicle is stored in a data middleware; determining, by means of a large language model, a database query in a graph-based query language to determine the aspect based on the utterance; determining the aspect of the driving state based on the database query by means of the data middleware; and executing the steering instruction taking into account the aspect of the driving state.

[0012] The data middleware can receive and store information relating to the vehicle's driving state from a variety of sources. The driving state can be dynamically determined at any time based on the stored information. In particular, a specific aspect of the driving state can be identified by an engine appropriately linking relevant information. Examples of aspects of the driving state include vehicle speed, geographical position, or seat occupancy. The complete driving state can be determined by all available information. As explained in more detail below, the data middleware can also be deployed in a distributed manner, with part of the middleware running on a device outside the vehicle.

[0013] The LLM can analyze the person's utterance and recognize the control instruction. Furthermore, the LLM can determine which aspect of the driving state is required for selecting or parameterizing the control instruction and provide a suitable query.

[0014] The graph-based query language can query content from a predefined description system, such as the Resource Description Framework (RDF). This allows logical statements to be formulated about any item in a database. The query language can, in particular, incorporate SPARQL (SPARQL Protocol And RDF Query Language). This enables the retrieval of information about a specific aspect of the vehicle's condition through a suitable query.

[0015] Advantageously, a wider range of control instructions can be supported. A control instruction can be provided by the user in improved natural language. Contextual references or aspects of the driving state embedded in the control instruction can be recognized and resolved. The user's intention can be determined more accurately, and the verbally expressed control instruction can be recognized quickly and correctly. The intended instruction can then be executed directly and reliably.

[0016] In a preferred embodiment, the database query is written to a request list of the data middleware; and a response to the database query is written to the request list. The list can be subscribed to by a database engine, which executes the database query on the data in the data middleware. Artifacts of the data middleware can be readable, writable, and / or subscribeable, allowing arbitrary entities to communicate via an object, such as a list. This enables the implementation of asynchronous exchange, where a sender and a receiver are dynamically connected.

[0017] Similarly, according to the invention, the control instruction is written to a request list of the data middleware; and the current driving state is then passed to the control instruction. A suitable agent can be provided for executing control instructions in the request list. Tasks such as plausibility or safety checks can be implemented by the agent.

[0018] The database query can be determined based on a manifest. The manifest can describe which database queries are possible, how specific aspects of the vehicle's state can be determined, and which function is to be called to obtain a particular piece of information. The manifest is preferably in text form so that it can be processed by the LLM (Logical Lifecycle Management). This also makes the manifest easy to manage and allows for human review for consistency if necessary. The manifest can be easily modified, extended, or replaced. This allows for the easy addition or improvement of new functions or control instructions. An outdated or unsafe control instruction can be easily removed.

[0019] Preferably, the manifest includes a collection of predefined database queries. Each database query can be associated with an aspect of the vehicle's condition that can be determined by the query. The collection can be compiled and maintained by one or more experts. For example, a domain expert can contribute knowledge about the vehicle and its characteristics. A knowledge engineer can provide knowledge about processing information and determining a situation based on information stored in a database. Together, the experts can provide valuable database queries that can be used universally and efficiently to determine aspects of the vehicle's condition.

[0020] In another embodiment, the manifest includes a generic endpoint that returns different responses depending on the different SPARQL payload from the LLM.

[0021] In another embodiment, the database query is determined dynamically. For this purpose, the query can be formulated based on the aspect of interest at the moment a response is desired. More preferably, the database query is determined using a Large Language Model. This can be done using the aforementioned LLM or a separate, additional LLM. The LLM can be trained to determine the database query in a predetermined graph-based query language, in particular SPARQL.

[0022] According to a further aspect of the present invention, a system for controlling a motor vehicle comprises an input device for capturing a verbal utterance from a person on board the motor vehicle; wherein the utterance comprises a control instruction relating to an aspect of a driving state of the motor vehicle; data middleware in which information relating to the driving state of the motor vehicle is stored; a large language model (LLM) configured to determine a database query for identifying the aspect in a graph-based query language based on the utterance; and a query engine for the data middleware. The query engine is configured to determine the aspect of the driving state based on the database query; and an execution device for executing the control instruction taking into account the aspect of the driving state.

[0023] The system is preferably configured to partially or completely execute a method described herein. For this purpose, the system may include a processing unit, preferably of electronic design, which may, for example, comprise an integrated circuit, a programmable logic device, or a programmable microcomputer. The method may be implemented as a configuration or as a computer program product with program code means for the processing unit. The configuration or the computer program product may be stored on a computer-readable data carrier. Features or advantages of the method may be transferred to the device or vice versa.

[0024] The LLM can be implemented multimodally and work directly on acoustic data sampled from a speaker. Alternatively, a speech-to-text conversion can be provided to transform acoustic data into text data, which can then be supplied to a text-based LLM.

[0025] The data middleware can be implemented on multiple processing systems. Information can be regularly synchronized between these systems. One processing system might be located on board the vehicle, while another could be mounted externally. A third processing system could be mobile, running on a mobile device, for example. This means the location of the mobile processing system is irrelevant. The speed of information exchange can vary depending on the quality of the communication connection. If communication is unavailable, synchronization can be suspended.

[0026] The processing systems can have the same interfaces for working with data from the data middleware. In particular, objects in the data middleware can be accessed in a uniform manner. It is irrelevant on which processing system a read or write service resides.

[0027] For example, a person could provide a voice command on their mobile device, which would perform a speech-to-text conversion and write the result to a data middleware object. The LLM (Logical Media Manager) could be located remotely from the vehicle, such as in a data center. The LLM could extract the textual command from the object and write one or more database queries to another data middleware object. A database engine could execute the queries and write the results back to the object. An agent on board the vehicle could collect the results and configure a control function with them to achieve the desired effect. Synchronization of the objects between the individual processing systems is handled transparently by the data middleware.

[0028] The query engine can implement a Knowledge Graph (KG). The KG can include a stream reasoner that performs ontology-based reasoning. In a further embodiment of the invention, rules can also be provided in addition to the ontologies. In this case, stream reasoning can also be performed based on rules. The Knowledge Graph preferably includes a query engine for Graph Query Language, here, for example, SPARQL. It allows new SPARQL queries to be added statically and at runtime and dynamically provides an API endpoint for each dedicated SPARQL query, e.g., a REST API, and a generic SPARQL endpoint. The Knowledge Graph can also include code logic that manages Knowledge Graph tasks, e.g., SHACL conversions, SPARQL API calls, etc.

[0029] The system can also comprehensively include a Large Language Model (LLM) that is set up to dynamically determine the database query.

[0030] According to yet another aspect of the present invention, a motor vehicle comprises a device that implements, in whole or in part, a system described herein. It should be noted that the device does not necessarily include all elements of the system. However, an agent executing the control instruction should preferably be located on board the motor vehicle. The person speaking is also located in the vicinity of the motor vehicle, usually even on board, so that an element of the system that receives the utterance is preferably also located in the vicinity of the motor vehicle. In a further preferred embodiment, this element is fixedly attached to the motor vehicle and may, for example, comprise an interior microphone of the motor vehicle.

[0031] The invention will now be described in more detail with reference to the attached drawings, in which: Fig. 1. a system; and Fig. 2. A flowchart of a process illustrated.

[0032] Fig. Figure 1 shows a system 100. On board a motor vehicle 105 is a device 110, which is configured to control a function of the motor vehicle 105 or of a system or subsystem it comprises. It should be noted that the function may also include providing information, which can be determined on the basis of available information in the context of the motor vehicle 105.

[0033] The device 110 preferably comprises a processing unit 120, which may be connected to an interface 125 via which a predetermined function of the motor vehicle 105 or one of its systems can be controlled. A microphone 130 may be provided to capture a spoken utterance of the person 115. A loudspeaker 135 is optionally shown to provide acoustic feedback to the person 115. Information can be stored in a data storage device 140. A communication device 145 may be provided for communication with an external entity 150.

[0034] Person 115 can make a verbal utterance in the motor vehicle 105, which can be evaluated by the device 110. In particular, the utterance can first be converted into text form. In response to the utterance, a function of the motor vehicle 105 can be controlled.

[0035] In one embodiment, person 115 can give an instruction that can be interpreted in the context of the vehicle 105 or its driving state. The recognized or parameterized instruction can then be executed automatically. In another embodiment, an answer to a question from person 115 can also be provided. For example, person 115 could say: "Hello BMW, please tell me if I need windshield washer fluid." System 100 could then a) read the current windshield washer fluid level, b) retrieve stored BMW information such as an owner's manual, and c) request the weather forecast for the next few days from an external service. Based on this information, it could conclude that heavy snowfall is expected in the coming days and that the existing windshield washer fluid will not be sufficient for several days, and reply: "Yes, it will snow a lot in the next few days, please top it up."

[0036] Fig. Figure 2 shows a flowchart of a procedure 200 for controlling a motor vehicle 105. The system 100 is shown from a functional point of view with more details.

[0037] On board the vehicle 105 and on the part of the external entity 150, an instance of a data middleware 205 is provided, whereby the instances are synchronized bidirectionally with each other (where possible) in a process 210. The communication required for this preferably takes place wirelessly, for example via mobile network or WLAN.

[0038] A data middleware 205 can contain vehicle data 215, passenger data 220, and / or other data 225. A request list 230 is implemented as an action and response list and serves for communication between two external processes or services. Input text 235 can be provided by a user interface 238, which can be operated by a person 115 on board the vehicle 105. The user interface 238 can convert the input text from an audible format to a textual format. A mapping file 240 contains the information on how data from the data middleware 205 can be converted from a tree format, such as JSON, to a graph format, such as RDF / OWL, and vice versa.

[0039] An initial LLM 250, a second LLM 255, and a third LLM 260 are provided as examples by the external entity 115. It should be noted that two or more LLMs 250-260 can be integrated or identically configured. The first LLM 250 is configured to determine control instructions. The second LLM 255 is used for the static determination of SPARQL queries during operation. The third LLM 260 is configured for the dynamic determination of SPARQL queries during operation. Operation is defined as the operation of the vehicle 105 in the context described herein, i.e., when the person 115 is on board and can make a verbal utterance containing a control instruction. The third LLM 260 can provide a dynamic SPARQL query 265 to the first LLM 250.

[0040] An initial ontology 270 and an initial SPARQL query 275 can be stored in the data middleware 205. A manifest 278 can be specified for the initial LLM 250, which may contain, for example, an entry for static SPARQL queries, an entry for dynamic generic SPARQL queries, an entry for conditions under which the third LLM 260 should be queried, and / or an entry for required data points. An expert 280, such as a domain expert or a knowledge engineer, can provide entries in the manifest 278.

[0041] In the illustrated embodiment, a Knowledge Graph 282 is provided by the motor vehicle 105 and can include a JSON-to-RDF converter 284. The converter 284 can be configured to format content to JSON and may include an RDF-to-JSON converter.

[0042] For controlling the motor vehicle 105, an execution engine 286 is provided, which can communicate with the vehicle's on-board network 290 via a vehicle API gateway 288 (see interface 125). One or more control units can be connected to the on-board network 290, which can implement a control instruction and control a function of the motor vehicle 105. A function call manager 292 can implement a control instruction in the corresponding control of the motor vehicle 105.

[0043] Procedure 200 will now be explained using an illustrative example. Exemplary steps of Procedure 200 are listed in Fig. 1 shown in circles.

[0044] Step 1: Data from vehicle sensors, actuators or interfaces are converted via the vehicle API gateway 288 into a standardized vehicle data model such as VSS and written to the data middleware 205, so that it always contains the latest state of the vehicle on-board network data.

[0045] Step 2: Personal data 215 of a person 115 on board the motor vehicle 105 as well as other data 225 are written to the data middleware 205, either from sensors or applications in the motor vehicle 105 (e.g. camera recognition) or from the cloud or the external entity 115 (e.g. customer data such as age, name)

[0046] Step 3: Mapping file 240 (here SHACL) contains the information on how data middleware data in a tree format (here JSON) can be converted into a graph format (here RDF / OWL) and vice versa. For this purpose, SHACL maps an ontological concept (class or attribute) for each data point using sh:targetClass and sh:path.

[0047] Step 4: Using the mapping file 240, vehicle, passenger, and other data 215-225 are converted into graph format and materialized within the Knowledge Graph 282 as a triplet (here RDF / OWL). This preferably occurs in real time after a data change has taken place in the Data Middleware 205. In our case, this is done by the bidirectional JSON-to-RDF converter. It is connected to the Data Middleware 205, for example, via WebSocket.

[0048] Step 5: Representations of vehicle, person, and other semantic concepts and data are then available in a graph format, for example, A-Box and T-Box. The A-Box definitions are taken from the SHACL file 240, and the T-Boxes are the materialized facts of the Knowledge Graph 282.

[0049] Step 6: The following information is provided to the first LLM 250: a) Vehicle functions that can be executed via API gateway 288, such as changeSeatHeating(temperature, seatnumber). This is done, for example, by adding a function API description in natural language, as described for OpenAI function calls. Further entries in manifest 278 are described in later steps.

[0050] Step 7a = Stage 1: The domain expert 280 wants to make the existing vehicle 105 more intelligent and therefore provides a more intelligent function that can be called by the first LLM 250 in a later step, e.g., "Query the age of all passengers in the car, including their seat number." Together with a knowledge engineer 280, they create a SPARQL 1 query in text format, plus the necessary T-box concepts (if not already present in the knowledge graph 282), which are already represented in SHACL (otherwise, the new RDF triples cannot be created based on the data middleware data). The SPARQL 1 query is written in text format to the data middleware 205 to be synchronized with the vehicle 205. Finally, a manifest entry must be created for the first LLM 250 to inform it about the new SPARQL 1 function.

[0051] Step 7b = Stage 2a: Instead of the Knowledge Engineer 280 and the Domain Expert 280 manually creating SPARQL queries and A-Box concepts, the second LLM 255 is now also tasked with this. The second LLM 255 is an expert in creating SPARQL queries and A-Box concepts in general (due to the previous request to fine-tune general examples) and is aware of available concepts and materialized facts in the vehicle knowledge graph by synchronizing them from the vehicle 105 to the cloud 150 via the Data Middleware 205 and also providing them to the second LLM 255. The rest is the same as in Step 7a (which corresponds to Stage 1).

[0052] Step 7ci = Stage 2b: While the only difference between Stage 1 and Stage 2a is that the second LLM 255 takes over the manual (static) work of the Domain Expert 280 and the Knowledge Engineer 280 (the rest of the process is the same), Stage 2b operates somewhat differently and more generically. Equivalently, the third LLM 260 (like the second LLM 255) is an expert in creating SPARQL queries and A-Box concepts in general (through the previous request to fine-tune general examples) and is aware of available concepts and materialized facts in the vehicle knowledge graph by synchronizing them from the vehicle 105 to the cloud 150 via the Data Middleware 205 and also making them available to the third LLM 260. The difference: the third LLM 260 can be called by the first LLM 255 during the runtime of the vehicle 105.

[0053] Step 7cj: The first LLM 250 needs to know when to call the third LLM 160 to create a dynamic SPARQL query. This can be set by adding a manifest entry for the LLM3 API to the LLM1 manifest 278.

[0054] Step 7ck: Furthermore, the first LLM 150 needs to know where to send the dynamically created query (created in a later step 10b). Therefore, a generic SPARQL endpoint of the Knowledge Graph 282 is added to the manifest 278.

[0055] Step 8: Stage 1 and Stage 2a: The SPARQL 1 query is called by Knowledge Graph 282, and a new endpoint is offered, e.g., via REST. If new T-box concepts are needed for SPARQL query 1, these are now created in the same way as in Step 4. The SPARQL 1 endpoint is now ready to be called. Important: This step is not yet the execution of the query; it only makes Knowledge Graph 282 aware and capable of adding new and callable static SPARQL functions (e.g., during the startup phase). This step does not exist for Stage 2b because it is not needed, as the SPARQL payload is dynamically generated after the natural language input from Person 115, as described in the following steps.

[0056] Step 9: Now an example (use case) begins. Person 115 gives a verbally formulated control instruction to their vehicle 105, saying: "Hey BMW, please activate the seat heating for my child." Initially, it is unclear which seat the child is sitting in, so the corresponding seat heating cannot be specified. Furthermore, the term "my" needs clarification, and the term "child" may also be open to interpretation.

[0057] The user interface 238 converts the utterance into text, which is synchronized with the cloud 150 via the data middleware 205 and forwarded there to the first LLM 250.

[0058] Step 10a: Stage 1 and Stage 2a: Based on its general knowledge and the input manifest 278, the first LLM 250 knows that it must first use a SPARQL1 query to determine the baby's location and then call the function changeSeatHeating (temperature, seat number) at the vehicle API gateway 288. To achieve this, in this step, the first LLM 250 responds to the SPARQL1 query with the first function call (according to the predefined structure in the manifest file 278). This response is written by code to an LLM1 Action & Response List 230, which is a readable / writable / subscribeable object in the Data Middleware 205, like the other artifacts in the Data Middleware 205.

[0059] Step 10b: Stage 2b: LLM1 knows, based on its general knowledge and input manifest 278, that it must first call the third LLM 260 to request a suitable dynamically constructed SPARQL query that will find out where the baby is.

[0060] In a well-functioning third LLM 260, the dynamically created query is the same as one that would be manually created by an expert 280 in phase 3.1 or by the second LLM 255 in phase 3.2a. The dynamically created SPARQL query is sent back to the first LLM 250, which receives it and, according to manifest 278, creates a new function call in the action & response list 230 of the first LLM 250 to call the generic SPARQL endpoint in the knowledge layer.

[0061] Step 11a: Stage 1 and Stage 2a: Since the Knowledge Graph 282 subscribes to the Action & Response List 230 of the first LLM 250, it is triggered, knows that it needs to call the SPARQL1 query, and then sends the answer back to the first LLM 250 by writing the answer to the Action & Response List 230.

[0062] Step 11b: Stage 2b: When the Knowledge Graph 282 subscribes to the Action & Response List 230, it is triggered, knows that it must call the generic SPARQL endpoint with the SPARQL query payload previously generated by the third LLM 260 (one can imagine a function argument payload), and then sends the response back to the first LLM 250 by writing the response to the LLM1 Action & Response List 230.

[0063] Step 12: The first LLM 250 takes the response from call 1 and calls function 2 by responding again based on the structured function description from manifest 278 for the function changeSeatHeating().

[0064] Step 13: The function call manager 292 also subscribes to the LLM1 action and response list 230. It is therefore triggered and calls the function changeSeatHeating().

[0065] Step 14: After receiving confirmation from the vehicle network 290 that activation was successful, the Function Call Manager 292 optionally sends a short informational text to the user interface 238, which is converted into a speech output, e.g., "The seat heating is now activated for your child." This confirmation text can also be generated by the first LLM 250 (for the sake of clarity in the architecture diagram, it is not shown here). This completes the selected use case. Reference sign 100 System 105 motor vehicles 110 Device 115 people 120 processing unit 125 interface 130 microphone 135 speakers 140 data storage devices 145 Communication device 150 external positions 200 procedures 205 Data Middleware 210 Synchronization 215 Vehicle data 220 passenger data 225 more data 230 requirements list 235 Input text 238 User interface 240 Mapping file 250 LLM 1 for determining tax instructions 255 LLM 2 for static determination of SPARQL queries during runtime 260 LLM 3 for dynamic determination of SPARQL queries during runtime 265 dynamic SQL query 270 Ontology 1 275 SPARQL query 278 manifestos 280 Expert 282 Knowledge Graph 284 JSON-to-RDF Converters 286 Execution Engine 288 Vehicle API Gateway 290 Vehicle electrical system 292 Function call Manager

Images