Method and system for controlling a motor vehicle, and motor vehicle

Abstract

The invention relates to a method (200) for controlling a motor vehicle (105), comprising the following steps: detecting (9) a speech expression of a person (115) onboard the motor vehicle (105); the expression comprises a control command relating to an aspect of a vehicle state of the motor vehicle (105); information relating to a vehicle state of the motor vehicle (105) is stored in the data middleware (205); determining, by means of a large language model (250), a database query based on the expression, the database query for determining the aspect in a graph-based query language; determining an aspect of the vehicle state based on the database query by means of the data middleware (205); and executing (13) the control command taking into account the aspects of the vehicle state. The invention also relates to a system for controlling a motor vehicle and to a motor vehicle.

Description

Technical Field

[0001] This invention relates to an in-vehicle information system for motor vehicles. In particular, it relates to the control of an in-vehicle voice-activated intelligent assistant. Specifically, this invention relates to a method for controlling a motor vehicle, a system for controlling a motor vehicle, and a motor vehicle. Background Technology

[0002] The functions of a motor vehicle can be controlled via voice commands. To do this, the speech of the occupants can be detected and analyzed, and functions associated with the recognized voice commands can be triggered accordingly.

[0003] The meaning of an expression can depend on the context in which it is delivered. For example, the voice command "Please open my car window" could relate to the window of the vehicle closest to the speaker. Typical speech processing cannot determine such a context because it lacks the necessary information.

[0004] Speech processing can be performed using large language models (LLMs), which already possess a certain level of world knowledge. However, information specific to vehicles or speakers is not available. Furthermore, training an LLM is extremely complex, making it difficult to adapt the trained speech processing to new functionalities. Summary of the Invention

[0005] The objective of this invention is to provide an improved technique for controlling motor vehicles based on voice control commands. This invention addresses this objective through the subject matter of the independent claims. The dependent claims describe preferred embodiments.

[0006] A method for controlling a motor vehicle includes the following steps: detecting a voice expression of a person in the motor vehicle; wherein the expression includes control commands relating to aspects of the motor vehicle's driving state; storing information relating to the motor vehicle's driving state in a data middleware; determining a database query based on the expression using a large language model, the database query being used to determine the aspects using a graph-based query language; determining aspects of the driving state based on the database query using the data middleware; and executing control commands while taking into account aspects of the driving state.

[0007] Data middleware can receive and store information indicating the driving status of a motor vehicle from multiple sources. The driving status can be dynamically determined at any time based on the stored information. In particular, aspects of interest in the driving status can be determined by the engine correlating relevant information together. Exemplary aspects of the driving status may include, for example, the vehicle's speed, geographical location, or seat occupancy. The complete driving status can be determined by summing all available information. As will be explained in more detail below, data middleware can also be constructed in a distributed manner, with a portion of the middleware running on devices external to the motor vehicle.

[0008] Large language models can analyze human utterances and identify control commands. Furthermore, they can determine which aspects of the driving state are necessary for the selection or parameterization of control commands and provide matching queries.

[0009] Graph-based query languages ​​can retrieve content from predefined description systems, such as Resource Description Framework (RDF). This allows for the construction of logical statements about any thing within a database. Query languages ​​can, in particular, include SPARQL (a combination of the SPARQL protocol and RDF query language). This enables the understanding of aspects of the driving state through matching queries.

[0010] It advantageously supports a wider range of control commands. Personnel can provide control commands in a more natural language. It can recognize and parse cues about the scene or driving status embedded in control commands. It can better determine the driver's intent and better, faster, and more accurately recognize control commands expressed in speech. It can execute the indicated commands directly and precisely.

[0011] In a preferred embodiment, database queries are written to a request list of the data middleware; and responses to the database queries are written to the request list. This list can be subscribed to by a database engine that executes database queries based on data from the data middleware. The artifacts of the data middleware can be read, written, and / or subscribed to, enabling any entity to communicate via objects, such as via lists. This allows for asynchronous exchange, in which the sender and receiver are dynamically and efficiently connected to each other.

[0012] Similarly, control commands can be written into the request list of the data middleware; and the current driving state can be adapted to the control commands. To execute the control commands in the request list, corresponding agents can be set up. Tasks such as rationality checks or safety checks can be performed on the agent side.

[0013] Database queries can be determined based on a manifest. The manifest describes which database queries are possible or how specific aspects of the vehicle's status can be determined, and which functions should be invoked to obtain specific information. The manifest is preferably in text form, allowing it to be processed by a large language model. This also simplifies manifest management and allows for human verification of consistency when needed. The manifest can be easily modified, expanded, or replaced. This facilitates the addition or improvement of new functions or control commands. Outdated or proven unsafe control commands can be easily removed.

[0014] More preferably, the list includes a predetermined set of database queries. These database queries can be associated with aspects of the vehicle's condition that can be determined using these queries. This set can be established or maintained by one or more experts. For example, a domain expert can contribute knowledge about motor vehicles and their characteristics. A knowledge engineer can provide knowledge about information processing and determining facts based on information stored in the database. The experts can collaboratively provide valuable database queries that can be used generally and efficiently to determine various aspects of the vehicle's condition.

[0015] In another implementation, the list includes a generic endpoint, but this generic endpoint returns different responses based on the different SPARQL payloads from the large language model.

[0016] In another implementation, the database query is determined dynamically. For this purpose, the query can be constructed based on aspects of interest at the moment a response is expected. More preferably, the database query is determined using a large language model. This can be achieved using the large language model mentioned above or another large language model independent of it. This large language model can be trained to determine the database query using a predetermined graph-based query language, particularly SPARQL.

[0017] According to another aspect of the present invention, a system for controlling a motor vehicle includes: an input device for detecting a voice expression of a person in the motor vehicle; wherein the expression includes control instructions relating to aspects of the driving state of the motor vehicle; a data middleware storing information relating to the driving state of the motor vehicle; a large language model (LLM) configured to determine a database query based on the expression, the database query being used to determine the aspects in a graph-based query language; and a query engine for the data middleware, wherein the query engine is configured to determine aspects of the driving state based on the database query; and an execution device for executing the control instructions taking into account aspects of the driving state.

[0018] The system is preferably configured to perform the methods described herein, in part or in whole. For this purpose, the system may include a processing device, preferably electronically constructed and, for example, an integrated circuit, a programmable logic element, or a programmable microcomputer. The method may be implemented in a configured form or as a computer program product having program code modules for the processing device. This configuration or computer program product may be stored on a computer-readable data carrier. Features or advantages of the method may be transferred to the device, and vice versa.

[0019] Large language models can be constructed multimodally and process acoustic data collected directly from speakers. Alternatively, a speech-to-text conversion can be configured to convert the acoustic data into text data, which can then be fed into a text-based large language model.

[0020] Data middleware can be implemented across multiple processing systems. Information can be periodically coordinated and synchronized among these systems. One processing system may be located on-board in a vehicle, while another may be mounted externally. Yet another processing system can be mobile, operating, for example, on a mobile device. Therefore, it is not crucial where the mobile processing system is located. Information exchange can occur at varying speeds, depending on the quality of the communication connection. If communication fails, coordination can be interrupted.

[0021] Each processing system can have the same interface to process data from the data middleware. In particular, objects in the data middleware can be accessed in a unified manner. It is not important which processing system the read or write service resides on.

[0022] Thus, a user can provide voice-based control commands on their mobile device, which performs speech-to-text conversion and writes the results to an object in the data middleware. The large language model can be located, for example, remotely from the vehicle, such as in a computing center. The large language model can extract text control commands from this object and write one or more database queries to another object in the data middleware. The database engine can execute these queries and write the results back to this object. The vehicle's onboard agent can collect the results and use them to parameterize the control functions to achieve the desired effect. Here, object synchronization between the various processing systems is transparently achieved by the data middleware.

[0023] The query engine can implement a knowledge graph (KG). The knowledge graph may include a stream reasoner that performs ontology-based reasoning. In another embodiment of the invention, rules can also be set attached to the ontology. In this case, stream reasoning can also be implemented based on rules. The knowledge graph further preferably includes a query engine for a graph query language (e.g., SPARQL). The query engine allows new SPARQL queries to be added statically and at runtime, and dynamically provides API endpoints (e.g., REST APIs) for each specific SPARQL query and general SPARQL endpoints. The knowledge graph may also include code logic for managing knowledge graph tasks, such as SHACL transformations, SPARQL API calls, etc.

[0024] The system can also fully include a large language model (LLM), which is configured to dynamically determine database queries.

[0025] According to yet another aspect of the invention, a motor vehicle includes a device that implements all or part of the system described herein. It should be noted that the device need not include all the components of the system. However, the agent executing control commands should preferably operate on-board of the motor vehicle. The speaker is also located within the motor vehicle, typically even inside the vehicle, so the element for recording the system's expression is preferably also mounted within the motor vehicle. In another preferred embodiment, this element is fixedly mounted on the motor vehicle and may, for example, include a microphone from inside the motor vehicle. Attached Figure Description

[0026] The invention will now be described in more detail with reference to the accompanying drawings, in which:

[0027] Figure 1 This illustrates a system; and

[0028] Figure 2 A flowchart of one method is shown. Detailed Implementation

[0029] Figure 1 A system 100 is shown. A device 110 is located on-board at a motor vehicle 105 and is configured to control the functions of the motor vehicle 105 or its included systems or subsystems. It should be noted that these functions may also include inquiries or information provision, which may be determined based on available information in the context of the motor vehicle 105.

[0030] Device 110 preferably includes a processing unit 120, which can be connected to an interface 125 via which predetermined functions of the vehicle 105 or one of its systems can be controlled. A microphone 130 may be provided for detecting the voice expressions of the person 115. A speaker 135 is further optionally shown for providing acoustic feedback to the person 115. Information may be stored in a data memory 140. A communication device 145 may be provided for communicating with an external station 150.

[0031] Personnel 115 can provide a voice expression within vehicle 105, which can be evaluated by device 110. Specifically, this expression can first be converted into text form. In response to this expression, the functions of vehicle 105 can be controlled.

[0032] In one implementation, person 115 can issue an instruction that can be interpreted within the context of vehicle 105 or its driving state. The identified or parameterized instruction can then be executed automatically. In another implementation, a response to person 115's question can also be provided. For example, person 115 might say, "Hello BMW, please tell me if I need windshield washer fluid." System 100 can then: a) read the current windshield washer fluid level, b) access BMW's internal stored knowledge, such as the owner's manual, and c) query the weather forecast for the next few days from an external service. Based on this information, it can conclude that heavy snowfall is expected in the next few days and that the existing windshield washer fluid is insufficient for several days, and reply, "Yes, there will be heavy snowfall in the next few days; it's best to replenish the windshield washer fluid."

[0033] Figure 2 A flowchart of a method 200 for controlling a motor vehicle 105 is shown. Here, the system 100 is shown in more detail from a functional perspective.

[0034] Instances of data middleware 205 are provided on the vehicle 105 and at the external station 150, and these instances coordinate with each other bidirectionally in process 210 whenever possible. The communication required for this is preferably wireless, for example via mobile radio or wireless local area network.

[0035] Vehicle data 215, passenger data 220, and / or other data 225 can be found in data middleware 205. Request list 230 is structured as a list of actions and responses and is used for communication between two external processes or services. Input text 235 can be provided by user interface 238, which can be operated by the occupants 115 of vehicle 105. User interface 238 can convert the input text from acoustic form to text form. Mapping file 240 contains information on how data from data middleware 205 can be converted from a tree format (e.g., JSON) to a graphical format (e.g., RDF / OWL) and vice versa.

[0036] For example, a first large language model 250, a second large language model 255, and a third large language model 260 are provided on the external site 115 side. It should be noted that two or more large language models 250-260 can also be integrated with each other or can be constructed identically. The first large language model 250 is configured to determine control commands. The second large language model 225 is used to statically determine SPARQL queries during runtime. The third large language model 260 is configured to dynamically determine SPARQL queries during runtime. Runtime is understood as the operation of the vehicle 105 in the scenario described herein, i.e., the operation when the person 115 is inside the vehicle and can give a voice expression with control commands. The third large language model 260 can provide dynamic SPARQL queries 265 to the first large language model 250.

[0037] The first ontology 270 and the first SPARQL query 275 can be stored in the data middleware 205. A list 278 can be set up for the first major language model 250, which may, for example, contain entries for static SPARQL queries, entries for dynamic general SPARQL queries, entries for conditions (under which a third major language model 260 should be queried), and / or entries for required data points. An expert 280, such as a domain expert or knowledge engineer, can provide entries to list 278.

[0038] In the illustrated embodiment, a knowledge graph 282 is provided on the vehicle 105 side, and the knowledge graph may include a JSON to RDF converter 284. The converter 284 may be configured to format content according to JSON and includes an RDF to JSON converter.

[0039] To control the vehicle 105, an actuator 286 is provided, which can communicate with the vehicle's onboard electrical network 290 via a vehicle API gateway 288 (see interface 125). One or more control units can be connected to the onboard electrical network 290, and these control units can implement control commands and control the functions of the vehicle 105. A function call manager 292 can translate control commands into corresponding controls for the vehicle 105.

[0040] Method 200 will now be illustrated with explanatory examples. The exemplary steps mentioned in method 200 are as follows: Figure 1 It is indicated by a circle.

[0041] Step 1: Convert the data from vehicle sensors, actuators, or interfaces into a standardized vehicle data model (such as VSS) via the vehicle API gateway 288 and write it into the data middleware 205, so that the data middleware always contains the latest status of the on-board electrical network data.

[0042] Step 2: Write the personal data 215 and other data 225 of the occupants of the vehicle 105 into the data middleware 205. This data comes either from sensors or applications in the vehicle 105 (e.g., camera recognition) or from the cloud or external site 115 (e.g., customer data such as age and name).

[0043] Step 3: Mapping file 240 (here, SHACL) contains information on how the data middleware's data can be converted from a tree format (here, JSON) to a graphical format (here, RDF / OWL) and vice versa. To do this, SHACL maps an ontology concept (class or attribute) for each data point using sh:targetClass and sh:path.

[0044] Step 4: Using mapping file 240, vehicle data, passenger data, and other data 215-225 are converted into a graph format and materialized as triples (here, RDF / OWL) within knowledge graph 282. This is preferably performed in real-time after data changes are implemented in data middleware 205. In this case, this occurs via a bidirectional JSON-to-RDF converter. This converter connects to data middleware 205, for example, via WebSocket.

[0045] Step 5: The representations of vehicle, personnel, and other semantic concepts and data are then presented in graphical formats, such as A-Boxes and T-Boxes. The A-Box definition is provided by SHACL file 240, while the T-Box is a concrete fact of knowledge graph 282.

[0046] Step 6: Provide the following information to the first large language model 250:

[0047] a) Vehicle functions that can be executed via API gateway 288, such as changeSeatHeating(temperature, seatnumber). This is achieved, for example, by adding a function API description in natural language, as described for OpenAI function calls. Other entries in Listing 278 will be described in subsequent steps.

[0048] Step 7a = Phase 1: Domain expert 280 wants to make the existing vehicle 105 smarter and therefore provides smarter functions that can be called by the first large language model 250 in subsequent steps, such as "query the ages of all passengers sitting in the vehicle along with their seat numbers". The domain expert, together with knowledge engineer 280, creates a text-formatted SPARQL1 query plus the necessary T-Box concepts (if not already present in knowledge graph 282), which are already represented in SHACL (otherwise, new RDF triples cannot be created based on the data middleware data). The SPARQL1 query is written to data middleware 205 in text format for synchronization with vehicle 205. Finally, a list entry must be registered for the first large language model 250 to inform it of the new SPARQL1 function.

[0049] Step 7b = Stage 2a: Instead of manually creating SPARQL queries and A-Box concepts through Knowledge Engineer 280 and Domain Expert 280, the creation is now still delegated to the second language model 255. The second language model 255 is an expert in the general creation of SPARQL queries and A-Box concepts (through requests previously made to fine-tune general examples) and is aware of the available concepts and specific facts in the vehicle knowledge graph. This is achieved by the vehicle knowledge graph synchronizing these available concepts and specific facts from the vehicle 105 to the cloud 150 via data middleware 205 and also providing these available concepts and specific facts to the second language model 255. The rest is the same as step 7a (corresponding to Stage 1).

[0050] Step 7ci = Phase 2b: The only difference between Phase 1 and Phase 2a is that the second major language model 255 performs the manual (static) work of the domain expert 280 and knowledge engineer 280 (the rest of the process is the same), while Phase 2b works differently and more generally. Similarities: The third major language model 260 (like the second major language model 255) is an expert for generally creating SPARQL queries and A-Box concepts (through requests previously used for fine-tuning general examples) and is aware of the available concepts and specific facts in the vehicle knowledge graph, which synchronizes these available concepts and specific facts from vehicle 105 to cloud 150 via data middleware 205 and also provides these available concepts and specific facts to the third major language model 260. Differences: The third major language model 260 can be invoked by the first major language model 250 during the runtime of vehicle 105.

[0051] Step 7cj: The first major language model 250 must know when the third major language model 160 should be invoked in order to create dynamic SPARQL queries. This can be determined by adding an LLM3-API manifest entry to LLM1-Manifest 278.

[0052] Step 7ck: Furthermore, the first large language model 150 must know where the dynamically created queries (in subsequent step 10b) should be sent. For this reason, the generic SPARQL endpoint of the knowledge graph 282 is also added to Listing 278 here.

[0053] Step 8: Phases 1 and 2a: The SPARQL1 query is invoked by Knowledge Graph 282, and a new endpoint is provided, for example, via REST. If new T-Box concepts are needed for SPARQL query 1, these concepts are now created in the same manner as in Step 4. The SPARQL1 endpoint is now ready to be invoked. Importantly: this step is not yet the execution of the query; it only makes Knowledge Graph 282 aware of and able to add new, callable static SPARQL functions (e.g., during the startup phase). This step is not required for Phase 2b because the SPARQL payload (Nutzdaten) is dynamically created and described according to the natural language input of Person 115, as described in the following steps.

[0054] Step 9: Now let's begin with an example (use case). Person 115 gives a voice-constructed control command to their vehicle 105: "Hello BMW, please activate the seat heating for my child." First, it's unclear which seat the child is sitting in, thus making it impossible to describe the associated seat heating. Furthermore, the description "my" needs clarification, and the description "child" may also require explanation.

[0055] User interface 238 converts the expression into text, which is synchronized with cloud 150 via data middleware 205 and forwarded to the first large language model 250 in the cloud.

[0056] Step 10a: Phase 1 and Phase 2a: The first large language model 250, based on its common sense and input list 278, knows that it must first use a SPARQL1 query to determine where the child is sitting and then call the changeSeatHeating(temperature, seatnumber) function at the vehicle API gateway 288. To achieve this, the first large language model 250 responds to the SPARQL1 query in this step with the first function call (according to the predefined structure in list file 278), which is written into the LLM1 action & response list 230 by code. This list constitutes a readable / writable / subscribed object in the data middleware 205, like other artifacts in the data middleware 205.

[0057] Step 10b: Stage 2b: Based on its common sense and based on input list 278, LLM1 knows that it must first call the third language model 260 in order to request a suitable dynamically created SPARQL query that finds out where the child is sitting.

[0058] In the well-functioning third language model 260, the dynamically created query is identical to the query manually created by expert 280 in phase 3.1 or by the second language model 255 in phase 3.2a. The dynamically created SPARQL query is sent back to the first language model 250, which receives the query and creates a new function call in the action & response list 230 of the first language model 250 according to list 278, so that the generic SPARQL endpoint in the knowledge layer can now be invoked.

[0059] Step 11a: Phase 1 and Phase 2a: Because the knowledge graph 282 subscribes to the action & response list 230 of the first large language model 250, the knowledge graph is triggered to know that it must call the SPARQL1 query and then send the response back to the first large language model 250 by writing the response into the action & response list 230.

[0060] Step 11b: Phase 2b: When the knowledge graph 282 subscribes to the action & response list 230, the knowledge graph is triggered, knowing that it must use the SPARQL query payload previously generated by the third large language model 260 to call the general SPARQL endpoint (which can be envisioned as a function parameter payload), and then send the response back to the first large language model 250 by writing the response into the LLM1 action & response list 230.

[0061] Step 12: The first large language model 250 receives the response to call 1 and calls function 2 in such a way that it replies based on the structured functional description in Listing 278 used for the changeSeatHeating() function.

[0062] Step 13: Function call manager 292 also subscribes to LLM1 actions and response list 230. Therefore, function call manager is triggered and calls the changeSeatHeating() function.

[0063] Step 14: After receiving a "confirmation" of successful activation from the vehicle electrical network 290, the function call manager 292 optionally sends a short message text to the user interface 238, which is converted into voice output, such as "The seat heating for your child is now activated." This confirmation text can also be created by the first large language model 250 (not shown here for clarity of the architecture). This concludes the selected use case.

[0064] List of reference numerals

[0065] 100 System

[0066] 105 Motor vehicles

[0067] 110 equipment

[0068] 115 personnel

[0069] 120 processing device

[0070] 125 interface

[0071] 130 microphones

[0072] 135 speakers

[0073] 140 Data Storage

[0074] 145 Communication devices

[0075] 150 external sites

[0076] 200 methods

[0077] 205 Data Middleware

[0078] 210 Synchronization

[0079] 215 Vehicle Data

[0080] 220 crew data

[0081] 225 Other Data

[0082] 230 Request List

[0083] 235 Output text

[0084] 238 User Interface

[0085] 240 Mapped Text

[0086] 250 LLM1 used to determine control commands

[0087] 255 is used to statically determine SPARQL queries during runtime.

[0088] 260 is used to dynamically determine SPARQL queries during runtime.

[0089] 265 Dynamic SQL Queries

[0090] 270 body 1

[0091] 275SPARQL query

[0092] 278 List

[0093] 280 experts

[0094] 282 Knowledge Graph

[0095] 284 JSON to RDF converter

[0096] 286 Execution Engine

[0097] 288 Vehicle API Gateway

[0098] 290 Vehicle-mounted electrical grid

[0099] 292 Function Call Manager

Claims

1. A method (200) for controlling a motor vehicle (105), the method comprising the following steps: -Detect the speech expression of (9) motor vehicle (105) and vehicle occupants (115); - The expression includes control commands that relate to aspects of the vehicle state of the motor vehicle (105); - Information relating to the vehicle status of the motor vehicle (105) is stored in the data middleware (205); - Using a large language model (250), a database query (265) is determined based on the expression, the database query being used to determine the aspect in a graph-based query language; - The aspect of determining the vehicle status based on the database query using the data middleware (205); and - The control command (13) is executed taking into account the vehicle's state.

2. The method (200) according to claim 1, wherein, The database query is written into the request list of the data middleware (205); and the response to the database query is written into the request list.

3. The method (200) according to claim 1 or 2, wherein, The control command is written into the request list of the data middleware (205); and the current vehicle state is adapted to the control command.

4. The method (200) according to any one of the preceding claims, wherein, The database query is determined based on list (278).

5. The method (200) according to claim 4, wherein, The list (278) includes a set of pre-defined database queries.

6. The method (200) according to any one of claims 1 to 3, wherein, The database query is determined dynamically.

7. The method (200) according to claim 6, wherein, The database query is determined using a large language model (260).

8. A system (100) for controlling a motor vehicle (105), wherein, The system includes the following components: - Input device (130) for detecting the voice expression of the occupants (115) of the motor vehicle (105); - The expression includes control commands that relate to aspects of the vehicle state of the motor vehicle (105); - Data middleware (205), in which information relating to the vehicle status of the motor vehicle (105) is stored; - A large language model (250), the large language model being configured to determine a database query based on the expression, the database query being configured to determine the aspect in a graph-based query language; - A query engine (282) for the data middleware (205), wherein the query engine (282) is configured to determine aspects of the vehicle status based on the database query; as well as - An actuator (292) is used to execute the control commands taking into account aspects of the vehicle's state.

9. The system (100) according to claim 8, wherein, The data middleware (205) is implemented on multiple processing systems (110, 150); and information is periodically coordinated and consistent among the processing systems (110, 150).

10. The system (100) according to claim 8 or 9, wherein, The query engine implements a knowledge graph (282).

11. The system (100) according to any one of claims 9 to 10, the system further comprising a large language model (260) configured to dynamically determine the database query.

12. A motor vehicle (105), including equipment (110) that implements at least a portion of the system according to any one of claims 8 to 11.

Images