Method for training a machine learning model for a voice assistant and device for controlling a vehicle function
By employing context-free grammars and large language models, the method efficiently trains voice assistants for vehicles, enhancing model accuracy and robustness in recognizing vehicle functions and distinguishing user input from noise.
Patent Information
- Authority / Receiving Office
- DE · DE
- Patent Type
- Applications
- Current Assignee / Owner
- BAYERISCHE MOTOREN WERKE AG
- Filing Date
- 2024-12-06
- Publication Date
- 2026-06-11
AI Technical Summary
Existing methods for training voice assistants for vehicles are time-consuming and require significant effort, and the quality of the trained models is often inadequate for accurately processing natural language commands.
A method involving context-free grammars to define sentences with variable function parameters, generating datasets, and splitting them into training and test sets to efficiently train a machine learning model for vehicle functions, using large language models and supervised learning to enhance model quality and accuracy.
The method enables rapid and effective training of a machine learning model for voice assistants in vehicles, reducing the risk of misidentification of function parameters and improving the model's ability to recognize diverse natural language expressions and distinguish user input from ambient noise.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[0001] The invention relates to a method for training a machine learning model for a voice assistant. The invention further relates to a device for controlling a vehicle function.
[0002] Voice assistants are capable of processing natural language and assisting users in interacting with electronic devices. The processing of natural language by the voice assistant is achieved, for example, through software. Methods for training voice assistants require training datasets. These can be generated from public sources, although such datasets often require time-consuming manual cleaning to remove unwanted content. Alternatively, training datasets can be generated synthetically, but the generation criteria significantly influence the quality of the language model trained with the dataset—that is, its ability to process speech input as desired.
[0003] Document CN 117574913 A discloses a method for training a language model. This involves extending an existing training dataset using ChatGPT.
[0004] Document US 2024 / 0095460 A1 discloses a method for training language models. In this method, a language model trained on a task is retrained with a training dataset that enables the language model to assist in the execution of a subsequent task.
[0005] The object of the invention is to provide a method for training a machine learning model for a voice assistant for a vehicle, which enables the training of the machine learning model with very little effort and at the same time achieves a particularly high model quality.
[0006] This problem is solved by a method with the features of claim 1. Advantageous embodiments are specified in the dependent claims.
[0007] In a first aspect of the invention, a method defines at least one context-free grammar that defines at least one sentence with at least one variable function parameter and assigns it to a vehicle function. The sentence is thus defined in a machine-readable, structured form in which the at least one function parameter and the assigned vehicle function are uniquely identifiable for a language model. The context-free grammar is thus associated with the vehicle function. Based on the context-free grammar, a data set containing sentences that adhere to the context-free grammar is then generated and provided using a language model. In each sentence, the at least one function parameter is varied within a parameter space defined by the context-free grammar.The dataset is split into a training dataset and a test dataset, with the training and test datasets differing in every set. The machine learning model is then trained using the training dataset to determine the vehicle function and at least one function parameter based on user input and to output a corresponding result. The machine learning model is then tested using the test dataset to determine its model quality.
[0008] The method with the features of claim 1 enables the particularly easy generation of a dataset for training and testing a machine learning model for a vehicle voice assistant. Furthermore, the machine learning model can be trained especially effectively because the sentences contained in the dataset, due to the underlying context-free grammar, are suitable for the desired vehicle function, as each sentence contains all the necessary function parameters for a given vehicle function. This allows the machine learning model to reliably identify various function parameters of vehicle functions within the parameter space defined by the context-free grammar. In this way, the risk of a word being incorrectly identified as a function parameter is reduced.
[0009] A vehicle function is a function of the vehicle that can be controlled, for example, by a processing unit within the vehicle. Examples of vehicle functions include making a phone call, setting a target temperature for an air conditioning system, turning on a radio, or opening a glove compartment.
[0010] In one embodiment, the language model is trained using a generic text corpus. This generic text corpus comprises texts that are not limited to a specific subject area. For example, the Toronto Book Corpus or a filtered version of Wikipedia can be used as the generic text corpus. Training with the generic text corpus gives the language model a general understanding of language and a broad knowledge base. This general knowledge is also referred to as world knowledge. This world knowledge enables the language model to assign different user inputs with the same semantic content to an embedding in the same neighborhood of the language model's embedding space, even if it was not specifically trained on those user inputs.
[0011] The function parameter is a variable parameter of a parameter space in a sentence defined by the context-free grammar. In one embodiment, a parameter space "temperature" could, for example, include the function parameters "twenty degrees" or "sixty-eight degrees". Furthermore, the sentence defined by the context-free grammar could also include a parameter space "temperature unit" with the function parameters "Celsius" or "Fahrenheit". The sentence defined by the context-free grammar could, for example, read: Set the air conditioning to <temperatur> <temperatureinheit>a.
[0012] The output of the machine learning model includes, for example, the name of the vehicle function or another unique indicator for the vehicle function. Furthermore, the output includes, for example, at least one function parameter.
[0013] Model quality, for example, is a numerical value that describes how accurately the machine learning model can assign a voice input to a corresponding vehicle function. This value can indicate how much an expected output deviates from the actual output of the machine learning model. Such a value can be determined, for instance, using a cost function of the machine learning model, which is minimized during training.
[0014] In one embodiment, several context-free grammars are defined, each defining at least one sentence with at least one variable function parameter and each assigned to a vehicle function. Thus, each context-free grammar is associated with a vehicle function. Based on the context-free grammars, a dataset of sentences is then generated and provided using a language model, with each sentence adhering to one of the context-free grammars. Furthermore, in each sentence, the at least one function parameter is varied within a parameter space defined by the context-free grammar. This generates a dataset with which the machine learning model can be trained to determine different vehicle functions with their respective function parameters for various sentences, such as user input, and to output a corresponding result.This allows the machine learning model to be trained with particularly little effort.
[0015] It is advantageous if, in at least one sentence, a word choice and / or word order from the context-free grammar is varied while maintaining the meaning of the context-free grammar. The machine learning model is then trained with the training dataset to continue assigning the sentences to the vehicle function associated with the context-free grammar. This makes the machine learning model particularly efficient at identifying various function parameters and assigning synonymous sentence formulations that deviate from the context-free grammar to the vehicle function associated with it. In this way, the machine learning model can assign a wide variety of synonymous expressions in natural language to a desired vehicle function with very high precision and accuracy.
[0016] It is advantageous if the context-free grammar is defined in Backus-Naur Form. This allows for a particularly simple and precise definition of the context-free grammar. It is further advantageous if the context-free grammar is defined in Extended Backus-Naur Form. This allows for a more compact representation of the context-free grammar and thus faster processing.
[0017] It is advantageous if the data set generated and provided by the language model is a text data set, and if both the training and test data sets contain text data. This makes training the machine learning model particularly easy.
[0018] It is also advantageous if the dataset is provided as an audio dataset and both the training and test datasets contain audio data. In one embodiment, the dataset can be generated directly as an audio dataset by the speech model. In another embodiment, the dataset is generated as a text dataset by the speech model and subsequently converted into an audio dataset using a text-to-speech model and provided. The audio dataset can, in particular, contain audio data of sentences spoken in a dialect other than a standard language or with a speech impediment. The text-to-speech model can be part of the speech model or executed separately.The machine learning model is trained to better recognize natural language, since the dataset in spoken form may, for example, lack punctuation data, or spoken words may sound unclear or similar to other words.
[0019] Furthermore, the machine learning model is trained to recognize sentences that deviate from standard pronunciation. This allows the trained machine learning model to identify functional parameters in spoken sentences with particularly low deviation and to assign spoken sentences to a vehicle function associated with the context-free grammar.
[0020] It is advantageous to split a further dataset containing audio with ambient noise into a training dataset and a test dataset. The machine learning model is then trained on the training dataset and the additional training dataset to determine whether an audio input is user input or ambient noise. The machine learning model is then further tested on the test dataset and the additional test dataset to determine the model's performance. This allows the trained machine learning model to distinguish particularly well between user voice input and ambient noise. In this way, the risk of ambient noise being incorrectly identified as user input is reduced.
[0021] Ambient noise can be, for example, a sound from the vehicle itself, such as engine noise. It can also be generated by other vehicle occupants, such as during a conversation. Furthermore, ambient noise can include technical noises, such as microphone hiss.
[0022] It is advantageous if the language model is a large language model, especially a generative pre-trained transformer (GPT). This allows for the use of a particularly large dataset for varying the function parameters, one that contains a large number of possible function parameters within the parameter space. Furthermore, many equivalent variations of a sentence can be generated with minimal effort. Thus, the generated training dataset is well-suited for training the machine learning model, as the dataset accurately reflects the diverse expressions of natural language.
[0023] It is advantageous to label the dataset and train the machine learning model using a supervised learning approach. For example, one element of the training dataset consists of the phrase "Open the driver's side window" with the function parameter "Driver's side window" and the corresponding vehicle function OPEN_MOVABLE, which triggers the opening of a movable part of the vehicle. The machine learning model generates an output for each phrase in the training dataset, containing a unique indicator of the vehicle function and at least one determined function parameter. The vehicle function serves as the label. This training enables the machine learning model to robustly associate variations in user input with the corresponding vehicle function, even if the phrase in the user input was not part of the training dataset.For example, the following sample sentences for the vehicle function OPEN_MOVABLE can be used as part of the training dataset: "Open the driver's side window" and "Open the driver's door." After training, variations of these sentences, such as "Open the tailgate," "Open the sunroof," and "Please open the glove compartment," will also have a high probability of being correctly assigned to the vehicle function OPEN_MOVABLE. This allows the machine learning model to be trained very effectively to assign sentences to a desired vehicle function with very high precision and accuracy.
[0024] A second aspect of the invention discloses a device for controlling a vehicle function. This device implements a vehicle voice assistant. It comprises a storage element on which the machine learning model for the voice assistant is stored and a receiver module configured to receive spoken user input from a vehicle occupant. In one embodiment, the spoken user input is captured by a microphone in the vehicle or the microphone of a mobile device and transmitted to the receiver module.
[0025] Furthermore, a processing module is included, which is configured to load and execute the machine learning model from the storage element. In one embodiment, the processing module is further configured to process the spoken user input into text data using a speech-to-text model. A control module is also included, which is configured to select and control a vehicle function based on an output from the machine learning model. The machine learning model is trained, using one of the embodiments of the first aspect of the invention already described, to determine the desired vehicle function based on the user input and to generate the output. The desired vehicle function can be determined, for example, if a sentence in the user input can be assigned to a vehicle function associated with the context-free grammar.The output then indicates which of the vehicle functions controllable by the control module corresponds to the desired vehicle function and which function parameters necessary for controlling the desired vehicle function are contained in the user input. This ensures that the machine learning model can assign a function parameter in a set to the correct function particularly well and that the correct, desired vehicle function can be activated with high reliability.
[0026] In one embodiment, the processing module is part of the vehicle. For example, the processing module is part of a processing unit within the vehicle, such as a central vehicle computer. Alternatively, the processing module can be implemented, at least partially, by a processing unit located remote from the vehicle, such as a server located remote from the vehicle, in a cloud computing environment, or by a mobile device connected to the vehicle. In this case, the machine learning model is executed on the server located remote from the vehicle or in the cloud computing environment. In such an embodiment, the machine learning model is preferably stored on a memory element of the processing unit located remote from the vehicle. This allows the use of a machine learning model that might not be able to run on the limited hardware of the vehicle, or only with very high latency.In another embodiment, the receiver module is either part of the vehicle's processing unit or located outside of it. In another embodiment, the control module is either part of the vehicle's processing unit or located outside of it.
[0027] In a third aspect of the invention, a vehicle is disclosed with the device for controlling a vehicle function according to a described embodiment. This allows the voice assistant to be provided locally, so that no internet connection is required for the operation of the voice assistant.
[0028] Examples of implementation are explained in more detail below with reference to the figures. These show: Fig. 1. A flowchart of a procedure for training a machine learning model for a voice assistant for a vehicle; and Fig. 2 a schematic representation of a device of a vehicle for controlling a vehicle function.
[0029] Fig. Figure 1 shows a flowchart of a procedure for training a machine learning model for a voice assistant for a vehicle. The procedure is started in step S100.
[0030] In step S102, at least one context-free grammar is defined, in which at least one sentence with at least one variable function parameter is defined and assigned to a vehicle function.
[0031] For example, the phrase "Call Peter Müller" is assigned the vehicle function FN_CALL_CONTACT, which triggers a call. A possible voice command for the vehicle function FN_CALL_CONTACT in Backus-Naur Form (BNF) syntax would then be: Calls<Com_Contact_FirstName><Com_Contact_LastName> [<Com_Phonetype> ] an | I would like<Com_Contact_FirstName><Com_Contact_LastName> call
[0032] Various syntactic symbols indicate an OR operation (|), an optional input ([ ]), or a function parameter (<,>), such as the contact name. Here,<Com_Contact_FirstName> for any function parameter that is a first name,<Com_Contact_LastName> represents any function parameter that is a last name and [<Com_Phonetype> The space `]` represents any function parameter that is a telephone type, e.g., "landline" or "mobile". The square brackets indicate that the function parameter is optional. The words "call" and "call" are fixed, at least for this BNF syntax. "Call Peter Müller" is a sentence that fulfills the first alternative of the grammar given above. "I want to call Peter Müller" is a sentence with the same semantic meaning according to the second alternative.The function parameters<Com_Contact_FirstName> and<Com_Contact_LastName> are the same in both sentences, where the optional parameter [<Com_Phonetype> ] was not used in the second sentence.
[0033] Subsequently, in step S104, a dataset containing sentences that adhere to the context-free grammar is generated using a language model. Each sentence includes at least one function parameter of the vehicle function. The function parameter is varied in each sentence while respecting a parameter space defined by the context-free grammar. In one embodiment, the dataset can be generated and provided as text data. In another embodiment, the dataset can be generated as audio data. For example, the language model can generate the dataset directly as an audio dataset. In yet another embodiment, the language model can generate the dataset as a text dataset and then convert it into an audio dataset using a text-to-speech model.The text-to-speech model can be part of the language model or executed separately.
[0034] For example, using a large language model with world knowledge, a large variation of training data is generated from the existing BNF definition. The large language model is instructed to generate a multitude of variations of natural language input.
[0035] In one embodiment, only the function parameter is varied, and the BNF grammar is strictly adhered to. In this example, the language model incorporates world knowledge about personal names into the data set generation (Peter Müller, Johannes Meier, ...). Thus, the sentence "I would like" becomes<Com_Contact_FirstName><Com_Contact_LastName> call." For example, at least the following variations were generated: "I want to call Anna Schmidt." "parameters": { "Com_Contact_FirstName": "Anna", "Com_Contact_LastName": "Schmidt"} "I want to call Johannes Meier." "parameters": { "Com_Contact_FirstName": "Johannes", "Com_Contact_LastName": "Meier"} "I want to call Peter Müller." "parameters": { "Com_Contact_FirstName": "Peter", "Com_Contact_LastName": "Müller"} "I want to call Thomas Weber." "parameters": { "Com_Contact_FirstName": "Thomas", "Com_Contact_LastName": "Weber"} "I want to call Rafael Busch."" "parameters": { "Com_Contact_FirstName": "Rafael", "Com_Contact_LastName": "Busch"}. .
[0036] In another embodiment, the large language model generates at least the following variants of the voice command, which go beyond the rigid form of BNF syntax but do not deviate from the semantic meaning: "Call Anna Schmidt." "parameters": { "Com_Contact_FirstName": "Anna", "Com_Contact_LastName": "Schmidt"} "Connect me to Johannes Meier." "parameters": { "Com_Contact_FirstName": "Johannes", "Com_Contact_LastName": "Meier"} "I want to speak to Peter Müller now." "parameters": { "Com_Contact_FirstName": "Peter", "Com_Contact_LastName": "Müller"} "Please call Thomas on his main number." "parameters": { "Com_Contact_FirstName": "Thomas", "Com_Contact_LastName": ""} "Call Mr. Busch!" "parameters": { "Com_Contact_FirstName": "", "Com_Contact_LastName": "Busch"}.
[0037] Here too, the language model incorporates world knowledge about personal names into data generation and, in the variety of voice commands, goes significantly beyond the syntactic rules of BNF. However, the actual user request (calling the telephone function) remains intact.
[0038] In step S106, the dataset is then split into a training dataset and a test dataset, which differ in all sets. Subsequently, in step S108, the machine learning model is trained with the training dataset to determine the vehicle function and at least one function parameter based on user input and to output a corresponding result. For this purpose, the training dataset is fed into the machine learning model as input.
[0039] In one embodiment, the data set is provided as a text data set, so that both the training and test data sets contain text data. The machine learning model then receives, for example, the sentences generated in step S104 in the form of text data and generates an output for each sentence.
[0040] In another embodiment, the data set is provided, for example, as an audio data set, so that both the training and test data sets contain audio. The machine learning model then receives, for example, the sentences generated in step S104 in the form of audio data and generates an output for each sentence. In one embodiment, the machine learning model generates the output directly from the audio data. In another embodiment, the machine learning model uses a speech-to-text model to generate text data from the audio data and creates the output from this text.
[0041] The machine learning model is then tested in step S110 using the test dataset to determine its model quality. For this purpose, the test dataset is fed into the machine learning model as input to determine, for each sentence in the test dataset, whether a generated output is associated with the correct vehicle function. The input in this step is performed in the same way as when training the machine learning model.
[0042] In one embodiment, the determined model quality is a value that represents, for example, the proportion of correctly determined vehicle functions. The process then ends in step S112.
[0043] Fig. Figure 2 shows a schematic representation of a device of a vehicle 200 for controlling a vehicle function. The device comprises a receiver module 202, a processing module 204, and a control module 206, which are shown only as examples of parts of the vehicle 200. Furthermore, a vehicle occupant 214 is in the vehicle 200, and the tailgate 216 is open.
[0044] The receiver module 202 is designed to receive spoken user input from a vehicle occupant. The receiver module 202 can receive the user input, for example, from a microphone 210, such as a microphone 210 of the vehicle 200 or a microphone of a mobile device paired with the vehicle 200. From the user input, the receiver module 202 can generate audio data or convert the spoken user input into text and make it available for further processing by the device. The receiver module 202 is shown purely as an example, as part of the processing unit 208 of the vehicle 200.
[0045] The processing module 204 is trained to operate a machine learning model.
[0046] This means that the processing module 204 is configured to load and execute the machine learning model from a memory element, for example, a memory element 212 of the processing unit 208 of the vehicle 200. The machine learning model has been trained, for example, using a generic text corpus. Furthermore, the machine learning model has been trained according to a method described in the embodiments, e.g., the method from Fig. 1. The processing module 204 has been trained to determine, based on the audio data, a desired vehicle function to be activated and to determine all function parameters required for activation. The processing module 204 can then output to the control module 206 a vehicle function controllable by the control module 206, corresponding to the desired vehicle function, and the required vehicle parameters. The control module 206 is trained to control the vehicle function determined in step S204 and to apply the function parameters.
[0047] The processing module 204 is also shown, purely by way of example, as part of the processing unit 208 of the vehicle 200. In other embodiments, however, the processing module 204 can also be formed wholly or partially by a processing unit located remote from the vehicle 200. In particular, the machine learning model can be executed on such a processing unit located remote from the vehicle 200. In such an embodiment, the machine learning model is preferably stored on a memory element of the processing unit located remote from the vehicle 200. Like the receiver module 202 and the processing module 204, the control module 206 is also shown, purely by way of example, as part of the processing unit 208 of the vehicle 200.
[0048] In an application example of the voice assistant in vehicle 200, the vehicle occupant 214 says: "Close the tailgate." The user input "Close the tailgate" is detected by a microphone 210 in vehicle 200. The receiver module 202 then receives this user input from the vehicle occupant 214 using the microphone 210 and generates audio data from the user input. This audio data is then transmitted to the processing module 204. Based on the audio data, the processing module 204 then determines, for example, the vehicle function CLOSE_MOVABLE, which causes a closable part of vehicle 200 to be closed. In further embodiments, the function for closing a closable part of vehicle 200 can have any other name.Furthermore, in this embodiment, it is determined that the CLOSE_MOVABLE function requires a function parameter to be executed, and the function parameter "tailgate" is determined. Subsequently, the processing module 204 outputs information about the vehicle function CLOSE_MOVABLE and the required vehicle parameter "tailgate" to the control module 206. The control module 206 then activates the vehicle function CLOSE_MOVABLE for the tailgate 216, causing it to close.
[0049] In another application example of the voice assistant in vehicle 200, the vehicle occupant 214 says: "I would like to buy a Christmas present from an online retailer today." The user input "I would like to buy a Christmas present from an online retailer today" is captured by a microphone 210 in vehicle 200. As described previously, the user input is received by the receiver module 202, and corresponding audio data is transmitted to the processing module 204. Based on the audio data, the processing module 204 then determines that there is no vehicle function controllable by the control module 206 that corresponds to the desired function. Therefore, the processing module 204 does not output any information about a controllable vehicle function to the control module 206.
[0050] The device for a vehicle for controlling a vehicle function comprises at least a receiver module 202, a processing module 204 and a control module 206. Method steps described with reference to the device for controlling a vehicle function can be part of the claimed methods. Reference symbol list 200 vehicles 202 Receiver module 204 Processing module 206 Control module 208 processing units 210 microphone 212 storage unit 214 vehicle occupants 216 Tailgate S100 to S112 procedure steps QUOTES INCLUDED IN THE DESCRIPTION
[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature
[0000] CN 117574913 A
[0003] US 2024 / 0095460 A1
[0004] < / temperatureinheit> < / temperatur>
Claims
Method for training a machine learning model for a voice assistant for a vehicle (200), wherein at least one context-free grammar is defined which defines at least one sentence with at least one variable function parameter and is assigned to a vehicle function, wherein, starting from the context-free grammar, a data set with sentences that follow the context-free grammar is generated and provided using a language model, wherein in each sentence the at least one function parameter is varied in compliance with a parameter space specified by the context-free grammar, wherein the data set is divided into a training data set and a test data set, each comprising different sentences, wherein the machine learning model is trained with the training data set to determine the vehicle function and the at least one function parameter based on user input and to output a corresponding output,and in which the machine learning model is tested with the test data set to determine model quality. The method of claim 1, wherein at least two context-free grammars are defined, each defining at least one sentence with at least one variable function parameter and each assigned to a vehicle function, wherein, starting from the context-free grammars, a data set with sentences is generated and provided using the language model, wherein each sentence follows one of the context-free grammars, and wherein in each sentence the at least one function parameter is varied in compliance with the parameter space specified by the respective context-free grammar, and wherein the machine learning model is trained to determine, based on the user input, one of the vehicle functions and the at least one function parameter of the respective vehicle function and to output a corresponding output. Method according to one of the preceding claims, wherein in at least one sentence generated by the language model a word choice and / or word sequence of the context-free grammar is varied while maintaining the semantic meaning. Method according to one of the preceding claims, wherein the context-free grammar is defined in the Backus-Naur form. A method according to one of the preceding claims, wherein the data set is provided as a text data set and the training data set and the test data set contain text data. A method according to any of the preceding claims, wherein the data set is provided as an audio data set and the training data set and the test data set contain audio data. Method according to one of the preceding claims, wherein the language model is a large language model, in particular a generative pre-trained transformer. Method according to one of the preceding claims, wherein the data set is labelled and wherein the machine learning model is trained using a supervised learning method. Device of a vehicle (200) for controlling a vehicle function, comprising: a storage element (212) on which a machine learning model is stored; a receiver module (202) configured to receive spoken user input from a vehicle occupant (214); a processing module (204) configured to load and execute the machine learning model from the storage element (212); and a control module (206) configured to select and control the vehicle function corresponding to a desired vehicle function based on an output of the machine learning model, wherein the machine learning model is trained by means of one of the methods according to claims 1 to 7 to determine the desired vehicle function based on the user input and to generate the output indicatingwhich of the vehicle functions controllable by the control module (206) corresponds to the desired vehicle function and which function parameters necessary for controlling the desired vehicle function are included in the user input. A vehicle (200) with the device according to claim 9.