Device of a vehicle and method for controlling vehicle functions

Abstract

A device (100) of a vehicle (102) for controlling vehicle functions comprises a storage element (122) on which a language model (116) and a classification layer (118) are stored, a receiver module (104) configured to receive spoken user input from a vehicle occupant (110) corresponding to a vehicle function desired by the vehicle occupant (110), a processing module (106) configured to load and execute the language model (116) and the classification layer (118) from the storage element (122), and a control module (108) configured to select and control a vehicle function most likely corresponding to the desired vehicle function based on an output of the classification layer (118). The language model (116) is trained to generate an embedding based on the user input that corresponds to the semantic meaning of the user input.The classification layer (118) is trained to generate, based on the embedding, the output that indicates which of the vehicle functions controllable by the control module (108) most likely corresponds to the desired vehicle function.

Description

[0001] The invention relates to a device for a vehicle for controlling vehicle functions. The invention further relates to a method for controlling vehicle functions.

[0002] Many vehicle functions in modern vehicles can be activated with spoken user input. This allows, for example, a driver to operate vehicle functions without taking their hands off the steering wheel or their eyes off the road. However, so-called voice assistants are limited by their language understanding and can often only correctly interpret specific, predefined commands. Variations in voice input that deviate from these predefined commands pose a challenge. Since modern vehicles have a multitude of functions, for example, 1000, selecting the correct function is particularly challenging. Users can express their commands in very different ways, which increases the complexity of correctly classifying and assigning voice inputs to the corresponding vehicle functions.

[0003] CN 112927695 discloses a speech recognition method. This method uses two language models. A first model assigns a category to a speech input, while a second model processes the speech input according to the category.

[0004] US patent 2024 / 0095460 A1 discloses a dialogue system for vehicles that is capable of capturing and categorizing user questions using a language model. Depending on the categorization, information can then be retrieved.

[0005] The object of the invention is to provide a device for a vehicle and a method for controlling vehicle functions that can implement spoken user input better than known devices and methods.

[0006] This problem is solved by a device having the features of claim 1 and by the subject matter of the dependent claims. Further developments are specified in the dependent claims.

[0007] The proposed device for controlling vehicle functions comprises a memory element on which a language model and a classification layer are stored, a receiver module trained to receive spoken user input from a vehicle occupant corresponding to a vehicle function desired by the occupant, a processing module trained to load and execute the language model and the classification layer from the memory element, and a control module trained to select and control a vehicle function most likely corresponding to the desired vehicle function based on an output from the classification layer. The language model is trained to generate an embedding based on the user input that corresponds to the semantic meaning of the user input.The classification layer is trained to generate, based on the embedding, the output that indicates which of the vehicle functions controllable by the control module most likely corresponds to the desired vehicle function.

[0008] The proposed device implements a voice assistant for the vehicle that controls a vehicle function based on spoken user input. The user input is first processed by the language model. The output of the language model is the embedding, a mathematical representation of the semantic content of the user input, for example, as a vector in a high-dimensional vector space. Embeddings of user inputs with similar semantic content will be located closer together in this high-dimensional vector space, the embedding space, than embeddings of user inputs with different semantic content. This enables the appropriately trained classification layer to assign different user inputs, but related to the same vehicle function, to the same vehicle function.For example, a user input might be used to open a vehicle window. The user input could be: "Open the window." However, the vehicle occupant could also use alternative phrases, such as: "Open the window" or "Lower the window." The language model assigns an embedding to each of these user inputs, all of which lie within a limited neighborhood of the embedding space. The classification layer then assigns the vehicle function "Open window" to all embeddings in this neighborhood. The control module then executes the vehicle function determined by the classification layer. Thus, the voice assistant created by the device is able to control vehicle functions even based on complex and variably formulated user inputs.

[0009] In one embodiment, the classification layer is trained using a training dataset containing exemplary user inputs, each labeled with a corresponding vehicle function, and using the language model, in which the exemplary user inputs are fed into the language model as input. The classification layer is trained using a supervised learning approach. During the training of the classification layer, the language model remains unchanged. For example, one element of the training dataset consists of the user input "Open the window" and the associated vehicle function "Open window." The language model generates an embedding for each exemplary user input, which is fed into the classification layer as training input. The labeled output is the vehicle function that corresponds to the respective exemplary user input.This training enables the classification layer to robustly assign variations in user input to the corresponding vehicle function, even if these variations were not part of the training dataset. For example, the following sample user inputs for the vehicle function "Open window" were used as part of the training dataset: "Open the window" and "Make the window open." After training, variations of these user inputs, such as "Window up," "Open window," and "Please open the window," will also have a high probability of being correctly assigned to the vehicle function "Open window."

[0010] Alternatively or additionally, a training dataset can be used which includes the corresponding output of the language model instead of the example user inputs.

[0011] In one embodiment, the language model is trained using a generic text corpus. This generic text corpus comprises texts that are not limited to a specific subject area. For example, the Toronto Book Corpus or a filtered version of Wikipedia can be used as the generic text corpus. Training with the generic text corpus gives the language model a general understanding of language and a broad knowledge base. This general knowledge is also referred to as world knowledge. This world knowledge enables the language model to assign different user inputs with the same semantic content to an embedding in the same neighborhood of the embedding space, even if it was not specifically trained on these user inputs.

[0012] In one embodiment, the classification layer comprises at least one neural network. Neural networks are capable of reliably recognizing complex patterns even in high-dimensional datasets, such as the embedding space. This enables the neural network to correctly assign embeddings to the same vehicle function, even if they relate to the same vehicle function but have semantic content that is so different they do not lie within a geometrically simple neighborhood within the embedding space, for example, "Open the window" and "The air is very bad." Alternatively or additionally, the classification layer can include further elements, such as elements of a transformer architecture like one or more attentionheads.

[0013] In one embodiment, the language model is either a Large Language Model or a Small Language Model. Large Language Models (LLM) and Small Language Models (SLM) are classes of language models that differ primarily in the size of the training dataset used to train them. SLMs can be optimized for a specific task. LLMs are typically trained with a text corpus that can be several hundred gigabytes in size. SLMs are typically trained with a text corpus that is only a few gigabytes in size. LLMs and SLMs also differ in the number of parameters and thus the size of the model itself. An LLM can have one hundred billion parameters; for example, GPT-3 has 175 billion parameters, while an SLM typically has no more than one billion parameters; for example, BERT has 340 million parameters.By appropriately selecting the model size, low latency can be ensured, and the speech model can also be run on the vehicle's limited hardware. For example, BERT can be operated with an inference time of 20 ms, which is imperceptible to humans. BERT, or one of its many successors and enhancements, such as DistilBERT, ALBERT, roBERTa, ELECTRA, and T5, can be used as the speech model.

[0014] In one embodiment, the receiving module is configured to convert the user input into a text format that can be processed by the language model. For example, the receiving module can be configured to generate a text-based list of words (tokens) based on the spoken user input, corresponding to the user input. The language model can be kept particularly simple if the input to the language model is in text form.

[0015] In one embodiment, the output of the classification layer comprises an ordered list containing a numerical value for each of the vehicle functions controllable by the control module. This value indicates the probability that this vehicle function corresponds to the desired vehicle function. In such an embodiment, the classification layer resolves the embedding by reducing the high-dimensional embedding to a vector whose dimensionality corresponds to the number of controllable vehicle functions. Each entry in this vector corresponds to the probability that one of the vehicle functions is the desired vehicle function. The control module can then, for example, determine the vehicle function with the highest probability and control it.If the desired vehicle function cannot be clearly determined, the control module can, for example, activate an output unit of the vehicle to prompt the vehicle occupant to repeat the user input.

[0016] In one embodiment, the output of the classification layer comprises unique identifiers of at least two of the vehicle functions controllable by the control module that most likely correspond to the desired vehicle function. Each identifier is assigned a numerical value indicating the probability that the respective vehicle function corresponds to the desired vehicle function. The control module is configured to select and control one of the at least two vehicle functions controllable by the control module based on these numerical values. In this embodiment, the classification layer generates a list of the vehicle functions that most likely correspond to the desired vehicle function. Using this list, the control module determines which of the vehicle functions should be controlled. If the desired vehicle function can be uniquely identified from the list, it can be controlled immediately.In some embodiments, the control unit may also be configured to control the vehicle's output unit in order to display the most likely vehicle functions to the vehicle occupant and to allow the selection of one of these vehicle functions.

[0017] In one embodiment, the processing module is part of the vehicle. For example, the processing module is part of a processing unit within the vehicle, such as a central vehicle computer. Alternatively, the processing module can be implemented, at least partially, by a processing unit located remote from the vehicle, such as a server, or in a cloud computing environment. For example, the language model is executed on a server remote from the vehicle or in a cloud computing environment. In such an embodiment, the language model is preferably stored on a memory element of the processing unit located remote from the vehicle. This allows the use of a language model that might not be able to run on the limited hardware of the vehicle, or only with very high latency.The processing module can also be part of a mobile device that can be installed in the vehicle, for example, a smartphone or a tablet computer belonging to a vehicle occupant. In such an embodiment, the language model is preferably stored on a memory element of the mobile device.

[0018] According to a further aspect, the invention relates to a method for controlling vehicle functions of a vehicle. The method involves at least the following steps: A spoken user input is received from a vehicle occupant, corresponding to a vehicle function desired by the occupant. Using a language model and based on the user input, an embedding is created that corresponds to the semantic meaning of the user input. Using a classification layer and based on the embedding, an output is generated that indicates which of the vehicle functions controllable by the control module most likely corresponds to the desired vehicle function. Based on the output of the classification layer, a vehicle function of the vehicle is selected and controlled.

[0019] The method has the same advantages as the claimed device. In particular, the method can be further developed with features described in this document in connection with the device. Furthermore, the claimed device can be further developed with features described in this document in connection with the method.

[0020] In one embodiment, the classification layer is retrained when a vehicle function changes, when a previously available vehicle function is no longer available, and / or when a new vehicle function becomes available. In this embodiment, the training of the classification layer is only repeated when, for example, new vehicle functions become available. This saves the considerable computational effort required for training the language model.

[0021] Exemplary embodiments of the invention are explained in more detail below with reference to the figures. These show: Fig. 1 a schematic representation of a device of a vehicle for controlling vehicle functions according to one embodiment; and Fig. 2 a flowchart of a procedure for controlling vehicle functions according to an embodiment.

[0022] Fig. Figure 1 shows a schematic representation of a device 100 of a vehicle 102 for controlling vehicle functions according to one embodiment. Vehicle functions controllable by the device 100 include, for example, opening a window of the vehicle 102, starting route guidance, activating seat heating, activating ventilation, initiating a call, controlling lighting, controlling an entertainment system, controlling a vehicle mode, providing an operating aid, querying the status, and providing a support function for the vehicle 102. The device 100 comprises a receiver module 104, a processing module 106, and a control module 108, which are shown only as examples of parts of the vehicle 102.

[0023] The receiver module 104 is configured to receive spoken user input from a vehicle occupant 110. In the following, the user input always corresponds to a vehicle function requested by the vehicle occupant 110, i.e., a vehicle function to be executed by the vehicle 102. To receive the user input in spoken form, the receiver module 104 can be configured to receive the user input as audio data from a microphone, for example, a microphone 112 of the vehicle 102 or a microphone of a mobile device connected to the vehicle 102. From the user input, the receiver module 104 can generate audio data or convert the spoken user input into text and make it available for further processing by the device 100. The receiver module 104 is shown purely as an example of a processing unit 114 of the vehicle 102, for example, a central vehicle computer.

[0024] The processing module 106 is trained to operate a language model 116 and a classification layer 118. This means that the processing module 106 is trained to load and execute the language model 116 and the classification layer 118 from a memory element, for example, a memory element 122 of the processing unit 114 of the vehicle 102. The language model 116 has been trained to generate an embedding from the user input, for example, based on a generic text corpus. The embedding is, for example, a vector in a high-dimensional embedding space and corresponds to the semantic meaning of the user input. The embedding is further processed by the classification layer 118, which has been trained to generate an output indicating which of the vehicle functions controllable by the control module 108 is most likely to be activated by the user input.The output can, for example, be an ordered list of numerical values ​​indicating, for each vehicle function, the probability that it will be activated by user input. Alternatively, the classification layer 118 can also output the most probable vehicle functions, for example, the two, three, or ten most probable vehicle functions. The processing module 106 is also shown, purely by way of example, as part of the processing unit 114 of the vehicle 102. In other embodiments, however, the processing module 106 can also be formed wholly or partially by a processing unit located remote from the vehicle 102. In particular, the language model 116 can be executed on such a processing unit located remote from the vehicle 102. In such an embodiment, the language model 116 is preferably stored on a memory element of the processing unit located remote from the vehicle 102.

[0025] Control module 108 is designed to control a vehicle function based on the output of classification layer 118. For example, control module 108 determines, based on the output of classification layer 118, which of the vehicle functions has been classified by classification layer 118 as the most probable vehicle function—for example, the vehicle function with the highest numerical value—and controls it. If control module 108 cannot unambiguously determine which vehicle function is the desired one based on the output of classification layer 118—for example, if none of the vehicle functions has been clearly classified as the most probable vehicle function—control module 102 can, for example, control an output unit 120 of the vehicle to prompt the vehicle occupant to repeat the user input.Like the receiver module 104 and the processing module 106, the control module 108 is also shown purely as an example as part of the processing unit 114 of the vehicle 102.

[0026] Fig. Figure 2 shows a flowchart of a method for controlling vehicle functions according to one embodiment. The method can be carried out, for example, using device 100 according to Fig. 1 will be carried out.

[0027] In step S200, the process is initiated. In step S202, the spoken user input from vehicle occupant 110 is received, corresponding to a vehicle function desired by vehicle occupant 110. With this user input, the vehicle occupant specifies which vehicle function should be activated. For example, the vehicle occupant says "open the window" or "make the window open" if they want a window of vehicle 102 to be opened. In another example, vehicle occupant 310 says "Navigate me to Lauchstädter Straße 11 in Munich" if they want to start route guidance. Vehicle occupant 310 could also say "Turn on the seat heating on the driver's seat" to activate the seat heating of a driver's seat, "Activate the ventilation" to activate the ventilation, "Call <person>"to call the person", or "Play the radio station BR-Klassik", "Play music on <drittanbieter-applikation>"or "Let the children play a video game" to control an entertainment system of the vehicle 102. The input can also be a generic input, such as "Confirm the input", or a status query or a request for support, such as "How can I find charging points along my route?", "Why is a warning light on?", or "Did I leave something in the car?" The user input is received, for example, by the receiver module 104 of the device 100.

[0028] In step S204, using language model 116 and based on the user input, an embedding is created that corresponds to the semantic meaning of the user input. The embedding is a mathematical representation of the meaning of the user input, for example, a high-dimensional vector in an embedding space. Embeddings of user inputs with the same or similar meaning are closer together than embeddings of user inputs with different meanings. This allows "similar meaning" to be defined mathematically. Step S204 is performed, for example, by processing module 106. Training of language model 116 can be performed as an optional step within the procedure. Alternatively, a pre-trained language model 116 can be used.

[0029] In step S206, using classification layer 118 and based on the embedding, an output is generated indicating which of the vehicle functions controllable by control module 108 most likely corresponds to the desired vehicle function. For example, classification layer 118 generates an ordered list of numerical values ​​from the high-dimensional vector in the embedding space, i.e., another vector with a significantly smaller dimension. Thus, classification layer 118 assigns one or more specific vehicle functions, likely to be controlled, to the meaning of the user input determined by language model 116. Step S206, like step S204, is performed, for example, by processing module 106.

[0030] Classification layer 118 has been trained to generate output, based on the embedding, indicating which of the vehicle functions controllable by control module 108 most likely corresponds to the desired vehicle function. For training classification layer 118, a training dataset is created containing sample user inputs, each labeled with a corresponding vehicle function. This training dataset is then fed into language model 116 as input to generate an embedding for each sample user input, which is then fed back into classification layer 118 as training input. The parameters of classification layer 118 are varied until its output consistently matches the correct vehicle functions. The parameters of language model 116 remain unchanged.This training can be performed as an optional step within the procedure. In particular, the training of classification layer 118 can be repeated as part of the procedure if, for example, new vehicle functions become available in vehicle 102.

[0031] In step S208, a vehicle function of vehicle 102 is selected and activated based on the output of classification layer 118. For example, the most probable vehicle function from the output of classification layer 118 is selected and activated. If, for example, several vehicle functions are equally probable, or if none of the vehicle functions is more probable than a predetermined threshold, such as 50%, the vehicle occupant may also be prompted to repeat their user input. Step S208 is performed, for example, by control module 108. The procedure is then terminated in step S210.

[0032] In the based on the Fig. 1 and Fig. In the embodiments described in section 2, at least the receiving module 104, the processing module 106, and the control module 108 form the device 100 of a vehicle 102 for controlling vehicle functions. Further embodiments are described in the Fig. 1 and Fig. The two elements and features shown and mentioned in the preceding description can be part of the device 100. Likewise, process steps described with reference to the device 100 can be part of the claimed method. Reference symbol list 100 Device 102 vehicles 104 Receiver module 106 Processing module 108 Control module 110 vehicle occupants 112 microphone 114 processing units 116 Language model 118 classification layers 120 output units QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] CN 112927695

[0003] US 2024 / 0095460 A1

[0004] < / person>

Claims

[1] Device (100) of a vehicle (102) for controlling vehicle functions, comprising, a storage element (122) on which a language model (116) and a classification layer (118) are stored, a receiving module (104) which is designed to receive spoken user input from a vehicle occupant (110) which corresponds to a vehicle function desired by the vehicle occupant (110), a processing module (106) trained to load and execute the language model (116) and the classification layer (118) from the storage element (122), and a control module (108) which is trained to select and control a vehicle function that most likely corresponds to the desired vehicle function based on an output of the classification layer (118), wherein the language model (116) is trained to generate an embedding based on the user input that corresponds to the semantic meaning of the user input, and wherein the classification layer (118) is trained to generate, based on the embedding, the output which indicates which of the vehicle functions controllable by the control module (108) most likely corresponds to the desired vehicle function. [2] Device (100) according to claim 1, wherein the classification layer (118) has been trained using a training data set comprising exemplary user inputs, each labelled with a corresponding vehicle function, and using the language model (116) in which the exemplary user inputs have been entered into the language model (116) as input. [3] Device (100) according to claim 1 or 2, wherein the language model (116) has been trained using a generic text corpus. [4] Device (100) according to one of the preceding claims, wherein the classification layer (118) comprises at least one neural network. [5] Device (100) according to any of the preceding claims, wherein the language model (116) is a Large Language Model or a Small Language Model. [6] Device (100) according to one of the preceding claims, wherein the receiving module (104) is configured to convert the user input into a text format that can be processed by the language model (116). [7] Device (100) according to one of the preceding claims, wherein the output of the classification layer (118) comprises an ordered list which includes, for each of the vehicle functions controllable by the control module (108), a numerical value indicating the probability with which this vehicle function corresponds to the desired vehicle function. [8] Device (100) according to any one of claims 1 to 6, wherein the output of the classification layer (118) comprises unique identifiers of at least two of the vehicle functions controllable by the control module (108) that most likely correspond to the desired vehicle function and to which each is assigned a numerical value indicating the probability with which the respective vehicle function corresponds to the desired vehicle function, and wherein the control module (108) is configured to select and control one of the at least two vehicle functions controllable by the control module (108) on the basis of the numerical values. [9] Device (100) according to one of the preceding claims, wherein the processing module (106) is part of the vehicle (102). [10] Device (100) according to one of the preceding claims, wherein the processing module (106) is part of a processing unit located away from the vehicle (102). [11] Method for controlling vehicle functions of a vehicle (102) wherein a) a spoken user input is received from a vehicle occupant (110) that corresponds to a vehicle function requested by the vehicle occupant (110); b) using a language model (116) and based on user input, an embedding is created which corresponds to the semantic meaning of the user input; c) using a classification layer (118) and based on the embedding, an output is generated indicating which of the vehicle functions controllable by the control module (108) most likely corresponds to the desired vehicle function; and d) based on the output of the classification layer (118) a vehicle function of the vehicle (102) is selected and controlled. [12] Method according to claim 11, wherein the classification layer (118) is retrained when a vehicle function has changed, when a previously available vehicle function is no longer available and / or when a new vehicle function is available.

Images