Computer-implemented method for providing a virtual assistant for controlling a vehicle function and vehicle

Abstract

The invention relates to a computer-implemented method (100) for providing a virtual assistant for controlling a vehicle function, comprising at least the following steps: generating (102) text data from a user's speech input using a pre-trained speech-to-text model (18); analyzing (104) the text data using a wake-word model (20) for a trigger text; if a trigger text is found in the text data: generating (106) a first-type output data set from the text data using a first LLM (24) comprising a first latency; providing (108) a first control signal indicating the first speech data generated from the first-type output data set, for controlling an output device (46) for outputting the speech data;wherein the following steps are performed: generating (110) second speech data based on a second-kind output data set generated from the text data and the first-kind output data set, and based on a similarity search (32) in a database (38) for at least the text data, using a second LLM (28) with a latency greater than the first latency; and replacing (112) the first control signal with a second control signal indicating the second speech data, to control an output device.

Description

[0001] The invention relates to a computer-implemented method for providing a virtual assistant for controlling a vehicle function and a vehicle.

[0002] Personalized virtual assistants can be used to control vehicle functions, communicating with the driver via spoken language. The user's speech is recorded and converted into text by a conversion program. This text is then fed into a language model that provides output data. This output data can be used to respond to the user and control vehicle functions.

[0003] For example, US patent 2020 / 0380139 A1 discloses the provision of an interface that mediates between a vehicle user and a voice model. The vehicle user can use voice commands to create a route for the vehicle in a navigation device. The voice model can interact with the vehicle user using spoken language to plan the route.

[0004] Furthermore, implementations are known from US 2024 / 0311405A1 that reveal the selection of a specific generative model in response to a query from among several candidates for generative models (e.g., several candidates for large language models (LLMs)) with varying computational performance to be used in generating a response to the query. These implementations reduce latency and / or conserve computational resources by selecting a computationally more efficient generative model instead of a less efficient one for different requirements. Moreover, these implementations aim to achieve such benefits by using computationally more efficient generative models while also selectively using less efficient generative models for certain queries to avoid the occurrence of inaccurate and / or insufficiently specified responses.This in turn can reduce the occurrence of computational and / or network inefficiencies that arise from a user making a follow-up request to correct inaccuracies and / or under-specification of a generated response.

[0005] The object of the invention is to provide a computer-implemented method for providing a virtual assistant for controlling a vehicle function, which has an increased response speed and an increased content accuracy.

[0006] The problem is solved by the features of the independent claims. Advantageous further developments are the subject of the dependent claims and the following description.

[0007] According to a first aspect, a computer-implemented method for providing a virtual assistant for controlling a vehicle function is described, wherein the method comprises at least the following steps: generating text data from at least one user speech input using a pre-trained speech-to-text model; analyzing the text data using at least one wake-word model for at least one trigger text; if at least one trigger text is found in the text data using the wake-word model: generating at least one first-type output data record from the text data using a first large-language model comprising a first latency; providing at least one first control signal indicating first speech data generated from the at least one first-type output data record, for controlling an output device to output the speech data;wherein the following steps are performed during the output of the first speech data: generating second speech data based on at least one second-type output data set generated from the text data and at least one first-type output data set by means of at least one second large-language model with a second latency greater than the first latency, and based on at least one similarity search in at least one database for at least the text data; and replacing the at least one first control signal with at least one second control signal indicating the second speech data, for controlling an output device.

[0008] The computer-implemented method uses the first large-language model (LLM) to generate first-type output data records with minimal latency, enabling the creation of initial speech data as a response from the virtual assistant to the user's at least one speech input with minimal delay. The user's speech input is first converted into text data using a pre-trained speech-to-text model. Then, the wake-word model, which can preferably be operated locally within a vehicle, can be used to check whether the virtual assistant has been addressed or whether a question from the virtual assistant has been answered. The text data is then transmitted to the first LLM. This means that initially only the first LLM generates at least one first-type output data record, without the second LLM providing at least one second-type output data record.The first type of output data record is then used to output initial speech data as a response from the virtual assistant via an output device. While the speech data is being output by the output device, the text data and at least one first type of output data record are transmitted to the second LLM, which has a higher latency than the first LLM. The second LLM uses a similarity search in at least one database, searching for records in the database that are similar to the text data. For this purpose, the text data can, for example, be embedded in the first type of output data records and / or in an intermediate output data record generated by the second LLM using the text data.By using the first LLM, which has a lower latency than the second LLM, the text data can be processed more quickly, allowing a spoken response to be output via the output device with minimal delay. The second LLM then generates a second set of speech data, the content of which can be more precisely tailored to the text data, i.e., the user's request. This can be achieved, in particular, through database queries within the framework of similarity searches, which increases the probability that the content accuracy of the second set of speech data is higher than that of the first. Content accuracy refers to how closely the speech data matches the content of the text data or the user's request.The first set of speech data can be replaced by the second set of speech data while it is being output, by substituting the first control signal for the second. Since the second set of speech data is based on at least one first-type output record, the semantics of the second set of speech data match the first set of speech data. Therefore, the exchange of speech data can occur without the user noticing. That is, while the first set of speech data is being output, the subsequent response of the virtual assistant is improved by the second LLM and the provision of the second set of speech data as a replacement for the first set of speech data. First-type output records may have lower semantic accuracy than second-type output records.The first LLM can be used to prepare an initial part of the virtual assistant's response, until the second LLM provides second-type output data sets that can be used to prepare the remaining parts of the virtual assistant's response.

[0009] An LLM is a trainable language learning model. This model might, for example, be a neural network with an input level and an output level. Data is provided to the LLM at the input level, and data from the LLM can be provided at the output level. Both input and output data can be text. The output data sets described above can contain the output data.

[0010] According to some embodiments, it is conceivable that the step: generating second language data, may further include at least the following substep: generating at least one query term for the second LLM from the similarity search with at least the text data.

[0011] The query term allows the generation of the second type of output data to be carried out effectively and efficiently, with the second type of output data then being based on the information from the database.

[0012] According to some embodiments, it is conceivable that the similarity search can be carried out using at least one search model, preferably a BM25 model, in particular in combination with a ranking model, preferably an OpenAI™ Embedding model.

[0013] The search model can, for example, identify those records from the database that are most similar to the output records of the first and / or second type. The ranking model can then rank the records identified in this way, with the best-matching record appearing at the top of the ranking. This enables efficient similarity searches, saving resources and thus reducing costs.

[0014] According to some embodiments, it is conceivable that the database could be designed as a vector database, preferably a Pinecone™ database.

[0015] By using a vector database, datasets with embedded text or documents can be retrieved using similarity searches. In particular, a Pinecone™ database offers a fast response time, thus minimizing further delays in the processing of the computer-implemented method.

[0016] According to some embodiments, it is conceivable that the step "Generating second language data" may further comprise at least the following substeps: transmitting result data sets from the similarity search to the second large-language model; and generating at least one output data set of the second type as a basis for the second language data using the second large-language model based on the result data sets.

[0017] The similarity search can yield result datasets, which may contain embedded texts in the form of documents. These result datasets can be provided to the second LLM (Language Language Management) system, which can generate at least one second-type output dataset from them. This further improves the content accuracy of the second language dataset.

[0018] According to some embodiments, it is conceivable that the first speech data and / or second speech data can be generated using a text-to-speech model, preferably an ElevenLabs™ model.

[0019] The text-to-speech model can be a trainable model, for example, one that can be run on a neural network. Specifically, when using the ElevenLabs™ model, a voice that sounds almost human can be provided for speech output. Furthermore, it can offer high accuracy in converting text data into speech data. The ElevenLabs™ model can be pre-trained and already fine-tuned.

[0020] According to some embodiments, it is conceivable that the first Large Language model can be designed as a Llama 3 model and the second Large Language model can be designed as a ChatGPT™ model.

[0021] The first LLM can, in particular, be a Llama 3 80B model. It can further be executed, for example, on at least one language processor unit (LPU), which may be part of an inference engine. The language processor unit may be specifically designed for modeling LLMs and preferably feature a tensor streaming processor architecture. In this way, the processing speed of the first LLM can be further increased, thus reducing the response time of the virtual assistant.

[0022] Furthermore, the use of the ChatGPT™ model can provide increased accuracy in the generation of the second type of output data sets and thus the corresponding language data based on them.

[0023] According to some examples, it is conceivable that the speech-to-text model could be designed as an Azure™ model.

[0024] The Azure™ model can be trained to provide so-called cognitive services. This enables the conversion of speech input to text with high accuracy and speed.

[0025] According to a second aspect, a computer program product is described, comprising instructions that, when the program is executed by a computer, cause it to perform the steps of the procedure according to the preceding description.

[0026] The advantages, effects, and further developments of the computer program product result from the advantages, effects, and further developments of the method described above. Therefore, reference is made to the preceding description in this regard. A computer program product can be understood, for example, as a data carrier on which a computer program element is stored, containing instructions executable by a computer. Alternatively or additionally, a computer program product can also be understood, for example, as a permanent or volatile data storage medium, such as flash memory or main memory, that contains the computer program element. However, this does not exclude other types of data storage media that contain the computer program element.

[0027] According to a third aspect, a vehicle is described comprising at least one computer that is trained to carry out the steps of the procedure according to the preceding description.

[0028] The advantages, effects, and further developments of the vehicle result from the advantages, effects, and further developments of the procedure described above. To avoid repetition, reference is therefore made to the preceding description in this regard.

[0029] The invention is described below with reference to an exemplary embodiment and the accompanying drawing. The drawing shows: Fig. 1. A schematic representation of the procedure; Fig. 2. A flowchart of the procedure; and Fig. 3 A schematic representation of the vehicle.

[0030] The computer-implemented method for providing a virtual assistant for controlling a vehicle function is referred to in the following in its entirety by reference numeral 100.

[0031] According to Fig. 1. The method 100 can be executed as a computer program product on a computer 10. The computer 10 can have an input device 44 and an output device 46. The input device 44 can be, for example, a microphone, and the output device 46 can be, for example, a loudspeaker. The input device 44 and the output device 46 can form a user interface 14 through which a user 12, in particular a vehicle user, can provide voice commands as voice input.

[0032] Voice input can be used according to step 102 of the procedure described in Fig. The flowchart of procedure 100 shown in section 2 is used to convert the data into text using a pre-trained speech-to-text model 18. The pre-trained speech-to-text model 18 could, for example, be an Azure™ model capable of providing cognitive services.

[0033] Furthermore, the speech-to-text model 18 can be executed outside of computer 10 on at least one external server. For example, computer 10 can transmit the speech input data to the external server via a wireless signal connection using a speech-to-text function 16. On the external server, the speech-to-text model 18 can convert the speech input data into text data. The text data can then be transmitted back to computer 10 via another wireless signal connection. However, this does not preclude the possibility of executing the speech-to-text model 18 on computer 10 itself.

[0034] The text data obtained in this way can be analyzed in a further step 104 by a Wake-Word model 20. The Wake-Word model 20 can analyze the text data for at least one trigger text.

[0035] A trigger text could be, for example, a specific combination of words such as "hey Mark".

[0036] If the Wake-Word model 20 does not detect a trigger text in the text data, it can wait for further voice input from the user 12.

[0037] If the Wake-Word model 20 detects at least one trigger text in the text data, the text data can be transmitted to a first LLM 24 in a further step 106. The first LLM 24 exhibits an initial latency.

[0038] The first LLM 24 can be configured as a Llama-3 model, in particular as a Llama-3-80B model. Furthermore, the first LLM 24 can be executed on an inference machine that can utilize a speech processing unit. The speech processing unit can be specifically designed and configured for executing LLMs and may have a tensor streaming processor architecture. Accordingly, in this embodiment, the speech processing unit is therefore not configured as a graphics processing unit or a central processing unit.

[0039] From the text data, the first LLM 24 can generate at least one output record of the first type. From this output record of the first type, 108 initial language data records can be generated in a further step.

[0040] The first-type output data set can be transmitted by a text-to-speech function 40 via a wireless signal link to one or more external servers, on which a text-to-speech model 42 can be executed to output initial speech data from the text data. The text-to-speech model 42 can, for example, be an ElevenLabs™ model. The text-to-speech model can be fine-tuned to increase recognition accuracy or output accuracy. The generated initial speech data can also be transmitted back to the text-to-speech function 40 via the wireless signal link. However, it can also be provided that the text-to-speech model 42 can be executed locally on the computer 10.

[0041] The first speech data are transmitted as a first control signal to an output device 46 according to step 108, with which the first speech data are then output.

[0042] The output of the speech data via the output device 46 requires a certain amount of time. During this time, a second set of speech data is generated according to step 110.

[0043] The second set of voice data is transmitted via a second LLM 28. This second LLM 28 has a second latency that is greater than the first. This means that the first LLM 24 can be configured as a Low Latency Responder 22.

[0044] The second LLM 28 can be trained as a GPT4 model. Specifically, the GPT4 model can be trained as a gpt-4o model based on an Azure™ service. The second LLM 28 can be trained as Agent 26 for providing second-type output datasets.

[0045] The terms "first type" and "second type" indicate which LLM 24 and 28 the output data set originates from. The second LLM 28 can provide further output data sets that may exhibit higher content accuracy than the output data sets of the first LLM 24.

[0046] To generate the second language data, the text data and at least one output record of the first kind are transmitted to the second LLM 28. The second LLM 28 then uses a similarity search 32 in at least one database 38 to search for records similar to the text data. For this purpose, the text data can, for example, be embedded in the at least one output record of the first kind. Alternatively or additionally, the text data can be embedded in an intermediate output record determined by the second LLM 28 based on at least the text data. Using the similarity search 32, the text data, and the output data of the first kind, the second LLM 28 can then generate at least one output record of the second kind.

[0047] For the similarity search 32, at least one search model 34 can be used, which examines the records of database 38 for the text data. The search model 34 can, for example, be a best-match model, in particular a BM25 model. Conveniently, the BM25 model is executed locally on computer 10.

[0048] The data records found using the search model 34 can be further ordered using a ranking model 36, whereby the data record that most closely matches the text data can receive the highest rank in the order.

[0049] The ranking model 36 can, for example, be an embedding model that provides a results dataset in which, for example, text and / or documents from the respective found dataset in database 38 can be embedded. Furthermore, the ranking model 36 can, for example, include an OpenAI™ embedding model.

[0050] Database 38 can be a vector database, in particular a Pinecone™ database. The result of the similarity search 32 can therefore be at least one result record with embedded text or with at least one document.

[0051] Step 110 may include the optional step 116, in which at least one result data record from the similarity search 32 is transmitted to the second LLM 28.

[0052] For the transmission of the result data set, according to optional substep 114, at least one query term 30 can first be generated from the result data set, which can then be provided to the second LLM 28. According to further optional step 118, the second LLM 28 can then generate at least one output data set of the second type from the query term 30, from which the second language data can then be generated.

[0053] Using the result data set, second speech data can be generated by means of the second LLM 28, which can represent a more accurate response from the virtual assistant to the user's speech input than the first speech data.

[0054] The second set of speech data can be transmitted to the output device 46 using a modified second control signal. This can also occur while the first set of speech data is still being output. Once the output device 46 has received the second set of speech data, the output of the first set of speech data can be stopped before it has been completely output. The output device 46 can then seamlessly play back the second set of speech data. This means that there can be no perceptible pause for a user between the stopping of the output of the first set of speech data and the start of the output of the second set of speech data.

[0055] Since the second set of speech data is based, among other things, on the output data sets of the first type, from which the first set of speech data can be generated using the text-to-speech model 42, the switch between the first set of speech data and the second set of speech data during output can be performed in such a way that a user does not notice any breaks in the content. That is, the user does not notice, even in terms of content, that a switch to the second set of speech data takes place during the output of the first set of speech data.

[0056] By initially providing only the first speech data, generated by the LLM 24 with low latency, a very fast response to user speech input is possible. During this initial response, the computer-implemented procedure can use the output time of the first speech data to generate a more precise response using the second set of speech data via the LLM 28, which has a higher latency than the first LLM 24.

[0057] Both the first LLM 24 and the second LLM 28 can be run on one or more external servers. For this purpose, computer 10 can transmit the text data to the external servers via a wireless signal connection so that the text data can be made available to the first LLM 24 and / or the second LLM 28. However, this does not preclude the first LLM 24 and / or the second LLM 28 from being run on computer 10.

[0058] In Fig. 3 is a vehicle 50 which may have a computer 10. The computer 10 may be configured to execute the method 100 described above. Furthermore, the computer 10 may be connected to or include an input device 44 and an output device 46. The input device 44 may, for example, include a microphone or a device for entering text. The output device 46 may, for example, include a loudspeaker or a screen.

[0059] Furthermore, the computer 10 can be connected to a transmitting and receiving device 48, with which the computer 10 can wirelessly transmit data to at least one external server 52. In particular, at least one of the following models can be executed on the external server 52: speech-to-text model 18, first LLM 24, second LLM 28, ranked model 36, and / or text-to-speech model 42. However, in some alternative embodiments, one or each of these models can also be executed locally on the computer 10.

[0060] Furthermore, the database 38 can also be stored on at least one external server 52. In some embodiments, it is also conceivable that the database 38 could be available locally on computer 10.

[0061] The example described above does not in any way limit the invention. Rather, the invention can be modified in numerous ways. All features of the invention described above can be essential to the invention, either alone or in combination. Reference symbol list 10 computers 12 users 14 User interface 16 Speech-to-text function 18 Speech-to-Text Model 20 Wake-Word Model 22 Low Latency Responder 24 Large-Language Model 26 Agent 28 Large-Language Model 30 query term 32 Similarity Search 34 Research model 36 Ranking model 38 ranking model 40 Text-to-speech function 42 Text-to-speech model 44 Input device 46 Dispensing device 48 Transmitting and receiving device 50 vehicles 52 external servers

Claims

[1] Computer-implemented method (100) for providing a virtual assistant for controlling a vehicle function, wherein the method (100) comprises at least the following steps: a. Generating (102) text data from at least one user speech input using a pre-trained speech-to-text model (18); b. Analyzing (104) the text data using at least one Wake-Word model (20) for at least one trigger text; c. If at least one trigger text was found in the text data using the Wake-Word model (20): Generate (106) at least one output record of the first kind from the text data using a first large-language model (24) comprising a first latency; d. Providing (108) at least one first control signal indicating the first speech data generated from the at least one first-type output data record, for controlling an output device (46) for outputting the speech data; e. wherein the following steps are performed during the output of the first language data: f. Generating (110) second language data based on at least one second-type output data set generated from the text data and at least one first-type output data set using at least one second large-language model (28) with a second latency greater than the first latency, and based on at least one similarity search (32) in at least one database (38) for at least the text data; and g. Replacing (112) the at least one first control signal with at least one second control signal indicating the second speech data, for controlling an output device (46). [2] Computer-implemented method (100) according to claim 1, characterized by , that the step: Generating (110) second language data, further includes at least the following substep: a. Generate (114) at least one query term (30) for the second large-language model (28) from the similarity search (32) with at least the text data . [3] Computer-implemented method (100) according to claim 1 or 2, characterized by that the similarity search (32) is carried out using at least one search model (34), preferably a BM25 model, in particular in combination with a ranking model (36). [4] Computer-implemented method (100) according to any one of the preceding claims, characterized by , that the database (38) is designed as a vector database. [5] Computer-implemented method (100) according to any one of the preceding claims, characterized by , that the step Generate (110) of second language data, further comprises at least the following substeps: a. Transmitting (116) result data sets from the similarity search (32) to the second large-language model (28); and b. Generating (118) at least one second-type output data set as the basis for the second language data set using the second large-language model (28) based on the result data sets. [6] Computer-implemented method (100) according to any one of the preceding claims, characterized by , that the first language data and / or the second language data are generated using a text-to-speech model (42). [7] Computer-implemented method (100) according to any of the preceding claims, wherein the first large-language model (24) is designed as a Llama 3 model and the second large-language model (28) is designed as a GPT4 model. [8] Computer program product comprising instructions which, when the program is executed by a computer (10), cause it to perform the steps of the method (100) according to any one of claims 1 to 7. [9] Vehicle (50) comprising at least one computer (10) configured to perform the steps of the method (100) according to any one of claims 1 to 7.

Images