A method of operating an assistance system of a vehicle, assistance system and vehicle

Abstract

A method of operating an assistance system of a vehicle, which converts an input expressed by natural language into a control command, comprising the following steps: In a first step receiving a firs

Description

FIELD OF THE INVENTION

[0001] The present invention relates to the field of vehicles, in particular automobiles, such as vans, cars, or trucks. More specifically, the present invention relates to a method of operating an assistance system of a vehicle, which converts an input expressed by natural language into a control command. Furthermore, the present invention relates to a corresponding assistance system as well as a corresponding vehicle. BACKGROUND INFORMATION

[0002] Modern vehicles are more and more frequently equipped with assistance systems that offer voice command control. A chat bot or a voice assistant is prompted with natural language queries. Natural language is ambiguous and subject of interpretation by nature. Two things are crucial for such a system: firstly, it must be assessed whether the system is able to understand the query and secondly, it must be checked whether the query is unambiguous or whether queries are necessary. SUMMARY OF THE INVENTION

[0003] It is an object of the present invention to provide a method, an assistance system, and a vehicle, which are able to check if the natural language input of a user of the vehicle is unambiguous or not and after deciding that, perform a control command, for example, for controlling the vehicle.

[0004] A method, an assistance system, and a vehicle according to the present invention solve this object. Advantageous embodiments are presented in the dependent claims, the description, and the drawings.

[0005] A first aspect of the present invention relates to a method of operating an assistance system of a vehicle which converts an input expressed by natural language, for example via voice command or via text or chat input, into a control command.

[0006] The method according to the invention comprises the following steps:

[0007] A first step is receiving a first prompt comprising a natural language input from a user.

[0008] The second step is generating a first intermediate representation of the input by an encoder-decoder structure of an autoregressive large language model (LLM).

[0009] A third step is creating or generating a second prompt as output from the first intermediate representation by the large language model or via the large language model.

[0010] A fourth step is generating a second intermediate representation from the output by the encoder-decoder structure.

[0011] A fifth is determining a relationship between the first intermediate representation and the second intermediate representation, whereby a dissimilarity between decoding and encoding may be determined or is determinable.

[0012] A sixth step is depending on the dissimilarity determined in the fifth step, providing a query or executing the control command.

[0013] In other words, it uses the solution of an autoregressive large language model which key property is finding a latent space representation that is useful to semantically operate on text. The large language model uses latent embeddings for the intermediate representations. Therefore, the method allows to arithmetically operate on the embeddings. For example, an embedding may realize “king-male+woman = queen”.

[0014] Therefore, the method generates a latent embedding from a prompted query. From that embedding, it reconstructs the question and generates another latent embedding from the generated question. These two embeddings are now part of the same latent feature space. The closeness or proximity of both vectors may have the following two properties:

[0015] The first property indicates how ambiguous the query is. If the query has only one unique understanding, it will be matched to an embedding from which the original semantics of the query may be generated again. Otherwise, the method will generate a natural language query from the latent embedding or the first intermediate representation with different semantics. After mapping that to the latent space again, the two different embeddings will have a high distance.

[0016] Second property may indicate if the system or method grasp the query. For example, if it found a good latent representation, that matches the natural language query.

[0017] The method may answer the overall question: How well the method will reconstruct the original query from the latent representation? If this may be done will, measured by the distance and latent space of both latent embeddings, it means that the large language model has an informative embedding for the query and that the query tends not to be ambiguous.

[0018] Optionally, noise may be applied to the first latent embedding to better assess how robust the latent embedding is. If it keeps a low distance between the first embedding and the second embedding or between the first intermediate representation and the second intermediate representation, which means that the method has an informative embedding, that is uniquely mapped to the semantics of the original query or input prompted.

[0019] An advantage of the present invention is that the method may ask the questions back to the user only if needed. This makes interaction with the method or the assistance system or the vehicle comprising the assistance system much more efficient and focuses the conversation to the exact information that is additionally needed. A further advantage is that the method provides a precise way to detect, if it is able to grasp the query. This avoids the method providing weird answers. Another advantage is that the large language model may be trained by large corpuses of unlabeled text.

[0020] In an embodiment of the present invention, the first intermediate representation and / or the second intermediate representation are generated as latent embeddings or the large language model to provide the intermediate representations uses latent embeddings.

[0021] In still another embodiment of the present invention, the control command comprises a driving instruction, a query to an electronic computing device, and / or a control of an infotainment system.

[0022] In yet another embodiment of the present invention, noise is additionally used to generate the first intermediate representation and / or the output.

[0023] In another embodiment of the present invention, a distance metric is used to determine the dissimilarity.

[0024] In still another embodiment of the present invention, the dissimilarity is determined by a classification function, which is provided by another neural network (NN).

[0025] A second aspect of the present invention relates to an assistance system of a vehicle, which is configured to generate a control command based on a natural language input by a method according to the first aspect of the present invention and to execute this control command.

[0026] Advantages and advantageous embodiments of the first aspect of the present invention are to be regarded as advantages and advantageous embodiments of the second aspect of the present invention and vice versa.

[0027] A third aspect of the present invention relates to a vehicle comprising an assistance system according to the second aspect of the present invention and / or configured to perform a method according to the first aspect of the present invention.

[0028] Advantages and advantageous embodiments of the first and second aspect of the present invention are to be regarded as advantages and advantageous embodiments of the third aspect of the present invention and vice versa.

[0029] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figure and / or shown in the figure alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0031] The drawings show in:

[0032] Fig. 1 a schematic diagram of a method for operating an assistance system of a vehicle. DETAILED DESCRIPTION

[0033] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0034] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0035] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0036] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0037] Fig. 1 shows a schematic diagram of a method for operating an assistance system. The assistance system is represented by the Fig. 1. The assistance system may be part of a vehicle, which is also represented by Fig. 1. The assistance system converts an input expressed by natural language into a control command.

[0038] The method comprises the following steps:

[0039] A first step is receiving a prompt 10 comprising a natural language input or a natural language query q from a user.

[0040] In a second step, a first intermediate representation e is created or generated as a first latent embedding 12 by an encoder-decoder structure 14 of an autoregressive large language model LLM.

[0041] In a third step a second prompt 16 is created as output from the intermediate representation e by the large language model LLM. The second prompt 16 comprises a natural language query q' generated by the large language model LLM.

[0042] In a fourth step, from the output or second prompt 16, a second intermediate representation e' is generated by the encoder-decoder structure 14 whereby only the encoder half is used.

[0043] In a fifth step, a relationship between the first intermediate representation e and the second intermediate representation e’, an therefore in particular a relationship between the latent embeddings 12, is or are determined, whereby a dissimilarity between decoding and encoding performed by the encoder-decoder structure 14 is also determinable or predictable.

[0044] In a sixth step, depending on the dissimilarity or on the value of the dissimilarity a query is asked to the user or the control command is executed.

[0045] Therefore, in other words, a prompt “Paraphrase” together with a natural language query q is fed into the LLM. The LLM is based on the encoder-decoder structure 14 and generates a latent embedding 12 as intermediate representation e = enc(q).

[0046] The LLM generates a natural language query q' as output q' = dec(e) = dec(enc(q)).

[0047] Again, the prompt “Paraphrase” together with the natural language query q' is fed into the same large language model LLM. The large language model LLM then generates another latent embedding 12 with a second intermediate representation e’ with e’ = enc (q') = enc (dec(e)) = enc(dec(enc(q))). This equation shows the relation between an intermediate representation e and e’ with e’ = enc(dec(e)).

[0048] Now, it is possible to use a dissimilarity function to compare e with e’, essentially checking how well this combination of decoding and encoding is to an identity. Another possibility to evaluate the dissimilarity is to use a distance metric, for example, the L2 norm (Euclidean): D(e, e’) = ||e, e’||2-

[0049] In addition, another neural network (NN) may be used to learn a classification function C(e, e’). If the result of the search similarity function is small, the decoder encoder combination works well, which implies that the model is probably capable of reasoning over this question and the question is unambiguous.

[0050] Therefore, there are at least two dissimilarity evaluation options, option 1 D(e, e’) = ||e, e’||2 and the second option is learning to classify with C(e, e’) via separate neural network.

[0051] One example for question or query q in natural language may be “where did you come from”, which is ambiguous and could mean, “where have I been just before now” or “where did I originate from”. Depending on a large language model LLM, the interpretation may be more towards the first or the later, whereby the later would result in a more distant embedding e’ for the use case of a vehicle.

[0052] So, if the interpretation is the later, the query q' could transform from query q from “where did you come from” to “what is your place of origin”. With the method, the assistance system in the vehicle shown here, a system may be provided for an autoregressive large language model LLM, in particular used in a vehicle, to translate an input expressed by natural language into a correct control command. Reference signs 10 first prompt 12 latent embedding 14 encoder-decoder structure 16 second prompt e first intermediate representation e’ second intermediate representation q q’ LLM first natural language query second natural language query large language model

Claims

1. A method of operating an assistance system of a vehicle, which converts an input expressed by natural language into a control command, comprising the steps:- Receiving a first prompt (10) comprising a natural language input from a user;- Generating a first intermediate representation (e) of the input by an encoderdecoder structure (14) of an autoregressive large language model (LLM);- Generating second prompt (16) as output from the first intermediate representation (e) by the large language model (LLM);- Generating a second intermediate representation (e’) from the output by the encoder-decoder structure (14);- Determining a relationship between the first intermediate representation (e) and the second intermediate representation (e’), whereby a dissimilarity between decoding and encoding is determinable; and- Depending on the dissimilarity, a query is prompted back to the user or the control command is executed.

2. The method according to claim 1,characterized in thatthe first intermediate representation (e) and / or the second intermediate representation (e’) are generated as latent embeddings.

3. The method according to claim 1 or 2,characterized in thatthe control command comprises a driving instruction, a query to an electronic computing device, and / or a control of an infotainment system.

4. The method according to any one of claims 1 to 3, characterized in thatnoise is additionally used to generate the first intermediate representation (e) and / or the output.

5. A method according to any one of the preceding claims, characterized in thata distance metric is used to determine the dissimilarity.

6. A method according to any one of the preceding claims, characterized in thatthe dissimilarity is determined by a classification function, which is provided by a neural network.

7. An assistance system for a vehicle, which is configured to generate a control command based on a natural language input by a method according to any one of the preceding claims and to execute this control command.

8. A vehicle comprising an assistance system according to claim 7 and / or configured to perform a method according to any one of claims 1 to 6.12

Images