System, method and computer program product for generating a control command for a machine, and machine comprising a system for generating a control command

Abstract

The invention relates to a system, method and computer program product for generating a control command for vehicles or machines, as well as a vehicle and a machine comprising a system for generating a control command. The system according to the invention comprises an input unit (1), an output unit (2), and a command generation unit (3), which includes a small language model (4) with context memory (5) and a stabilization unit (6). The input unit receives information and sends it in text form to the command generation unit to generate a control command. Depending on the progress of the control command generation, the stabilization unit (6) generates a prompt as input for the small language model (4), which contains an incomplete control command or permissible texts that the language model (4) may generate as output text, and / or adjusts the contents of the context memory (5). The small language model (4) receives a prompt from the stabilization unit and generates a text for a control command in a machine-readable format as output. The vehicle or machine according to the invention comprises a system according to the invention. The method according to the invention comprises the steps for its execution on the system according to the invention. The computer program product according to the invention comprises computer-readable instructions for executing the method according to the invention on one or more computing units.

Description

[0001] The invention relates to a system, method, and computer program product for generating a control command for a machine and a machine comprising a system for generating a control command.

[0002] Systems, methods, and computer program products for generating a control command for a machine as well as machines comprising a system for generating a control command are known from the prior art.

[0003] The system according to the invention has the features of claim 1, the machine according to the invention has the features of claim 7, the method according to the invention has the features of claim 8, and the computer program product according to the invention has the features of claim 10. The dependent claims describe advantageous embodiments.

[0004] A first aspect of the invention relates to a system for generating a control command in text form in machine-readable format for a machine. The system according to the invention comprises an input unit, an output unit, and a command generation unit. The command generation unit comprises a small language model with context memory and a stabilization unit.

[0005] The input unit is configured to receive information and send it in text form to the command generation unit. The command generation unit is configured to receive information in text form from the input unit, and based on this, generate a control command in text form in machine-readable format for a vehicle or machine and send it to the output unit. The small language model is configured to receive a prompt in text form as input and generate a text for a control command in machine-readable format as output. The stabilization unit is configured to provide the small language model with an incomplete control command in machine-readable format depending on the progress of the control command generation.In addition, the stabilization unit can be configured to provide the small language model with permissible texts that may be generated as output text, depending on the progress of the generation of a control command by the language model. In addition, the stabilization unit can be configured to adapt the content of the context memory of the small language model depending on the progress of the generation of a control command.

[0006] Machine means, for example, a machine for the production and processing of objects or materials, a transport machine and a vehicle, such as a road, rail, water or aircraft.

[0007] Machine-readable format means a format that has been developed so that information in this format can be electronically read and easily processed by a computing device and a computer program running on it.

[0008] In an advantageous embodiment of the system according to the invention, the machine-readable format is an easily readable text form for data exchange between applications, such as the JavaScript Object Notation (JSON).

[0009] In a further advantageous embodiment, the permissible words or texts are determined depending on the progress of generating a control command using a formal grammar. Such a grammar can be described, for example, using the Backus-Naur form.

[0010] In a further advantageous embodiment of the system according to the invention, the stabilization unit is configured to read probabilities for predicted tokens determined by the small language model and compare them with a predefined threshold. Furthermore, the stabilization unit is configured to classify a text received from the input unit as irrelevant for generating a control command if all determined probabilities are below the predefined threshold.

[0011] In a further embodiment of the system according to the invention, the stabilization unit is configured to read an inner embedding vector generated by the language model and determine its distance from a predefined target space. Furthermore, the stabilization unit is configured to classify a text received from the input unit as relevant or irrelevant for the generation of a control command, depending on the progress of the generation of a control command and the determined distance.

[0012] In a further advantageous embodiment of the system according to the invention, the input unit comprises a system for analyzing sensor data and describing it in text form. This can be, for example, a speech recognition system for converting spoken language into text or an image description system for analyzing and describing an image in text form. Furthermore, the input unit can be configured to receive information in text form and send it to the command generation unit.

[0013] A second aspect of the invention relates to a machine comprising a system according to the first aspect of the invention. The machine is configured to execute the small language model with context memory and the stabilization unit essentially locally, i.e., on one or more computing units within the machine.

[0014] A third aspect of the invention relates to a method for generating a control command for a machine in machine-readable text format using an input unit and a command generation unit. The command generation unit comprises a small language model with a context memory and a stabilization unit. The machine-readable format of the control command can be, for example, JavaScript Object Notation (JSON). A machine can, in particular, be a vehicle such as a road, rail, water, or aircraft.

[0015] The method according to the invention comprises the steps of receiving information by the input unit, sending the received information by the input unit to the command generation unit in text form and generating a control command in machine-readable format in text form.

[0016] The step of generating a control command in machine-readable format in text form may comprise providing an incomplete control command in machine-readable format in text form to the small language model depending on the progress of the generation of a control command.

[0017] This step can also include the stabilization unit providing the small language model with permissible texts, depending on the progress of generating a control command, that may be generated by the language model as output text. This can be done, for example, using a formal grammar, which can be described, for example, using the Backus-Naur form.

[0018] Furthermore, this step may comprise the stabilization unit accessing the content of the context memory of the small language model and adapting the content of the context memory depending on the progress of the generation of a control command.

[0019] In an advantageous embodiment of the method according to the invention, the stabilization unit reads probabilities for predicted tokens determined by the small language model and compares these probabilities with a predefined threshold. If all probabilities are below the predefined threshold, the stabilization unit classifies the information received as text from the input unit as irrelevant for generating a control command.

[0020] In a further advantageous embodiment of the method according to the invention, the stabilization unit can read an inner embedding vector generated by the small language model and determine the distance of this inner embedding vector to a predefined target space. Depending on the progress of generating a control command and the determined distance, the stabilization unit can classify the information received as text from the input unit as relevant or irrelevant for generating a control command.

[0021] A fourth aspect of the invention relates to a computer program product. The computer program product according to the invention comprises computer-readable instructions which, when executed by one or more data processing units, cause the one or more data processing units to execute an inventive method according to the third aspect of the invention.

[0022] The invention can help stabilize small language models, in particular, by preventing the context of the generated text from quickly being ignored and the generated text from referring to a different topic. Thus, the invention can enable the use of small language models that place lower demands on computing power and thus hardware. Furthermore, the invention can help reduce the effort required to send user data to non-local computing units, especially from providers of large language models.

[0023] Furthermore, the invention can help to enable the fine-tuning of models specifically for the desired application case, since the number of parameters of a small language model is much smaller than the number of parameters of large language models.

[0024] In the following, exemplary embodiments according to the invention are explained in more detail. Fig. 1 a graphic representation on a touch-sensitive screen as a user interface for operating a vehicle; Fig. 2 shows an example of an architecture of a system according to the invention; Fig. 3 shows an example of a method according to the invention as a flow chart; Fig. 4 shows an example of a method according to the invention with steps according to the invention for stabilizing a small language model as a flow chart; Fig. 5 an example of an analysis of probabilities for predicted tokens and Fig. 6 an example of the changes in the position of internal embedding vectors during text generation.

[0025] Interaction between humans and machines, including vehicles, can be difficult, often because the human-machine interface (HMI) doesn't use natural language. Machines require precise information to function efficiently, while humans can cope with less precise statements. The latter arises from humans' ability to enrich given information with their wealth of experience and knowledge of previous events, allowing them to derive a complete picture from incomplete information.

[0026] In artificial intelligence, a model structure has been developed to replicate such human behavior. Large language models based on this structure are called transformers, such as models of the generative, pre-trained transformer (GPT) family.

[0027] Such large language models comprise a very large number of parameters, currently more than 1.8 trillion, which must be calculated in order to work with the model. This can currently only be done efficiently in data centers using specialized hardware. The following exemplary embodiment of the method according to the invention will explain how, in accordance with the teachings of the invention, small language models, which are, for example, 1600 times smaller than large language models, can be used efficiently and reliably, in particular to simplify human-machine communication or to generate control commands in machine-readable format based on information in text form.

[0028] Large language models for text generation are models that, according to current knowledge, have more than 1*10 12Parameters and, in particular, cannot be executed exclusively locally on common hardware. Rather, they usually require an external connection for execution on a server or specialized hardware, such as a large graphics card with a lot of memory or specialized hardware such as the current NVIDIA A100 or NVIDIA H100. A current example of a large language model for text generation is ChatGPT-4 with an estimated 1.8*10 12 parameters.

[0029] A small language model refers to a language model for text generation that has at least 1500 times fewer parameters than current large language models and can be executed locally on hardware commonly used in vehicles and machines. A current example of a small language model for text generation is TinyLlama-1 with approximately 1.1*10 9 parameters.

[0030] Language models for processing human language in text form work with text and word fragments that are comparable to the syllables of a word and are called "tokens." The task of the language model is to predict the next token in a sequence of tokens.

[0031] Fig. Figure 1 shows a user interface of a touchscreen for displaying vehicle information and operating a vehicle. Similar user interfaces are used to control subsystems of trucks, cars, buses, trains, aircraft, ships, or machinery. To improve efficiency or safety, it is often necessary to keep the operator's hands free for other, parallel tasks. Voice control can contribute to this solution, provided it works quickly and reliably.

[0032] Fig. 2 shows an embodiment of a system according to the invention and Fig. 3 of a method according to the invention for the rapid and reliable generation of a control command in machine-readable format in text form. Fig. 2 or the system shown in Fig. The methods shown in Figure 3 use a speech recognition system, such as the current Android Automotive OS (AAOS), as input unit 1, which translates spoken language into written text. A command generation unit 3, which comprises a small language model 4 with context memory 5 and a stabilization unit 6, analyzes the written texts and creates formal descriptions of control commands in text form. These are transmitted to an output unit 2 for controlling the machine. During the generation of a control command, texts, formal descriptions, and instructions are exchanged between the models and applications of the system via a Data Distribution Service (DDS) network.

[0033] A small language model 4 for a system according to the invention or for executing a method according to the invention comprises, for example, 1.1 billion parameters, such as TinyLlama-1. The model data can take up approximately 650 megabytes of space on a hard disk. On a currently standard CPU, such as an Intel NUC 10 i7-10710 without hardware acceleration (GPU), a generation throughput of approximately 40 tokens per second can thus be achieved. A computer program product according to the invention can be written, for example, in C++. A library such as Ilama.cpp with an activated internal API interface can be used as a library for connecting and evaluating a small language model 4 in order to access additional functions for manipulating the language model 4 and its context memory 5.

[0034] The Fig. 2 shown example of a system according to the invention or the one in Fig. The method shown in Figure 3 is suitable for converting a natural language text into a machine-readable control command in text form. The integrated small language model 4 is requested to perform an operation on a given text and generate an output in the format of a formal description language. In the example described here, this is the JSON (JavaScript Object Notation) format, which can be easily processed by a subsequent application.

[0035] Generating consistent JSON output is difficult with small language models, as they are prone to generating incorrect tokens, which subsequently lead to further errors. This will be illustrated with a simplified example:

[0036] System prompt: Read the following instruction from the user and convert it to JSON.

[0037] User instruction: Change the clock color to red.

[0038] Typical JSON output generated by a small language model: { "Operation": "changeColor", "Clock": {"Type": "Clock", "ID": "1234567890", "Name": "My Watch", "Model Name": "Dual LED", ...}

[0039] In the example shown, the small language model begins the generation process, starting with the first curly brace "{." The small language model recognizes that there is a "changeColor" statement and that it is a "clock." After that, the small language model begins to generate erroneous output, i.e., generating erroneous and potentially nonexistent properties, such as a name or model name of a clock.

[0040] In order to counteract this behavior and to help to make the output always usable, the system according to the invention or the method according to the invention applies in step S30, as exemplified in Fig. 4, special methods for stabilizing the small language model are used. These will be explained below with the help of examples.

[0041] The local execution of a small language model 4 enables fast, fine-grained access to the model's internal processes during text generation. According to the invention, this is used to allow a stabilization unit 6 according to the invention, in a step 30a of the method according to the invention, to record the state of the command generation and the language model 4 and to make everything that is known with certainty at a particular point in time during command generation directly available to the language model. This stabilizes and reduces the effort required by the language model 4 for evaluations and for predicting the next token.

[0042] In the example of generating a command in JSON format, it is known from the outset that the output will be a short text in JSON format. Accordingly, a structured string with some keys and key values ​​is expected. In particular, the names of the keys are also specified in advance. Therefore, the stabilization unit 6 first generates the beginning of a statement in JSON format, which must begin with a curly brace and contain "object" as the first key, and makes this available to the language model 4 as part of the output text to be generated: {Object: "

[0043] This particularly reduces the number of evaluations and predictions by the small language model 4. In order to also improve the prediction of key values, the stabilization unit 6, in a step 30b of the method according to the invention, utilizes the fact that only previously known keys and key values, such as objects, operators, values, and parameters, are suitable for controlling a machine. According to the invention, the texts permissible during command generation are made available by the stabilization unit 6 to the small language model 4. In the example described here, a formal grammar in Backus-Naur form is used to provide the small language model 4 with a list of alternative words that are permissible as key values ​​for the word "object": Clock | Speedometer | Distance | Gear selection | Driving mode

[0044] The small language model 4 applies this list of words to the probabilities of the predicted next tokens. All invalid tokens are set to a probability of zero, thus restricting the possibilities for text generation and ensuring that only valid words are generated.

[0045] In the example shown here, language model 4 generates the word “clock” based on the prompt with the incomplete command in JSON format, the list of permissible words and the text received from input unit 1 with the user instruction.

[0046] Stabilization unit 6 then stops the text generation by the small language model 4 immediately after the generation of the word" and supplements the command again with the now known additional elements and generates, for example, the following incomplete command text: {Object: "Clock", Operation: "

[0047] The stabilization unit 6 inserts this command text into the small language model 4 and restarts the command generation by the small language model 4.

[0048] In the example shown, the stabilization unit 6 can additionally access the context memory 5 of the language model 4 before restarting the language model 4, in a step 30c of the method according to the invention.

[0049] A context memory refers to the memory of a language model, which includes all tokens that are in the attention area of ​​the language model and some additional information, such as the keys and values ​​of all past tokens.

[0050] By accessing the context memory 5 and deleting and manipulating tokens, keys, and values, the stabilization unit 6 can narrow down the prediction possibilities of the language model 4 and improve the efficiency of the predictions. In particular, this allows the stabilization unit 6 to optimize the question order for a given use case or for a given small language model 4.

[0051] For example, in the above example, it may be advantageous for the stabilization unit 6 to directly query the language model 4 for the property to be changed for the "Clock" object, rather than for an operation to be applied to the "Clock" object. In such a case, the stabilization unit 6 provides the language model 4 with the following incomplete command as part of the output text to be generated: {Object: "Clock", Property: "

[0052] Based on the incomplete command and the text of the user instruction, the language model 4 completes, for example, the word "color." The stabilization unit 6 then saves the color property and modifies the context memory 5 of the small language model 4 so that the content corresponds to a different or "earlier" state of the command generation. The stabilization unit 6 then generates the following new incomplete control command in JSON format and inserts it into the small language model 4: {Object: "Clock", Operation: "changeColor", Color: "

[0053] The steps S30a, S30b and S30c described by way of example are repeated by the stabilization unit 6 and the language model 4 until all desired keys of the JSON structure have been provided by the model and, for example, the following command has been generated: {Object: "Clock", Operation: "changeColor", Color: "red"}

[0054] In the above command, only the words clock, changeColor, and red, or only clock and red, were generated by language model 4. The final quotation mark and the closing parenthesis are also generated by stabilization unit 6, since no further information was needed and the syntax of a command in JSON format was known.

[0055] The procedure described above has the further advantage that parsing of the generated JSON command by the output unit 2 or an application for command execution is not necessary, since when the text is generated it is known for which key which text or key value is generated and the generated text can be immediately stored in the corresponding variables upon transfer.

[0056] To enable the small language model 4 to recognize that a text does not contain a user instruction but rather has a different topic as its content, the stabilization unit 6, in a step S30d of the inventive method, analyzes the probabilities of the tokens considered for generating the next token by the small language model 4. The probability of the token with the highest probability, whose probability was not set to zero by a grammar in step S30b due to its inadmissibility, is compared with a predefined threshold. If the probability is below the predefined threshold, the stabilization unit 6 determines that the received text does not contain a user instruction but rather has another content that is not relevant for controlling the machine.

[0057] This is to be achieved by the Fig. The example shown in Figure 5 will be explained in more detail. Fig. Figure 5 shows the calculated probabilities of language model 4 for the sentences "Show the speed in green." in English "Show the speed in green." and "The weather is nice." in English "The weather is nice." The former is an input that is relevant, for example, for a display on a dashboard, while the latter is irrelevant. In the first case, the tokens "green" or "speed" are permitted after being checked using a grammar in step S30b, i.e., are not set to zero. They achieve a high probability and therefore very likely contain a user instruction for controlling the given machine, in the example, the dashboard of a vehicle. In the second case, the first token that was not set to zero after being checked for admissibility using a grammar in step S30b is the single letter "s" (for "speedometer").In the example, it only achieves a (normalized) probability of 0.09%, which is below a previously defined threshold. In this case, the stabilization unit 6 recognizes that the user's input text did not contain a control instruction and should not be used to generate a control command.

[0058] Furthermore, according to the invention, the state of the locally executed small language model 4 itself can be examined to decide whether information received from the input unit 1 is relevant for generating a control command or not. According to the invention, the internal embedding vector can be read and stored by the stabilization unit 6 in step S30e of the method according to the invention.

[0059] An internal embedding vector refers to the vector processed by a language model's decoder to generate an output. The internal embedding vector is typically processed by the attention heads and feedforward network of the decoder's layers, the results are added to the internal embedding vector, and the internal embedding vector is (re)normalized. Such an internal embedding vector points to a region in a high-dimensional space. By analyzing the target region of the internal embedding vector, the topic a language model is "thinking about" can be determined. In particular, by reducing the dimensionality, for example, with PCA, t-SNE, or similar algorithms, the position of an internal embedding vector and its change during text generation can be reduced to two dimensions and represented graphically accordingly.

[0060] Fig. Fig. 6 shows a graphical representation of an example in which the internal embedding vector for analyzing the internal state of the small language model 4 is read out and analyzed by the stabilization unit 6 in a step S30e. In the example shown, the Fig.6 shows that, for example, all instructions related to a change in the dashboard end up in similar target regions, while unrelated expressions are positioned significantly differently. To determine whether information received by the input unit 1 is relevant for generating a control command for a given function of a machine, a target region of interest is defined according to the invention. Depending on the distance of the position of the internal embedding vector and the progress of the command generation, the stabilization unit 6 decides whether the received information in text form should be considered for further processing and generation of a control command, or whether processing should not be continued.

Claims

[1] System for generating a control command in text form in machine-readable format for a machine, comprising an input unit (1), an output unit (2) and a command generation unit (3) which comprises a small language model (4) with context memory (5) and a stabilization unit (6), wherein the input unit (1) is configured to receive information and send it in text form to the command generation unit (3), and the command generation unit (3) is configured to receive information in text form from the input unit (1), to generate a control command in text form in machine-readable format based thereon and to send it to the output unit (2), wherein the small language model (4) is configured to receive a prompt in text form as input and to generate a text for a control command in machine-readable format as output, and the stabilization unit (6) is configured, depending on the progress of the generation of a control command, to provide the small language model (4) with an incomplete control command in machine-readable format and / or permissible texts which may be generated by the small language model (4) as output text, and / or to adapt the content of the context memory (5) of the small language model (4). [2] The system of claim 1, wherein the machine-readable format is an easily readable text form for data exchange between applications, such as JavaScript Object Notation (JSON). [3] System according to one of the preceding claims, wherein the determination of the texts permissible depending on the progress of the generation of a control command is carried out by means of a formal grammar which can be described, for example, by means of the Backus-Naur form. [4] System according to one of the preceding claims, wherein the stabilization unit (6) is arranged to read probabilities for predicted tokens determined by the language model (4), to compare them with a predefined threshold and to classify information received from the input unit (1) as not relevant if all determined probabilities are below the predefined threshold. [5] System according to one of the preceding claims, wherein the stabilization unit (6) is arranged to read an inner embedding vector generated by the language model (4), to determine its distance to a predefined target space and, depending on the progress of the generation of a control command and the determined distance, to classify information received from the input unit (1) as relevant or not relevant. [6] System according to one of the preceding claims, wherein the input unit (1) comprises a system for analyzing sensor data and describing it in text form, such as a speech recognition system for converting spoken language into text or an image description system for describing an image as text, and / or for receiving information in text form. [7] A machine comprising a system according to any one of the preceding claims, wherein the machine is arranged to execute the small language model (4) with context memory (5) and the stabilization unit (6) locally on one or more computing units within the machine. [8] Method for generating, by means of an input unit (1) and a command generation unit (3), which comprises a small language model (4) with a context memory (5) and a stabilization unit (6), a control command for a machine in text form in machine-readable format, the method comprising the following steps: (S10), receiving information by the input unit (1); (S20), sending the received information by the input unit (1) to the command generation unit (3) in text form; (S30), generating a control command as text in machine-readable format by performing one or more of the following steps by the stabilization unit (6): (S30a), depending on the progress of the generation of a control command, providing an incomplete control command in machine-readable format for the small language model (4); (S30b), depending on the progress of the generation of a control command, providing for the small language model (4) permissible texts which may be generated by the language model (4) as output text, for example by means of a formal grammar which can be described for example by means of the Backus-Naur form; (S30c), depending on the progress of the generation of a control command, adapting the content of the context memory (5) of the small language model (4). [9] Method according to the preceding claim, wherein in the step of generating a control command (S30) one of the following steps is carried out by the stabilization unit (6): (S30d), reading the probabilities for predicted tokens determined by the language model (4), Comparing the read probability with a predefined threshold and Classifying information received by the input unit (1) as not relevant for generating a control command if all determined probabilities are below the predefined threshold; (S30e), reading an inner embedding vector generated by the small language model (4), Determining the distance of the inner embedding vector to a predefined target space and depending on the progress of the generation of a control command and the determined distance, classifying information received from the input unit (1) as relevant or not relevant for the generation of a control command. [10] A computer program product comprising computer-readable instructions which, when executed by one or more data processing units, cause the one or more data processing units to perform a method according to claim 8 or 9.

Images