System, method, and computer program product for generating a control instruction for a machine, as well as a machine comprising a system for generating a control instruction

Abstract

The invention relates to a system, method and computer program product for generating a control command for vehicles or machines, as well as a vehicle and a machine comprising a system for generating a control command. The system according to the invention comprises an input unit (1), an output unit (2), and a command generation unit (3), which includes a small language model (4) with context memory (5) and a stabilization unit (6). The input unit receives information and sends it in text form to the command generation unit to generate a control command. Depending on the progress of the control command generation, the stabilization unit (6) generates a prompt as input for the small language model (4), which contains an incomplete control command or permissible texts that the language model (4) may generate as output text, and / or adjusts the contents of the context memory (5). The small language model (4) receives a prompt from the stabilization unit and generates a text for a control command in a machine-readable format as output. The vehicle or machine according to the invention comprises a system according to the invention. The method according to the invention comprises the steps for its execution on the system according to the invention. The computer program product according to the invention comprises computer-readable instructions for executing the method according to the invention on one or more computing units.

Description

[0001] The invention relates to a system, method, and computer program product for generating a control command for a machine, as well as a machine comprising a system for generating a control command.

[0002] Systems, methods, and computer program products for generating a control command for a machine, as well as machines comprising a system for generating a control command, are known from the prior art.

[0003] The system according to the invention has the features of claim 1, the machine according to the invention has the features of claim 7, the method according to the invention has the features of claim 8, and the computer program product according to the invention has the features of claim 10. The dependent claims describe advantageous embodiments.

[0004] A first aspect of the invention relates to a system for generating a control command in text form in a machine-readable format for a machine. The system according to the invention comprises an input unit, an output unit, and a command generation unit. The command generation unit includes a small language model with context memory and a stabilization unit.

[0005] The input unit is configured to receive information and send it in text form to the command generation unit. The command generation unit is configured to receive information in text form from the input unit, generate a control command in machine-readable format for a vehicle or machine based on this information, and send it to the output unit. The small language model is configured to receive a text prompt as input and generate a control command text in machine-readable format as output. The stabilization unit is configured to provide the small language model with an incomplete control command in machine-readable format, depending on the progress of the control command generation.Furthermore, the stabilization unit can be configured to provide the small language model with permissible texts that may be generated as output text, based on the progress of the generation of a control command by the language model. Additionally, the stabilization unit can be configured to adjust the contents of the small language model's context memory depending on the progress of the control command generation.

[0006] Machine refers, for example, to a machine for the manufacture and processing of objects or substances, a transport machine, and a vehicle, such as a road, rail, water, or air vehicle.

[0007] Machine-readable format refers to a format that has been developed so that information in that format can be electronically read and easily processed by a computing unit and a computer program running on it.

[0008] In an advantageous embodiment of the system according to the invention, the machine-readable format is an easily readable text format for data exchange between applications, such as JavaScript Object Notation (JSON).

[0009] In a further advantageous embodiment, the permissible words or texts, depending on the progress of generating a control command, are determined by means of a formal grammar. Such a grammar can, for example, be described using the Backus-Naur Form.

[0010] In a further advantageous embodiment of the system according to the invention, the stabilization unit is configured to read probabilities for predicted tokens determined by the small language model and to compare them with a predefined threshold. Furthermore, the stabilization unit is configured to classify text received from the input unit as irrelevant for generating a control command if all determined probabilities are below the predefined threshold.

[0011] In a further embodiment of the system according to the invention, the stabilization unit is configured to read an inner embedding vector generated by the language model and to determine its distance to a predefined target space. Furthermore, the stabilization unit is configured to classify text received from the input unit as relevant or irrelevant for generating a control command, depending on the progress of the generation of a control command and the determined distance.

[0012] In a further advantageous embodiment of the system according to the invention, the input unit comprises a system for analyzing sensor data and describing it in text form. This can be, for example, a speech recognition system for converting spoken language into text or an image description system for analyzing and describing an image in text form. Furthermore, the input unit can be configured to receive information in text form and send it to the command generation unit.

[0013] A second aspect of the invention relates to a machine comprising a system according to the first aspect of the invention. The machine is configured to execute the small language model with context memory and the stabilization unit essentially locally, i.e., on one or more computing units within the machine.

[0014] A third aspect of the invention relates to a method for generating a control command for a machine in machine-readable text format using an input unit and a command generation unit. The command generation unit comprises a small language model with a context memory and a stabilization unit. The machine-readable format of the control command can be, for example, JavaScript Object Notation (JSON). A machine can be, in particular, a vehicle such as a road, rail, water, or air vehicle.

[0015] The method according to the invention comprises the steps of receiving information by the input unit, sending the received information by the input unit to the instruction generation unit in text form, and generating a control instruction in machine-readable text format.

[0016] The step of generating a control command in machine-readable text format can include providing an incomplete control command in machine-readable text format to the small language model, depending on the progress of generating a control command.

[0017] This step can also include the stabilization unit providing the small language model with permissible texts, depending on the progress of generating a control command. These texts may then be generated by the language model as output text. This can be done, for example, using a formal grammar, such as the Backus-Naur Form.

[0018] Furthermore, this step can include the stabilization unit accessing the contents of the context memory of the small language model and adapting the contents of the context memory depending on the progress of generating a control command.

[0019] In an advantageous embodiment of the method according to the invention, the stabilization unit reads certain probabilities for predicted tokens from the small language model and compares these probabilities with a predefined threshold. If all probabilities are below the predefined threshold, the stabilization unit classifies the information received as text from the input unit as not relevant for generating a control command.

[0020] In a further advantageous embodiment of the method according to the invention, the stabilization unit can read an inner embedding vector generated by the small language model and determine the distance of this inner embedding vector to a predefined target space. Depending on the progress of the generation of a control command and the determined distance, the stabilization unit can classify the information received as text from the input unit as irrelevant or relevant for the generation of a control command.

[0021] A fourth aspect of the invention relates to a computer program product. The computer program product according to the invention comprises computer-readable instructions which, when executed by one or more data processing units, cause one or more data processing units to execute a method according to the third aspect of the invention.

[0022] The invention can help stabilize small language models, in particular by preventing the context of the generated text from being quickly disregarded and the generated text from addressing a different topic. Thus, the invention can enable the use of small language models that place lower demands on computing power and therefore hardware. Furthermore, the invention can help reduce the effort required to send user data to non-local computing units, especially those provided by large language model vendors.

[0023] Furthermore, the invention can help to enable the fine-tuning of models specifically for the desired application, since the number of parameters of a small language model is much smaller than the number of parameters of large language models.

[0024] The following are further explanations of exemplary embodiments of the invention. Fig. 1. A graphical representation on a touch-sensitive screen as a user interface for operating a vehicle; Fig. 2 an example of an architecture of a system according to the invention; Fig. 3 an example of a method according to the invention as a flowchart; Fig. 4 an example of a method according to the invention with steps according to the invention for stabilizing a small language model as a flowchart; Fig. 5. An example of an analysis of probabilities for predicted tokens and Fig. 6. An example of the changes in the position of internal embedding vectors during text generation.

[0025] The interaction between humans and machines, including vehicles, can be difficult, often because the human-machine interface (HMI) does not use natural language. Machines require precise information to operate efficiently, while humans can cope with less precise statements. This latter ability stems from humans' capacity to enrich given information with their experience and knowledge of past events, allowing them to derive a complete picture from incomplete information.

[0026] In artificial intelligence, a model structure has been developed to replicate such human behavior. Large language models based on this structure are called transformers, such as models of the generative pretrained transformer family (GPT).

[0027] Such large language models comprise a very large number of parameters, currently, for example, more than 1.8 trillion parameters, which must be calculated in order to work with the model. This can currently only be done efficiently in data centers on specialized hardware. The following embodiment of the method according to the invention will explain how, according to the teachings of the invention, small language models, which are, for example, 1600 times smaller than large language models, can be used efficiently and reliably, in particular to simplify human-machine communication or to generate control commands in machine-readable text format based on information in text form.

[0028] Large language models for text generation are defined as models that, according to current knowledge, have more than 1*10 12They use parameters and, in particular, cannot be executed exclusively locally on standard hardware. Rather, they typically require an external connection for execution on a server or specialized hardware, such as a large graphics card with ample memory or specialized hardware like the current NVIDIA A100 or NVIDIA H100. A current example of a large language model for text generation is ChatGPT-4, with an estimated size of 1.8 x 10⁻⁶. 12 Parameters.

[0029] A small language model refers to language models for text generation that have at least 1500 times fewer parameters than current large language models and that can be executed locally on hardware commonly found in vehicles and machines. A current example of a small language model for text generation is TinyLlama-1, with approximately 1.1 * 10 9 Parameters.

[0030] Language models for processing human language in text form work with text and word fragments that are comparable to the syllables of a word and are called "tokens." The task of the language model is to predict the next token in a sequence of tokens.

[0031] Fig. Figure 1 shows a user interface of a touchscreen for displaying vehicle information and operating a vehicle. Similar user interfaces are used to control subsystems of trucks, cars, buses, trains, airplanes, ships, or machines. To improve efficiency or safety, it is often necessary to keep the operator's hands free for other, parallel tasks. Voice control can contribute to this solution, provided it operates quickly and reliably.

[0032] Fig. Figure 2 shows an embodiment of a system according to the invention and Fig. 3 of a method according to the invention for the rapid and reliable generation of a control command in machine-readable text format. The method described in Fig. 2 system shown or the one in Fig. The three methods shown utilize a speech recognition system, such as the current Android Automotive OS (AAOS), as an input unit 1, which translates spoken language into written text. A command generation unit 3, comprising a small language model 4 with context memory 5 and a stabilization unit 6, analyzes the written text and creates formal textual descriptions of control commands. These are then transmitted to an output unit 2 for controlling the machine. During the generation of a control command, texts, formal descriptions, and instructions are exchanged between the system's models and applications via a Data Distribution Service (DDS) network.

[0033] A small language model 4 for a system according to the invention, or for carrying out a method according to the invention, comprises, for example, 1.1 billion parameters, as is currently the case with TinyLlama-1. The model data can occupy a size of approximately 650 megabytes on a hard drive. On a currently standard CPU, such as the Intel NUC 10 i7-10710 without hardware acceleration (GPU), a generation throughput of approximately 40 tokens per second can be achieved. A computer program product according to the invention can, for example, be written in C++. A library such as the currently available llama.cpp with an activated internal API interface can be used as a library for connecting to and evaluating a small language model 4, in order to access additional functions for manipulating the language model 4 and its context memory 5.

[0034] The in Fig. 2 Example of a system according to the invention shown, or the one shown in Fig. The method shown in Figure 3 is suitable for converting natural language text into a machine-readable control command in text form. The integrated small language model (Figure 4) is instructed to perform an operation on a given text and generate output in the format of a formal description language. In the example described here, this is the JSON format (JavaScript Object Notation), which can be easily processed by a subsequent application.

[0035] Generating consistent JSON output is difficult with small language models because they are prone to generating incorrect tokens, which in turn leads to further errors. This can be illustrated with a simplified example: System prompt: Read the following user instruction and convert it to JSON. User instruction: Change the color of the clock to red. Typical JSON output generated by a small language model: { "Operation": "changeColor", "Clock": {"Type": "Clock", "ID": "1234567890", "Name": "My Clock", "Model Name": "Dual LED", ...}

[0036] In the example shown, the small language model begins generating code, starting with the first curly brace "{". The small language model recognizes that there is a "changeColor" instruction and that it is a "clock". Afterward, the small language model begins generating erroneous output, i.e., it generates incorrect and potentially non-existent properties, such as a name or model designation for a clock.

[0037] To counteract this behavior and to help ensure that the output is always usable, the system or method according to the invention applies in step S30, as exemplified in Fig. Figure 4 presents specific methods for stabilizing the small language model. These will be explained below with the help of examples.

[0038] The local execution of a small language model 4 enables fast, granular access to the model's internal processes during text generation. According to the invention, this is utilized by a stabilization unit 6, in step 30a of the inventive method, to detect the state of the command generation and the language model 4, and to directly provide the language model with everything that is known with certainty at any given time during command generation. This stabilizes and reduces the workload of the language model 4 for evaluations and for predicting the next token.

[0039] In the example of generating a command in JSON format, it is known from the outset that the output will be a short text in JSON format. Accordingly, a structured string with several keys and key values ​​is expected. In particular, the names of the keys are also predefined. Therefore, stabilization unit 6 first generates the beginning of a statement in JSON format, which must begin with curly braces and contain "object" as the first key, and provides this to language model 4 as part of the output text to be generated: {object: "

[0040] This particularly reduces the number of evaluations and predictions by the small language model 4. Furthermore, to improve the prediction of key values, the stabilization unit 6, in step 30b of the inventive method, utilizes the fact that only previously known keys and key values, such as objects, operators, values, and parameters, are suitable for controlling a machine. According to the invention, the texts permissible during command generation are provided to the small language model 4 by the stabilization unit 6. In the example described here, a formal grammar in Backus-Naur form is used to provide the small language model 4 with a list of alternative words that are permissible as key values ​​for the word "object": clock | speedometer | distance | gear selection | driving mode

[0041] The small language model 4 applies this list of words to the probabilities of the predicted next tokens. All invalid tokens are set to a probability of zero, thus limiting the possibilities for text generation and ensuring that only allowed words are produced.

[0042] In the example shown here, language model 4 generates the word "clock" based on the prompt with the incomplete command in JSON format, the list of allowed words and the text received from input unit 1 with the user instruction.

[0043] Subsequently, stabilization unit 6 stops the text generation by the small language model 4 immediately after the word is generated and adds the now known additional elements to the command, generating, for example, the following incomplete command text: {Object: "Clock", Operation: "

[0044] The stabilization unit 6 inserts this command text into the small language model 4 and restarts the command generation by the small language model 4.

[0045] In the example shown, the stabilization unit 6 can additionally access the context memory 5 of the language model 4 before the language model 4 is restarted, in step 30c of the method according to the invention.

[0046] In this context, a context memory refers to the memory of a language model that includes all tokens that are within the attentional area of ​​the language model and some additional information, such as the keys and values ​​of all past tokens.

[0047] By accessing the context memory 5, deleting and manipulating tokens, keys, and values, the stabilization unit 6 can limit the possibilities of predictions by the language model 4 and improve the efficiency of the predictions. In particular, it gives the stabilization unit 6 the ability to optimize the question sequence for a given use case or for a given small language model 4.

[0048] In the example above, it can be advantageous for stabilization unit 6 to query language model 4 directly for the property to be modified for the object "clock", rather than for an operation to be applied to the object "clock". In such a case, stabilization unit 6 provides language model 4 with, for example, the following incomplete command as part of the output text to be generated: {Object: "Clock", Property: "

[0049] Based on the incomplete command and the text of the user instruction, language model 4, for example, adds the word "color". Stabilization unit 6 then stores the property "color", modifies the context memory 5 of small language model 4 so that its content corresponds to a different, or "earlier", state of the command generation. Furthermore, stabilization unit 6 generates the following new incomplete control command in JSON format and inserts it into small language model 4: {Object: "Clock", Operation: "changeColor", Color: "

[0050] The exemplary steps S30a, S30b and S30c are repeated by the stabilization unit 6 and the language model 4 until all desired keys of the JSON structure have been provided by the model and, for example, the following command has been generated: {Object: "Clock", Operation: "changeColor", Color: "red"}

[0051] In the command above, only the words "Uhr" (clock), "changeColor", and "rot" (red), or just "Uhr" and "rot", were generated by language model 4. The final quotation mark and closing parenthesis were also generated by stabilization unit 6, since no further information was required and the syntax of a command in JSON format was known.

[0052] The procedure described above has the further advantage that parsing the generated JSON command by output unit 2 or an application for command execution may not be necessary, since when generating the text it is known for which key which text or key value is generated and the generated text can be immediately stored in the corresponding variables when passed.

[0053] To enable the small language model 4 to recognize that a text does not contain a user instruction but rather a different topic, the stabilization unit 6, in step S30d of the inventive method, analyzes the probabilities of the tokens considered by the small language model 4 for generating the next token. The probability of the token with the highest probability, whose probability was not set to zero in step S30b due to its invalidity by means of a grammar, is compared with a predefined threshold. If the probability is below the predefined threshold, the stabilization unit 6 determines that the received text does not contain a user instruction but rather other content irrelevant for controlling the machine.

[0054] This is to be achieved through the in Fig. The five examples shown will be explained in more detail. Fig. Figure 5 shows the calculated probabilities of language model 4 for the sentences "Show the speed in green." and "The weather is nice." The former is an input relevant, for example, for a display on a dashboard, while the latter is irrelevant. In the first case, the tokens "green" or "speed" are allowed after validation by a grammar in step S30b, meaning they are not set to zero. They achieve a high probability and therefore very likely contain a user instruction for controlling the given machine, in this example, the dashboard of a vehicle. In the second case, the first token that was not set to zero after validation by a grammar in step S30b is the single letter "s" (for "speedometer").In this example, it only achieves a (normalized) probability of 0.09%, which is below a previously defined threshold. In this case, stabilization unit 6 recognizes that the user input text did not contain a control instruction and should not be used to generate a control command.

[0055] Furthermore, according to the invention, the state of the locally executed small language model 4 itself can be examined to decide whether information received from the input unit 1 is relevant for generating a control command or not. According to the invention, the internal embedding vector can be read and stored by the stabilization unit 6 in step S30e of the method according to the invention.

[0056] An internal embedding vector is the vector processed by a language model's decoder to generate output. Typically, the internal embedding vector is processed by the decoder's attention heads and feedforward network layers, the results are added to the internal embedding vector, and the internal embedding vector is (re)normalized. Such an internal embedding vector points to a region in a high-dimensional space. By analyzing the target region of the internal embedding vector, the topic a language model is "thinking" about can be determined. In particular, by reducing the dimensionality, for example with PCA, t-SNE, or similar algorithms, the position of an internal embedding vector and its changes during text generation can be reduced to two dimensions and represented graphically.

[0057] Fig. Figure 6 shows a graphical representation of an example in which the internal embedding vector for analyzing the internal state of the small language model 4 is read and analyzed by the stabilization unit 6 in one step S30e. In the example shown, the Fig.Figure 6 shows that, for example, all instructions relating to a change in the dashboard end up in similar target regions, while unrelated expressions are positioned significantly differently. To determine whether information received by the input unit 1 is relevant for generating a control command for a given function of a machine, a target region of interest is defined according to the invention. Depending on the distance of the position of the internal embedding vector, and depending on the progress of the command generation, the stabilization unit 6 decides whether the received information in text form should be considered for further processing and generation of a control command, or whether the processing should not be continued.

Images