Method and apparatus for generating a response to a query formulated in natural language

Abstract

The invention relates to a computer-implemented method for generating a response to a natural-language query regarding a program-like sequence which comprises steps and conditions for advancing, in which - the sequence is converted into a graph with a set of nodes and edges connecting the nodes, - the graph is converted into embedded graph information by means of embedding, - the query is converted into embedded query information by means of embedding, - the embedded graph information and the embedded query information are combined to form common embedded information, - the common embedded information is sent to a large language model, - a result is received from the large language model and is sent as a response.

Description

[0001] Description

[0002] Method and apparatus for generating a response to a query formulated in natural language

[0003] A Large Language Model (LLM) is a language model distinguished by its ability to understand and generate general-purpose language. LLMs acquire these abilities by learning statistical relationships from very large sets of text documents during a computationally intensive self-supervised and semi-supervised training process. LLMs are neural networks that follow a transformer architecture.

[0004] As autoregressive language models, they work by using an input text and repeatedly predicting the next token or word. Until 2020, a so-called fine-tuning process was the only way to adapt a model to perform specific tasks. Larger models can achieve similar results using prompt engineering. It is assumed that they acquire knowledge about the syntax, semantics, and ontology contained in human language corpora, as well as inaccuracies and biases present in those corpora.

[0005] Notable examples include the GPT models from OpenAI (GPT, Generative Pretrained Transformer) e.g. GPT-3.5 and GPT-4, which are used in ChatGPT), PaLM from Google (used in Bard) and LLaMA from Meta, as well as BLOOM, Ernie 3.0 Titan and Claude 2 from Anthropic.

[0006] Recent developments in Large Language Models mean that they are not only used in natural language analysis, but are also being explored in other areas. For example, there is great interest in researching their potential use in the industrial sector.

[0007] One possible application is the analysis of information presented as a program-like sequence. Such a sequence could, for example, be a program for a programmable logic controller (PLC). This program might be structured according to the IEC 61131-3 standard. It could be, for instance, a sequence of control steps and transition conditions. A linear logic generator (LLM) can be used to evaluate and answer a user query about the program in the form of a natural language prompt. Such a user query might be, "Are signals S1 and S2 linked in the program?"

[0008] To analyze this question, the LLM needs information about the program. One common method is to convert the program into a textual description. This can be done, for example, using the program's JSON representation. However, this is not well-suited for analysis by an LLM. Better results are achieved by converting the JSON representation into descriptive, flowing text.

[0009] A disadvantage of this approach is that with more complex programs, the maximum context size of, for example, 32,000 tokens in ChatGPT-4 can be exceeded, meaning the entire program cannot be considered in the request, making a correct response impossible. Furthermore, even with extensive requests that do not exceed the context size, the problem remains that the LLM "forgets" some information, meaning not all information is given equal weight in the response, and processing is either impossible or yields incorrect answers.

[0010] The object of the invention is to provide an improved method for generating a response to a query formulated in natural language concerning a program-like sequence. A further object is to provide a corresponding system.

[0011] This problem is solved by a method having the features specified in claim 1. Another solution consists of the device having the features of claim 13.

[0012] In the computer-implemented method according to the invention for generating a response to a natural language query regarding a program-like sequence comprising steps and transition conditions, the following steps are performed. The sequence is transformed into a graph with a set of nodes and edges connecting the nodes. Furthermore, the graph is transformed into embedded graph information by means of embedding. The query is also transformed into embedded query information by means of embedding. The embedded graph information and the embedded query information are combined into a single embedded piece of information, and this single embedded piece of information is transmitted to a Large Language Model. A result is received from the Large Language Model and used as the response.The device according to the invention for generating a response to a query formulated in natural language regarding a program-like sequence comprising steps and switching conditions, has a user interface for inputting the query. It further comprises a user interface for linking or inputting the program-like sequence. It is also configured to convert the sequence into a graph with a set of nodes and edges connecting the nodes, to convert the graph into embedded graph information by means of embedding, and to convert the query into embedded query information by means of embedding.It is further designed to combine the embedded graph information and the embedded query information into a common embedded information, to send the common embedded information to a Large Language Model, and to receive a result from the Large Language Model and use it as a response.

[0013] Embedding for large language models is a method for representing words, sentences, texts, or other discrete categories as numerical vectors in a high-dimensional space. It is based on the idea that semantically similar words, texts, or categories should be located close to each other in the vector space.

[0014] These embeddings can then be used in various applications, such as text classification, semantic search, or machine translation, to name just a few. By using embeddings, large language models can better handle specific tasks.

[0015] Advantageous embodiments of the invention are described in the dependent claims. The embodiment of the independent claims can be combined with the features of one of the dependent claims or, preferably, with those of several dependent claims. Accordingly, the following additional features can be provided:

[0016] In one embodiment of the invention, the steps and the transition conditions are converted into nodes to transform the sequence into a graph. Separate nodes are created for the steps and transition conditions; in other words, they are not combined into nodes, even if they occur in pairs within the program-like sequence. In a further embodiment of the invention, an edge is inserted into the graph for connected pairs of a step and a transition condition during the conversion of the program-like sequence into a graph.

[0017] In a further embodiment of the invention, to convert the sequence into the graph, a text representation is generated for each step for each action that is part of the step, the text representations of the actions are chained to form a total text for the step, and the total text is used as the content of the node that represents the step.

[0018] In a further embodiment of the invention, a text representation is generated for each switching condition to convert the sequence into the graph, and the text representation is used as the content of the node that represents the switching condition.

[0019] The embedded graph information is preferably transformed into a textual embedding space using a mathematical transformation determined through training.

[0020] Training data can be provided for training the mathematical transformation, with the training data comprising a set of training datasets. A training dataset conveniently includes a graph, a training query for that graph, and a correct training response to the query.

[0021] For a number of training datasets, a response to the training query is determined using the previously described method, and the mathematical transformation is adjusted by comparing the determined response with the training response of the training dataset. Advantageously, no changes to other parameters of the method are permitted; in other words, only the mathematical transformation is modified.

[0022] The embedded graph information is conveniently a vector of real numbers, and the mathematical transformation is conveniently a multiplication of the vector by a matrix, where the matrix is ​​provided / adapted by the described training.

[0023] In a further advantageous embodiment of the invention, one of two alternative approaches is selected and used to generate the response to the query. In the first alternative, the previously described conversion of the graph into embedded graph information by means of embedding, followed by the subsequent steps, is performed. In the second alternative, the sequence is converted into a textual representation and sent to the Large Language Model as natural language text along with the query. A result is then received from the Large Language Model and used as the response.

[0024] Advantageously, the selection between the first and second alternatives is made based on the sequence or the graph. In one embodiment of the invention, the second alternative is selected if the length of the sequence or the size of the graph is less than a threshold value.

[0025] The embedded graph information and the embedded query information are preferably each a list or a vector of real numbers, and the lists are concatenated to generate the common embedded information.

[0026] The invention will now be described and explained in more detail with reference to the exemplary embodiments shown in the figures. The figures show:

[0027] Figure 1 shows an exemplary program,

[0028] Figure 2 schematically shows a flowchart for generating a response to a query.

[0029] Figure 3 shows a graph representation of the program,

[0030] Figure 4 shows an overall procedure for generating the response for different programs 10.

[0031] Figure 1 shows a schematic and pictorial representation of an exemplary program 10. The program 10 comprises a plurality of control steps S1...6. In a real program 10, these control steps contain, for example, control commands for a machine or production line in an industrial environment. Furthermore, the program includes transition conditions W1...5. The transition conditions W1...5 are arranged between each pair of control steps S1...6 and thus implicitly establish a connection between these control steps S1...6. However, control steps can also be connected directly, without a transition condition W1...5. The exemplary program 10 shown serves only for illustration; in real-world problems, a program can comprise more than 100 or even more than 1000 control steps. The transition conditions W1...Each of the 5 conditions contains a condition upon fulfillment of which the program switches from a preceding control step S1 ...6 to the control step S1 ...6 connected via the switching condition Q1 ...5.

[0032] Program 10 is a first input value for the exemplary sequence shown in Figure 2. For this purpose, the program is stored, for example, in the memory of a PC. A second input value is a query 12 in natural language relating to program 10, for example, the query "Are the two signals S1 and S2 linked together in the program?"

[0033] For entering request 12, there is usually a web-based input form that allows text input. It goes without saying that input can be made via PC, but also via other devices such as tablets or smartphones. Request 12 can also be accepted audibly. It can then be converted into a text-based request 12.

[0034] It is expedient not to enter program 10 via an input form, but rather to allow another form of linking. In the present example, the input form for query 12 is part of a development environment for programs of the aforementioned type. A query thus refers to an open program 10 that already exists in the development environment or is under development.

[0035] To process request 12 with program 10, the request is embedded in a form X via an embedding layer 14, which can be part of a Large Language Model 20. q This transforms query 12 into a list of real numbers, which can be understood as a vector. In terms of embedding, this vector allows the calculation of spatial distances that correspond to semantic distances, i.e., distances in the linguistic sense.

[0036] Program 10 is converted into a graph representation 16. Graph representation 16 follows the principle of the usual form of graph representation, G = (V, E), where V is a vector of nodes and E is a vector of edges, with the edges connecting the nodes. Figure 3 shows the graph representation for the example program 10 from Figure 1.

[0037] To convert program 10 into graph representation 16, the control steps S1...6 of program 10 are transformed into nodes Vi, 3, 5, 7, 9, 11 of graph representation 16. The content of each node Vi, 3, 5, 7, 9, 11 is a textual representation of the control actions of the respective control step S1...6 in natural language. For example, node V1 can contain the following information: "This node comprises two steps. In the first step, the variable xO is set to the value 1. The variable xO is a local variable of type Integer. In the second step, the variable b1 is set to true. The variable b1 is a local variable of type bool."

[0038] Furthermore, the switching conditions W1...5 are also converted into nodes V2, 4, 6, 8, 10 of the graph representation 16. The respective conditions of a

[0039] The switching condition is also represented in natural language. For example, node V2 of a switching condition can have the following content: "Continue the program if the variable b1 is true and the value of the variable xO is equal to 0."

[0040] The edges E1...10 of the graph representation 16 result from the connections between the control steps S1...6 and the switching conditions W1...5 in program 10. In a manner typical for graphs, the edges E x the connection between two nodes V x to.

[0041] The graph representation 16 obtained in this way is subsequently also converted into an embedded form 18.

[0042] In a subsequent step 19, the embedded form 18 of the graph is projected into the embedding space that is suitable for the embedded form X. qThe query 12 uses this. This projection is performed using a calculation rule with a projection matrix W: H. q = W ■ ZG, where ZG is the embedded form 18 of the graph and H q The embedded graph representation projected into the embedding space of the query is 16. Both in H q Both ZG and ZG are vectors of numbers.

[0043] The projection matrix W is determined through a training process, which is expediently performed before the method is used. For this training process, training data is provided, comprising a set, ideally a multitude, of training datasets. Each training dataset includes a graph, a query related to that graph, and a correct answer to the query. The training data is advantageously generated algorithmically by algorithmically creating or selecting a question ("Does the graph contain a closed path?") and the correct answer (yes / no), and then randomly generating a graph that matches the question and answer. For training the projection matrix, it is advantageous to leave all other parameters in the process unchanged and only adjust the matrix itself.

[0044] The result of the projection, i.e., H q , is created with the embedded form X qThe queries 12 are combined into a single vector. This combination is achieved by concatenating the vectors, i.e., stacking them. The resulting single vector is then passed to the Large Language Model 20 as input. In a concrete system such as a development environment, the Large Language Model 20 can be an external component accessed via the internet. The process then involves sending the single vector to the Large Language Model 20 and receiving the response 22 generated by it. The response 22 received is typically a natural language response such as "Signals S1 and S2 are not combined in this program."

[0045] It has been shown that the described procedure is particularly advantageous for large programs 10 and correspondingly extensive graph representations 16, while for small programs 10 and graph representations 16 a second procedure leads to better results.

[0046] This results in an overall procedure 40, which is shown schematically in Figure 4. Accordingly, in a test step 41, it is checked whether the program 10 exceeds a threshold value for the size. The threshold value can be, for example, 50 control steps or a sum of 500 control steps S1...6 and switching conditions W1...5. If this is the case, then processing continues with the first procedure 30 according to Figure 2. Otherwise, a second procedure is used.

[0047] In the second approach, in a conversion step 42, the program 10 or its graph representation 16 is converted into a textual form in natural language. This textual form is then combined (concluded) with the textual query 12 to form a complete query, and this complete query is transmitted to the Large Language Model 20. In this case, no embedding takes place. The response 22 generated by the Large Language Model 20 is received and output. Reference symbol

[0048] 10 Program

[0049] 12 Request

[0050] 14 Embedding layer

[0051] 16 Graph representation

[0052] 18 embedded form of the graph

[0053] 19 mathematical transformation

[0054] 20 Large Language Model

[0055] 22 answers

[0056] 30 First steps

[0057] HQ embedded form of the program

[0058] Xq embedded form of the request

[0059] W Projection Matrix

[0060] V1 ... 11 nodes of the graph representation

[0061] S1...6 control steps

[0062] W1 ...5 Switchover conditions

[0063] E1 ... 10 edges of the graph representation

[0064] 40 total procedures

[0065] 41 Test step

Claims

Patent claims 1. Computer-implemented method for generating a response to a query formulated in natural language concerning a program-like sequence comprising steps and switching conditions, in which - the sequence is converted into a graph with a set of nodes and edges connecting the nodes, - the graph is converted into embedded graph information using embedding, - the request is converted into embedded request information using embedding, - the embedded graph information and the embedded query information are combined into a single embedded information, - the shared embedded information is sent to a Large Language Model, - a result is received from the Large Language Model and used as a response.

2. The method of claim 1, wherein, in order to convert the sequence into the graph, the steps are converted to nodes and the switching conditions are converted to nodes.

3. Method according to claim 1 or 2, wherein an edge is inserted into the graph to convert the sequence into the graph for connected pairs of a step and a switching condition.

4. Method according to claim 1 or 2, wherein for the conversion of the sequence into the graph, for each step - for each action that is part of the step, a text representation is generated, - the text representations are linked together to form a complete text for the step, - the entire text is used as the content of the node that represents the step.

5. Method according to claim 1 or 2, wherein, for the conversion of the sequence into the graph, a text representation is generated for each switching condition and the text representation is used as the content of the node representing the switching condition.

6. Method according to one of the preceding claims, wherein the embedded graph information is transformed into a textual embedding space using a mathematical transformation determined by training.

7. Method according to claim 6, wherein for the training of the mathematical transformation - Training data is provided, wherein the training data comprises a set of training datasets, where a training dataset comprises a graph, a query to that graph, and a correct response to the query. - for a plurality of training datasets, an answer to the query of the training dataset is determined using a method according to one of the preceding claims, and an adjustment of the mathematical transformation is made by comparing the determined answer with the answer of the training dataset without changing other parameters of the method.

8. The method of claim 7, wherein the embedded graph information is a vector of real numbers and the mathematical transformation is a multiplication of the vector by a matrix, wherein the matrix is ​​provided by the training.

9. A method according to one of the preceding claims, wherein in a first alternative the conversion of the graph into embedded graph information by means of embedding and the subsequent steps are carried out, and in a second alternative the sequence is converted into a textual representation and is sent together with the request as text in natural language to the Large Language Model, and a result is received from the Large Language Model and used as a response, wherein a selection is made between the first and second alternative based on the sequence or on the graph.

10. Method according to claim 9, wherein the second alternative is selected if the length of the sequence or the size of the graph is less than a threshold value.

11. Method according to one of the preceding claims, wherein a training of the mathematical transformation and the Large Language Model is performed, wherein - Training data is provided, wherein the training data comprises a set of training datasets, where a training dataset comprises a graph, a query to that graph, and a correct response to the query. - for a plurality of training datasets, a response to the query of the training dataset is determined using a method according to one of the preceding claims, and an adjustment of the parameters for embedding the query, for the Large Language Model, and for the mathematical transformation is made by comparing the The determined response is compared to the response of the training data set without changing any other parameters of the procedure.

12. Method according to one of the preceding claims, wherein the embedded graph information and the embedded query information are each a list of real numbers and the lists are concatenated to generate the common embedded information.

13. Device for generating a response to a query formulated in natural language concerning a program-like sequence comprising steps and switching conditions, with - a user interface for entering the request, - an operator interface for linking or inputting the program-like sequence, wherein the device is designed, - to convert the sequence into a graph with a set of nodes and edges connecting the nodes, - to convert the graph into embedded graph information using embedding, - to convert the request into embedded request information using embedding, - to combine the embedded graph information and the embedded query information into a single embedded piece of information, - to send the common embedded information to a Large Language Model, - to receive a result from the Large Language Model and use it as a response.

Images