Supplementing large language model knowledge via immediate modifications

Abstract

The present disclosure provides techniques and solutions for utilizing information to be used by a large language model in formulating a response to automatically and dynamically complement user prompts to the large language model. In particular, an entity is identified in an original cue. Information about the entity is searched in a semantic framework, and this information is added to the original user cue to provide a modified user cue. In a particular example, the identified entity includes a triad, and the linguistized triad is added to provide a modified user cue. The modified cue may be hidden from the user such that the response of the large language model appears to be responsive to the original cue.

Description

Technical Field

[0001] The present disclosure generally relates to interaction with a large language model. Particular embodiments relate to searching for information relevant to a prompt and modifying the prompt to include such information before submitting the prompt to the large language model. Background Art

[0002] Big Language Models are a revolutionary technology that is rapidly becoming integrated into the daily lives of millions of people. Often referred to as “chatbots,” these models have a remarkable ability to process and understand natural human language input. They can then generate responses in the same fluent human language, making interaction with them highly accessible. The user-friendly nature of these models facilitates easy input and delivery of understandable responses, which, coupled with their exceptional accuracy, contributes to their outstanding capabilities and ease of adoption.

[0003] Nevertheless, large language models do face certain challenges. One such challenge involves their accuracy being dependent on relevant training data. If the most relevant data for a particular query is not publicly available, such as residing within a company repository, the model may not provide an answer or may provide an incorrect answer—sometimes referred to as "hallucination." Additionally, given the time and resources required for training, these models may not be updated frequently. As a result, an information gap may occur between the last training instance of the model and its actual use. Thus, there is room for improvement. Summary of the invention

[0004] This summary is provided to introduce some concepts in a simplified form that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] The present disclosure provides techniques and solutions for automatically and dynamically supplementing user prompts to a large language model with information to be used by the large language model in formulating responses. In particular, entities are identified in the original prompt. Information about such entities is searched in a semantic framework, and such information is added to the original user prompt to provide a modified user prompt. In a specific example, the identified entities include triples, and the linguisticized triples are added to provide a modified user prompt. The modified prompt can be hidden from the user so that the response of the large language model appears to be in response to the original prompt.

[0006] In one aspect, the present disclosure provides a process for modifying user input to a large language model before submitting the user input to the large language model to include supplementary facts for use by the large language model. Receive user input from a user. The user input includes multiple word-grams. Analyze at least a portion of the multiple word-grams. Based on the analysis, determine one or more entities of a semantic framework represented in at least a portion of the multiple word-grams. Determine one or more triples of a semantic framework for at least a portion of one or more entities or associated entities. At least a portion of the one or more triples or representations thereof are added to the user input to provide a modified user input. The modified user input is submitted to the large language model. The modified user input is processed using the large language model to provide a response. Return a response in response to receiving the user input.

[0007] The present disclosure also includes computing systems and tangible, non-transitory computer-readable storage media configured to perform the above-described methods or including instructions for performing the above-described methods. As described herein, various other features and advantages may be incorporated into the technology as desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An exemplary directed graph is shown.

[0009] Figure 2 The domain and range of attributes in the architecture of a directed graph are shown.

[0010] Figure 3 An exemplary SPARQL query and the results of the query are shown.

[0011] Figure 4 A block diagram representation of the subsequent figures is shown.

[0012] Figure 5 A canard-foot notation is shown for annotation in subsequent figures.

[0013] Figure 6 An exemplary system architecture for a method according to some embodiments is shown.

[0014] Figure 7 An algorithm for computing strings from a directed graph for input to a language model is shown.

[0015] Figure 8 A sorting algorithm for sorting statements of a directed graph is shown.

[0016] Fig. 9 shows a directed graph that may be provided as input to a sorting algorithm.

[0017] Fig.10 The relationship between a condition and one or more patterns is shown.

[0018] Fig.11 Is a flowchart of the process of creating a domain data model, including mapping elements of the domain data model to elements of the core data model.

[0019] Fig.12 shows the expansion of the problem mode Figure 7 The algorithm.

[0020] Fig.13 An embodiment of a method for providing data from a directed graph to a language model implemented as a remote service is shown.

[0021] Fig.14 A hybrid implementation of a method for providing data from a directed graph to a language model is shown.

[0022] Fig.15 A local implementation of a method for providing data from a directed graph to a language model is shown.

[0023] Fig.16 Shown are the results provided by a large language model when the model lacks sufficient information to correctly respond to user input. Fig.16 It is also shown how the disclosed innovation can provide additional information to a large language model for responding to user input and the resulting responses.

[0024] Fig.17 is a flow chart of a process of determining information relevant to user input to a large language model, supplementing the user input with such information, and linking answers to the supplemented user input to entities in a knowledge graph.

[0025] Fig.18A Illustrated is how elements of a modified user input may be mapped to: the original user input, facts retrieved from a search for information about entities recognized in the original user input, and additional instructions that may be added to the original user input before submitting the modified user input to a large language model.

[0026] Fig.18B and Fig. 18C A user interface screen is shown that displays information of recognized entities in a large language model's response in response to a user selecting such an entity.

[0027] Fig.19 An example knowledge graph query and example subject, object, predicate triples generated therefrom are shown.

[0028] Fig. 20 Shows knowledge Figure 3 Example functions for linguizing tuples, and how to use Fig.19 Knowledge Figure 3Example linguistic triples for tuple generation.

[0029] Fig.21 is a sample knowledge graph.

[0030] Figures 22A-22C Sample code for querying the knowledge graph is provided.

[0031] Fig.23 An example computing environment is shown in which the disclosed technology may be implemented.

[0032] Fig.24 is a flow chart of a process for modifying user input to a large language model to include supplemental facts for use by the large language model before submitting the user input to the large language model.

[0033] Fig.25 is a diagram of an example computing system in which some described embodiments may be implemented.

[0034] Fig.26 is an example cloud computing environment that can be used in conjunction with the techniques described herein. DETAILED DESCRIPTION

[0035] Example 1 - Overview

[0036] Big Language Models are a revolutionary technology that is rapidly becoming integrated into the daily lives of millions of people. Often referred to as “chatbots,” these models have a remarkable ability to process and understand natural human language input. They can then generate responses in the same fluent human language, making interactions with them highly accessible. The user-friendly nature of these models facilitates easy input and provides understandable responses, which, coupled with their exceptional accuracy, contributes to their outstanding capabilities and ease of adoption.

[0037] Nevertheless, large language models do face certain challenges. One such challenge involves their accuracy being dependent on relevant training data. If the most relevant data for a particular query is not publicly available, such as residing within a company repository, the model may not provide an answer or may provide an incorrect answer—sometimes referred to as "hallucination." Additionally, given the time and resources required for training, these models may not be updated frequently. As a result, an information gap may occur between the last training instance of the model and its actual use. Thus, there is room for improvement.

[0038] The present disclosure provides techniques that can be used to influence the results provided by a large language model. For example, a large language model may be provided with a set of "facts" that it can use in whole or in part to generate a response. As used in this context, "facts" refer to information that the large language model will use when preparing a response, regardless of whether the information is actually true. In some cases, the large language model can be told to explicitly assume that "facts" are true, such as bypassing functionality of the large language model that might otherwise question the authenticity of the information. Similarly, if desired, the large language model can be told to provide a response only using explicitly provided facts.

[0039] The present disclosure provides a technique in which a set of facts is determined from an initial user input and added to the user input to provide a modified user input, and in which a large language model will respond to the modifier user input. That is, the initial user input can be "intercepted" and modified before being submitted to the large language model. The user input includes multiple tokens that can be analyzed. As used herein, a "token" of a user input refers to a discrete unit of text or a sequence of characters within the user input, including tokens that can be analyzed by a named entity recognition service. Tokenization can be achieved by scanning the text and splitting it at specific boundaries (such as spaces or punctuation marks) to isolate different units or tokens, which can then be subjected to various language analysis or computational operations.

[0040] In a particular embodiment, user input is provided to a named entity recognition service that accesses one or more knowledge graphs. Information from the knowledge graph can be the basis for additional facts provided in the modified user input. In a specific example, information from the knowledge graph can be "verbalized" before being added to the user input. That is, the big language model is designed to work with normal human language input. Converting the "raw" results of a search from a knowledge graph into a verbalized form, even a simple form, can improve the results obtained from the big language model using the modified user input.

[0041] The knowledge graph can also be used to enhance the usefulness of the response. For example, the response can be submitted to a named entity recognition service, and the response can be linked to specific information in the knowledge graph (or information outside the knowledge graph that was recognized using the knowledge graph, such as by performing a query against another data source using the results of a search from the knowledge graph for entities recognized by the named entity recognition service).

[0042] Example 2 provides a general discussion of knowledge graphs, and specific techniques that can be used to convert elements retrieved from a knowledge graph into a linguistic format. Examples 3-9 describe various techniques for modifying user input with content that a large language model may not know in accordance with the present disclosure. The described innovations can be used in a variety of other ways, for example, a set of "prescribed facts" can be provided to a large language model, regardless of whether this information can be part of the training corpus of the large language model.

[0043] While the disclosed techniques are generally described with reference to a knowledge graph as a source of additional information to be added to a prompt and used by a large language model in formulating a response, the techniques may use information maintained in another format. Typically, the techniques retrieve information from a "semantic framework," which may be a knowledge graph, or may alternatively use techniques such as ontologies, other types of semantic webs, semantic databases, graph databases, linked data, or taxonomies or folk classifications.

[0044] Similarly, aspects of the present disclosure are described as being implemented using a resource description framework. However, other types of data structures or representations that convey equivalent information about semantic relationships or associations, such as expressing two related entities and the relationship between those entities, may be used.

[0045] Example 2 - Knowledge Figure 3 Verbalization of tuple examples

[0046] An enterprise may have a variety of different products, services, and teams. An enterprise may also have a comprehensive knowledge graph that stores knowledge about the skills, processes, experiences, capabilities, and insights that are relied upon in the day-to-day operations of the enterprise. The content of the knowledge graph may also include enterprise-specific acronyms, the enterprise's departments, and product specifications. This knowledge can enable the enterprise to react to business situations in a fast, professional, and flexible manner. The construction and maintenance of knowledge graphs can be expensive and labor-intensive. Knowledge graphs (i.e., semantic web and / or linked data web) can be specified using the Resource Description Framework (RDF).

[0047] In some cases, users want to ask questions or provide tasks to a language model, such as a large language model based on a generative pre-trained transformer (such as ChatGPT). However, language models are usually trained on unlabeled human-readable text in an unsupervised manner. Therefore, language models may not be able to directly process knowledge graphs or use knowledge graphs as input, such as for training.

[0048] Therefore, it may be desirable to maximize the usability of knowledge graphs, for example, to use knowledge graphs as the basis for artificial intelligence applications, and more specifically, to train or otherwise improve language models. When training a language model, the language model can be used to answer questions or perform tasks based on the knowledge stored in the knowledge graph.

[0049] Additionally, it may be desirable to extract human-readable text from a knowledge graph, for example, for interpreting answers provided by software (eg, a process advisor) that relies on the knowledge graph.

[0050] According to one aspect, a computer-implemented method for providing data from a directed graph to a language model is provided. The method includes defining a plurality of conditions and a plurality of patterns, wherein each condition has at least one corresponding pattern. The method also includes receiving a subset of the directed graph, wherein the subset of the directed graph includes a plurality of sentences. Each sentence includes a subject, an object, and a predicate that relates the subject to the object. For each of the sentences in the subset of the directed graph, the following operations are performed: when one of the conditions matches the corresponding statement and the pattern corresponding to the condition can be applied to the corresponding statement, a string for the corresponding statement is calculated using the pattern. The calculated string is provided as input to the language model.

[0051] Providing data from the directed graph to the language model may include extracting or reading data from the directed graph and feeding or sending the data as input to the language model. A statement may be referred to as a triple or triple statement having subject, predicate, and object components. A condition may correspond to a corresponding pattern in the sense that when the condition is determined to be true for the corresponding statement, it may be determined whether the corresponding pattern can be applied to the corresponding statement.

[0052] The clause, each of the conditions has at least one corresponding pattern, can be understood to mean that each of the conditions in the plurality of conditions has at least one corresponding pattern in the plurality of patterns. Therefore, each of the plurality of conditions can be assigned at least one pattern in the plurality of patterns. In other words, when a condition has at least one corresponding pattern, the at least one corresponding pattern is assigned to the condition. In addition, each of the plurality of conditions can be assigned a plurality of patterns in the plurality of patterns.

[0053] When one of the conditions matches the corresponding statement and the pattern corresponding to the condition can be applied to the corresponding statement, the clause may include determining whether at least one pattern corresponding to the condition can be applied to the corresponding statement. Therefore, determining whether at least one pattern corresponding to the condition can be applied to the corresponding statement may involve determining whether the pattern corresponding to the condition matches the corresponding statement, that is, testing whether the corresponding statement has a characteristic or element specified by the pattern.

[0054] For example, determining whether a pattern can be applied to a sentence that includes a subject can be performed as follows:

[0055] The plurality of conditions may include a first condition:

[0056] <?s><?p><?o> .

[0057] BIND(sap:BusinessActivity AS<?s> )

[0058] BIND(rdf:type AS<?p> )

[0059] The first mode may correspond to the first condition:

[0060] The<s.rdfs:label> is a business activity.

[0061] Since the first pattern includes a reference to a label of a subject s, if a statement including a subject satisfies the first condition, i.e., is a BusinessActivity, but the subject of the statement does not have the label required by the first pattern, the first pattern cannot be applied (i.e., will not match) the statement.

[0062] Using a pattern to evaluate a string from a corresponding statement can involve directly outputting the text of the pattern and matching the pattern operators with components of the statement.

[0063] Furthermore, when evaluating a string, a pattern corresponding to a condition may access not only the corresponding statement that matches the condition, but also one or more additional statements of the subset of the directed graph. In other words, the pattern may consider statements that do not match the condition. Thus, the condition may trigger execution of the pattern for the corresponding statement, which in turn may trigger processing of at least a portion of the subset of the directed graph (or the entire subset of the directed graph) using the pattern.

[0064] Subject, object, and predicate may be referred to as the components of their respective statements. A subject and / or predicate may be an RDF resource (e.g., a subject and / or predicate may have RDF resource properties and may be a type or tag). An object may be a literal (e.g., an RDF literal) with a defined data type, such as a string, integer, boolean, or double (as defined in the Extensible Markup Language (XML) Schema Definition Language (XSD)). For more information about RDF, refer to the RDF specification "Concepts and Abstract Syntax", https: / / www.w3.org / TR / rdf11-concepts /

[0065] The computed string can be provided directly to the language model (e.g., in the case of small strings, such as less than 1GB), or can be serialized into a text file before being provided to the language model (e.g., in the case of terabytes of data computed from a comprehensive directed graph). In summary, the method accepts a subset of a directed graph as input and generates grammatically correct sentences as computed strings. The method iterates over the input sentences (i.e., triples) in the subset of the directed graph and uses conditional filters to determine which patterns can be applied to each statement. The pattern is applied to the condition when the condition matches the statement and the statement has the elements required by the pattern corresponding to (e.g., assigned to) the condition. The result of the iteration can be a set of strings such that a string is computed for each statement in the input. The syntax and interpretation of the patterns are discussed in more detail below.

[0066] The statements of directed graphs can be close to human language.

[0067] Thus, providing the computed string as input to a language model can maximize the usability of a subset of the directed graph, for example, by using the subset of the directed graph as the basis for an artificial intelligence application. Once the language model has processed the computed string, the language model can be used to answer questions or perform tasks based on the knowledge stored in the subset of the directed graph. Thus, the time, labor, and expense used to construct the directed graph can be utilized in further ways (e.g., answering questions or by using the language model).

[0068] Additionally or alternatively, it may be desirable to extract human-readable text from a subset of the directed graph, for example, for interpreting answers provided by software (eg, a process advisor) that relies on the subset of the directed graph.

[0069] In some cases, each of the conditions includes at least three condition variables (variables appearing in a condition may be referred to as condition variables). Each of the condition variables may correspond to (e.g., store) a different component of the statement. For example, the first of the condition variables matches the subject, the second of the condition variables matches the predicate, and the third of the condition variables matches the object. At least one of the condition variables may be bound to at least one value, e.g., to an RDF property. In other words, at least one of the condition variables may specify at least one value (e.g., an RDF property) that a component of the statement must have. Each of the condition variables may specify an instance of a class (e.g., an RDF class) or a literal (e.g., an RDF literal). An instance of a class may be referred to as an instance. An instance may relate to a specific concept and have a definite article, while a class may relate to a general concept and have an indefinite article.

[0070] Each condition can be applied to a statement and can evaluate to true or false. In other words, a condition can return a Boolean value. For example, if the condition evaluates to true, the condition matches the corresponding statement and it is determined whether at least one pattern corresponding to the condition can be applied to the corresponding statement. The following are numbered examples of conditions that can be in multiple conditions:

[0071] 1.<?s><?p><?o> .

[0072] 2.<?s><?p><?o> .

[0073] BIND(rdfs:label AS<?p> )

[0074] 3.<?s><?p><?o> .

[0075] <?s> rdf:type sap:BusinessActivity .

[0076] BIND(rdfs:label AS<?p> )

[0077] 4.<?s><?p><?o> .

[0078] <?s> rdf:type sap:BusinessActivity .

[0079] <?o> sap:requires<?r> .

[0080] BIND(rdfs:label AS<?p> )

[0081] In the first condition,<?s> " is the variable corresponding to the subject in the directed graph, "<?p> " is the variable corresponding to the predicate in the directed graph, and "<?o> ” is a variable corresponding to an object in a directed graph. Thus, the first condition specifies (i.e., in order for the first condition to evaluate to true) that the statement must contain a subject, a predicate, and an object. The second condition requires that the statement contain a subject, a predicate, and an object, and that the predicate is an rdfs:label. The third condition requires that the statement contain a subject, a predicate, and an object, that the subject has the property (more specifically, is of type) sap:BusinessActivity, and that the predicate is an rdfs:label. The fourth condition requires that the statement contain a subject, a predicate, and an object, that the subject has the property (more specifically, is of type) sap:BusinessActivity, and that the object has “sap:requires” with the object “<?r> " (i.e., a subject-object relationship), and the predicate is rdfs:label. In this regard, "<?r> " is bound to the required variable.

[0082] A condition in a plurality of conditions can be used to prevent a pattern from being used to compute a semantically incorrect string. In other words, a condition can be used to ensure that the computed string is semantically and / or grammatically correct. Therefore, by assigning a pattern to a condition, the circumstances in which the pattern is applied can be limited, thereby ensuring or facilitating the computation of a semantically correct string (i.e., a sentence). Without a condition, a pattern can be applied to compute an exemplary fantasy string, such as "The Harry Potter Book is a business activity" or "The Star Wars Movie is a business activity". However, the exemplary fantasy string is semantically incorrect; therefore, the exemplary fantasy string will not be helpful as an input to a language model, and may prolong the training of the language model or even cause the language model to produce incorrect output.

[0083] In some cases, at least one of the conditions has multiple corresponding patterns. Thus, computing a string from the corresponding statement using the patterns may include computing multiple strings from the corresponding statement using each pattern corresponding to a condition that can be applied to the corresponding statement (i.e., a condition that matches the corresponding statement).

[0084] Alternatively, calculating the string from the corresponding statement using the pattern may include determining a random order of the patterns corresponding to the condition, and calculating the string from the corresponding statement using only the first one of the random order of the patterns that can be applied to the corresponding statement. For example, patterns 1 to 4 may be ordered as 2, 4, 3, 1, and pattern 2 may be applied to the corresponding statement, and therefore, pattern 2 is applied to the corresponding statement.

[0085] For example, multiple conditions can include a sequenceID condition:

[0086] <?I1> <sequenceid><?L1>.

[0087] The sequenceID condition can correspond to the following pattern (A):

[0088] (A) The sequence identifier of<?I1.rdf:type.rdfs:label><?I1.rdfs:label> is<?L1> .

[0089] Continuing with the example, a subset of the directed graph could include the following statements:

[0090] 1. Emissions Management isA BusinessCapability.

[0091] 2. BusinessCapability rdfs:label "Business Capability" .

[0092] 3. EmissionsManagement SequenceID "5".

[0093] The SequenceID condition matches only statement (3) because statement (3) includes "SequenceID" and statements (1) and (2) do not include "SequenceID".

[0094] Pattern (A) can be applied to statements (1), (2), and (3) to evaluate the following string:

[0095] The sequence identifier of Business Capability is 5.

[0096] All three statements are needed to evaluate the string above because statements (1) and (2) provide contextual information for statement (3).

[0097] Thus, as described above, when evaluating a string, a pattern corresponding to a condition may access not only the corresponding statement that matches the condition, but also one or more additional statements of a subset of the directed graph. In other words, the pattern may consider statements that do not match the condition. Thus, the condition triggers the execution of the pattern for the corresponding statement.

[0098] Furthermore, statements of a subset of the directed graph may be checked iteratively. Thus, in the example above, statements (1) and (2) do not cause the pattern to be triggered, but statement (3) does.

[0099] In this example, determining whether pattern (A) can be applied to statement (3) may include determining context information items of pattern (A), namely:

[0100] The label of the type of variable <?I1|

[0101] The label of variable <?I1|

[0102] The two context information items are not available in statement (3) itself, but are available elsewhere in a subset of the directed graph.

[0103] Given statement (3), the context information exists in a subset of the directed graph, and pattern (A) can be executed for statement (3).

[0104] Whenever forward dot notation is used, we include additional information that does not exist in the triple itself.

[0105] In some cases, each pattern includes one or more of the following:

[0106] - At least one variable, where the variable specifies (e.g., is bound to) a category, an instance of a category, a literal, or a predicate;

[0107] - Text, such as one or more articles (e.g., a grammatical definite article or a grammatical indefinite article);

[0108] - At least one property applied to the variable.

[0109] Each pattern may also include a language filter. The literal may specify a numerical value or text, where the literal may conform to an RDF schema class of the literal value.

[0110] Advantageously, the pattern can achieve a combination of static text and variables that specify the structure of a subset of the directed graph (e.g., the subject, object, and predicate of one of the statements), possibly supplemented with information generated by reifying the subset of the directed graph.

[0111] Additionally or alternatively, the pattern may include at least one specific pattern (i.e., a customized pattern) and multiple default patterns. When the condition corresponding to the specific pattern matches the corresponding statement and the specific pattern can be applied to the corresponding statement, calculating a string from the corresponding statement using the pattern may include using the specific pattern. When the condition corresponding to the specific pattern does not match the corresponding statement, the method may further include determining whether the condition corresponding to one of the default patterns matches the corresponding statement. When the condition corresponding to one of the default patterns in the default patterns matches the corresponding statement, calculating a string from the corresponding statement using one of the default patterns. Defining multiple conditions and multiple patterns may also include defining at least three conditions and at least three patterns, where at least one of the three patterns is a specific pattern, and at least one of the three conditions corresponds to the specific pattern.

[0112] The following are example patterns that can be included in multiple patterns:

[0113] Text<?I1.rdf:type.rdfs:label> Text

[0114] A variable can be bound to sap:printReceive. Patterns such as the example pattern above can use forward notation (also called notation) to refer to fields, components, or sub-properties of a property. This can provide an advantage to patterns over regular SPARQL, which does not support forward dot notation, because forward dot notation enables more compact expressions.

[0115] Thus, the above exemplary pattern may be applied to (eg, a subset of the directed graph may include) the following three statements:

[0116] sap:PrintReceipt rdf:type sap:Task.

[0117] sap:Task rdfs:label "Task"@en .

[0118] sap:Task rdfs:label "Process Task"@en .

[0119] In some cases, after applying a pattern to a corresponding statement, the pattern is not applied to additional statements that match the pattern, i.e., additional statements in a subset of the directed graph that match the pattern may be skipped. Matching only one statement and skipping additional statements may be configurable options. For example, as discussed below, a postfix operator may cause a Cartesian product to be computed.

[0120] For example, after applying the pattern to the corresponding statement including the rdfs:label for the object, further statements including the rdfs:label for the object may be skipped, i.e., the pattern is not applied to the further statements. Thus, computing the string from the above statement using the exemplary pattern will produce the following: "Text Task Text".

[0121] At least one of the patterns may include a filter condition and / or a post-operator. The filter condition may specify a language. The post-operator may cause a Cartesian product to be performed. As another example, a subset of a directed graph may include the following four statements:

[0122] sap:PrintReceipt rdf:type sap:Task.

[0123] sap:Task rdfs:label "Task"@en .

[0124] sap:Task rdfs:label "Process Task"@en .

[0125] sap:Task rdfs:label "Schritt"@de .

[0126] The following additional exemplary patterns may be included in the multiple patterns and may be applied to the above four statements:

[0127] Text<?I1.rdf:type.rdfs:label(lang = ‘en’)*> Text

[0128] The above further exemplary pattern includes a filter condition specifying a language and an asterisk postfix operator "*" that generates a Cartesian product. The Cartesian product can generate all possible combinations of the aforementioned elements. Therefore, the following string will be calculated by applying another exemplary pattern including a Cartesian product, because the other exemplary pattern is for English tags and there are two English tags in the above four sentences:

[0129] Text Task Text

[0130] Text Process Task Text

[0131] Therefore, the above sentence including "Schritt'@de" will not be processed because the sentence does not meet the filtering condition in another exemplary pattern (ie, the sentence is not in English language). Without the asterisk postfix operator in the above another exemplary pattern, only the first string "Text Task Text" will be calculated.

[0132] The above exemplary patterns and further exemplary patterns may be specific patterns, ie, patterns applicable to one directed graph or a group of directed graphs.

[0133] The string may be computed from the corresponding statement using only the default mode. However, the use of a specific mode may result in a computed string that more accurately and precisely describes the contents of the subset of the directed graph.

[0134] In addition to the asterisk, you can use other postfix operators (that is, operators provided at the end of a pattern, also called postfix operators). For example, an additional postfix operator can restrict the output of a Cartesian product to a specified number of combinations, such as about 10 combinations.

[0135] As another example, the following requirement pattern can be applied to the above four statements:

[0136] The<?I1.rdf:type.rdfs:label(lang = ‘en’)*><?I1.rdfs:label> requiresa<?I2.rdfs:label> .

[0137] In this example,<?I1> can be bound to sap:PrintReceipt, and<?I2> can be bound to sap:Printer. Therefore, by applying the requirement pattern to the above four statements, the following string can be calculated:

[0138] -A printer is required to print receipts for processing tasks.

[0139] -Task printing receipt requires a printer.

[0140] Without the asterisk postfix operator in the requirement pattern, only the first string, i.e. "Processing task print receipt requires printer" will be calculated.

[0141] In some cases, each computed string is a grammatically correct sentence, where the condition and / or pattern may ensure that the computed string is a grammatically correct sentence.

[0142] Additionally or alternatively, at least one specific mode may include multiple specific modes. Each specific mode may be applicable to a set of directed graphs defined according to the resource description framework or a set of knowledge graphs defined according to the resource description framework. Each default mode may be applicable to any directed graph defined according to the resource description framework or any knowledge graph defined according to the resource description framework.

[0143] Thus, each specific mode can be defined for a single local network and corresponding directed graph or a set of local networks and a corresponding set of directed graphs, while the default mode can be applied to any directed graph.

[0144] Modes can include one or more of the following five modes:

[0145] - Modes that apply to instance-to-instance statements, including variables<I1,p,I2>

[0146] - Patterns for instance-to-class statements, including variables<I1,p,C1>

[0147] - Modes for class-to-class statements, including variables<C1,p,C2>

[0148] - Modes for applying instances to literal statements, including variables<I1,p,L1>

[0149] - Patterns that apply to classes to literal statements, including variables<C1,p,L1>

[0150] The five modes mentioned above can be made to work by including the corresponding variables<I1,p,I2> ,<I1,p,C1> ,<C1,p,C2> ,<I1,p,L1> ,<C1,p,L1> The correspondence condition applies to instance-to-instance statements, instance-to-class statements, class-to-class statements, instance-to-literal statements, and class-to-literal statements.

[0151] The above five patterns can be the default patterns because they are applicable to any directed graph, or more specifically, any knowledge graph.

[0152] For the above five modes, "I1" and "I2" refer to variables of instances (i.e., instances of a class), "C1" and "C2" refer to variables of a class, "L1" refers to variables of a literal, and "p" refers to variables of a predicate. The first of the five modes applicable to instance-to-instance statements can be implemented as follows:

[0153] The<?I1.rdf:type.rdfs:label><?I1.rdfs:label><?p.rdfs:label> the<?I2.rdf:type.rdfs:label><?I2.rdfs:label> .

[0154] The second of the five patterns applicable to instance-to-class statements can be implemented as follows:

[0155] The<?I1.rdf:type.rdfs:label><?I1.rdfs:label><?p.rdfs:label> a<?C1.rdfs:label> .

[0156] The third of the five patterns applicable to category-to-category statements can be implemented as follows:

[0157] A<?C1.rdfs:label><?p.rdfs:label> a<?C2.rdfs:label> .

[0158] The fourth of the five patterns applicable to instance-to-literal statements can be implemented as follows:

[0159] The<?I1.rdfs:label><?p.rdfs:label><?L1> .

[0160] The fifth of the five patterns applicable to categories to literal statements can be implemented as follows:

[0161] A<?C1.rdfs:label><?p><?L1> .

[0162] A user or administrator may define additional default modes or change the exemplary default modes provided above.

[0163] Additionally or alternatively, multiple patterns may include at least one text pattern and at least one question pattern. The text pattern and / or the question pattern may be a specific pattern. The text pattern and / or the question pattern may be a default pattern. Each condition may correspond to at least one text pattern, and at least one condition may correspond to at least one question pattern. For example, conditions may be defined so that each condition must correspond to at least one text pattern, and each condition may correspond to at least one question pattern. In addition to or in place of the text pattern, a configuration option may be set to apply the question pattern. When one of the conditions matches a corresponding statement and the configuration option is set to apply the question pattern and the question pattern corresponding to the condition may be applied to the corresponding statement, the method may include: in addition to the string calculated from the corresponding statement using the text pattern, a string is calculated from the corresponding statement using the question pattern and / or another string is calculated from the corresponding statement using the question pattern.

[0164] For example, a subset of the directed graph might include the following seven statements:

[0165] sap:PrintReceipt sap:requires sap:Printer .

[0166] sap:PrintReceipt rdf:type sap:Task.

[0167] sap:PrintReceipt rdfs:label "Print Receipt" .

[0168] sap:Printer rdfs:label "Printer" .

[0169] sap:Task rdfs:label "Task"@en .

[0170] sap:Task rdfs:label "Process Task"@en .

[0171] sap:Task rdfs:label "Schritt"@de .

[0172] Continuing with the example, the plurality of modes may include the following modes (eg, text modes):

[0173] The<I1.rdf:type.label(lang = ‘en’)*><I1.rdfs:label> requires a<I2.rdfs:label> .

[0174] (<I1.rdf:type.label(lang = ‘en’)*><I1.rdfs:label> need<I2.rdfs:label> .)

[0175] Additionally, multiple patterns may include the following question patterns preceding the text pattern just above:

[0176] What is required by<I1.rdf:type.label(lang = ‘en’)*><I1.rdfs:label> ?

[0177] (<I1.rdf:type.label(lang = ‘en’)*><I1.rdfs:label> What do you need?

[0178] The "*" (asterisk) operator (i.e., postfix operator) in the question pattern causes the Cartesian product to be computed. In the text and question pattern above,<?I1> is bound to (i.e. holds the value of) sap:printReceive,<?p> is bound to sap:requires, and<?I2> is bound to sap:Printer. Therefore, the configuration options can be set to apply both question mode and text mode. Therefore, by applying both question mode and text mode to the above seven statements, the following question / answer string is calculated:

[0179] Q: What is needed to print a receipt for a task?

[0180] Answer: A printer is required to print the task receipt.

[0181] Q: What is needed to print a receipt for a task?

[0182] Answer: A printer is needed to print receipts for processing tasks.

[0183] Q: What is needed to print receipts for processing tasks?

[0184] Answer: A printer is required to print the task receipt.

[0185] Q: What is needed to print receipts for processing tasks?

[0186] Answer: A printer is needed to print receipts for processing tasks.

[0187] For clarity, the string above is preceded by "Question: (Q:)" and "Answer: (A:)". Therefore, the question pattern can be used to simulate a question-answer interaction. As discussed above in the example about the Cartesian product, without the asterisk operator of this example, the string would be evaluated only from the first statement to which the question and text patterns can be applied, that is, the first statement that matches the question and text patterns.

[0188] In some cases, the subset of the directed graph may be the entire directed graph. Alternatively, the subset of the directed graph may be a proper subset of the entire directed graph and may be determined by means of a query on the directed graph. The query may be a SPARQL Protocol and RDF Query Language (SPARQL) query.

[0189] In some cases, a subset of a directed graph includes multiple nodes connected by edges. Nodes can represent real-world entities, and edges can represent relationships between entities or between entities and types of entities (i.e., categories). Thus, predicates can be distinguished based on whether they connect two entities or connect an entity and an entity type. Entities can also be referred to as resources. For each statement, a subject can correspond to a node, an object can correspond to a (different) node, and an edge corresponding to the predicate can connect the subject node to the object node.

[0190] A node may have a corresponding category, such that each node has a corresponding category. The (corresponding) category may be part of (or organized in) a pattern (i.e., a data schema or ontology). The schema may be defined in RDF or a web ontology language.

[0191] The following are examples of categories:

[0192] :State a rdfs:Class .

[0193] :EuropeanState a rdfs:Class .

[0194] :City a rdfs:Class .

[0195] Thus, ":State" is a resource that is a class (more specifically, an RDF class). The class ":EuropeanState" is another resource that is a class, more specifically, a subclass of ":State". Thus, hierarchies of classes are possible. Furthermore, multiple inheritance is possible.

[0196] Additionally or alternatively, directed graphs can be labeled and multi-relational. Thus, both nodes and edges can have labels, and edges can have directions. The object of a statement can be a label of a directed graph. A directed graph can be multi-relational in the sense that the edges have different labels. The nodes of a directed graph can be subjects or objects, and the edges can be predicates.

[0197] Additionally or alternatively, a schema may include properties. Each property may apply to at least one of the categories of the schema. At least one of the properties may have a domain and / or a range. Each property may be used by (or applied to) at least one statement. A domain (e.g., rdfs:domain) may specify a category to which a subject belongs, and a range (e.g., rdfs:range) may specify a category to which an object belongs. More specifically, a domain may specify a category to which a subject of a statement belongs, and a range may specify a category to which an object of a statement belongs. For RDF schema, please refer to the W3C RDF Schema specification https: / / www.w3.org / TR / rdf-schema / .

[0198] The following are examples of properties:

[0199] rdf:type a rdf:Property

[0200] dbo:foundationPlace a rdf:Property.

[0201] :EuropeanState rdfs:subClassOf :State .

[0202] :locatedIn a rdf:Property .

[0203] :capitalOf a rdf:Property .

[0204] :capitalOf rdfs:subPropertyOf :locatedIn .

[0205] Thus, ":locatedIn" and ":capitalOf" are properties. Furthermore, ":capitalOf" is a sub-property of ":locatedIn". Thus, properties can also form hierarchies. The property ":EuropeanState rdfs:subClassOf :State" indicates that ":EuropeanState" is a sub-class in a class hierarchy that includes the class ":State" and the sub-class ":EuropeanState".

[0206] Thus, a schema can provide a vocabulary for a directed graph (e.g., a knowledge graph). A directed graph can have a predefined property prefix that can indicate whether a node (i.e., subject or object) is an instance of a category or a class (e.g., if a node has the prefix "dbo" representing a DBpedia ontology, the node can be a class, and if a node has the prefix "dbr" representing a DBpedia resource, the node can be an instance). In some cases, a directed graph can use URI design to distinguish between instances and categories. A directed graph can include statements that explicitly indicate that certain nodes are categories. In some cases, whether a particular node represents an instance or a class can depend on the underlying model. For example, whether a node is a class (and is included in the schema of the directed graph) or an instance (and therefore is not included in the schema of the directed graph) can be determined by examining the rdf:type property: if the type is owl:Class, the node is a class and is included in the schema; otherwise, the node is an instance (i.e., an instance of a class) and is not included in the schema.

[0207] In some cases, the total number of patterns is greater than or equal to the total number of attributes.

[0208] Furthermore, for an ontology (ie, schema) O with a set of categories C and a set of properties P, only P conditions are needed to compute a string from the complete directed graph without syntax errors.

[0209] Additionally or alternatively, data from directed graphs cover multiple subject areas. Each statement can be identified by at least one uniform resource identifier (URI). At least one of the nodes and edges can be identified by a URI or an internationalized resource identifier (IRI). More specifically, the nodes and edges can each be identified by a URI or an IRI. In some cases, one or more of the subject, object, and predicate can be a URI. Some nodes (e.g., nodes corresponding to objects) can be identified via text rather than URIs. RDF can be used to represent directed graphs. A directed graph can be a knowledge base and / or a knowledge graph. A statement can be referred to as a fact or a factual statement. Therefore, a directed graph can have a structure similar to a known knowledge graph, such as DBPedia, Wikidata, BabelNet, DBkWik, Freebase, and DBnaray.

[0210] Compared to relational databases, knowledge graphs have a more flexible data structure because the type of data provided by a knowledge graph can vary. For example, the attributes associated with different instances can be different, even if these instances share the same category (e.g., "SAP_SE" and "BASF_SE" can have different attribute data available, even though they share the same category "Company"). On the other hand, a relational database can be represented in a knowledge graph format, i.e., a knowledge graph can be a higher-level abstraction of a relational database.

[0211] In some examples, nodes in a directed graph (e.g., a knowledge graph) may be organized in a hierarchy, where lower-level nodes (representing more specific objects) may be connected to higher-level nodes (representing more general objects) by one or more edges. A lower-level node (or the lower-level object it represents) may be referred to as a descendant of a higher-level node (or the higher-level object it represents), and a higher-level node (or the higher-level object it represents) may be referred to as an ancestor of a lower-level node (or the lower-level object it represents).

[0212] The method may also include receiving one or more rules corresponding to the subset of the directed graph. The rules may be reasoning, logic, inference, or RDF schema rules. The method may also include materializing the subset of the directed graph by applying the rules to multiple statements to compute additional statements. Materializing the subset of the directed graph may be described as adding context data or a reference to context data to the subset of the directed graph.

[0213] Materializing a subset of a directed graph can be achieved by applying inferences or (inference) rules to the subset of the directed graph.

[0214] Examples of numbered rules are as follows:

[0215] 1. Every object of the predicate "dbo:foundationPlace" is a country

[0216] 2. Every subject of the predicate ":capitalOf" is a city

[0217] 3. Every object of the predicate ":capitalOf" is a country

[0218] The first rule can be implemented by setting the scope of the "dbo:foundationPlace" predicate so that its object must be an instance of the country class. The second rule can be implemented by setting the domain of the ":capitalOf" predicate so that its subject must be an instance of the city class. Similar to the first rule, the third rule can be implemented by setting the scope of the ":capitalOf" predicate so that its object must be an instance of the country class.

[0219] An example of reification (i.e., reasoning) is as follows. The reification is based on the following statement:

[0220] :Madrid :capitalOf :Spain .

[0221] and the following properties:

[0222] :capitalOf rdfs:domain :City .

[0223] :capitalOf rdfs:range :Country

[0224] :capitalOf rdfs:subPropertyOf :locatedIn .

[0225] Thus, concretization may include combining statements with one or more attributes. More specifically, concretization may include combining statements with attributes that restrict the subject or object of the statement (e.g., attribute restrictions). These combinations may be used to determine further statements, such as the class of which the subject of the statement is an instance and / or the class of which the object of the statement is an instance. Concretization may be understood as determining statements that can be implicitly derived from the directed graph and adding the determined statements to the directed graph. The following are three numbered examples of reasoning:

[0226] 1. :Madrid :capitalOf :Spain .

[0227] :capitalOf rdfs:domain :City .

[0228] → :Madrid a :City .

[0229] 2. :Madrid :capitalOf :Spain .

[0230] :capitalOf rdfs:range :Country

[0231] → :Spain a :Country .

[0232] 3. :Madrid :capitalOf :Spain .

[0233] :capitalOf rdfs:subPropertyOf :locatedIn .

[0234] → :Madrid :locatedIn :Spain .

[0235] Each of the three examples above combines the statement ":Madrid:capitalOf:Spain" with a different property in order to compute (i.e., derive) an additional statement. In the first example, the statement ":Madrid:capitalOf:Spain" is combined with the property ":capitalOf rdfs:domain:City" to compute ":Madrid a:City", which indicates that the subject of the statement ":Madrid" belongs to the class ":City" (i.e., is an instance of the class ":City"). In the second example, ":Madrid :capitalOf :Spain" is combined with the property ":capitalOf rdfs:range :Country" to compute ":Spain a :Country", which indicates that the object of the statement ":Spain" is an instance of the class ":Country". In the third example, ":Madrid :capitalOf :Spain" is combined with the property ":capitalOf rdfs:subPropertyOf :locatedIn" to compute ":Madrid :locatedIn :Spain", which indicates that the subject ":Madrid" has the property ":locatedIn" with respect to the object ":Spain".

[0236] Each additional computation statement (ie, reification statement) may be added to the subset of the directed graph before the subset of the directed graph is received and before the string is computed.

[0237] Continuing with the example, given the statements and properties before materialization, the following SPARQL query will return false:

[0238] ASK{ :Madrid a :City .}

[0239] After materialization, the same SPARQL query will return true. The materialized directed graph can increase the effectiveness of the computed strings in training a language model, given the additional reasoning provided and the logical connections created between statements. Furthermore, the power of the language model reasoning can increase with the level of detail of the input provided to the language model. Thus, because the materialized directed graph increases the level of detail in the directed graph, strings computed from the materialized directed graph can train a language model more effectively than strings computed from a directed graph that has not been materialized.

[0240] When statements are inserted into a directed graph, for example, before defining the plurality of conditions and the plurality of patterns, the directed graph may be concretized. Thus, the step of receiving one or more rules corresponding to a subset of the directed graph and the subsequent concretization step may be performed before defining the plurality of conditions and the plurality of patterns. This may result in faster computation of strings from the subset of the directed graph because concretization has already been performed.

[0241] Alternatively, the step of materializing the directed graph may be performed on a subset of the received directed graph. This may have the advantage of providing better performance without using materialization, or may improve the efficiency of creating the directed graph.

[0242] Additionally or alternatively, the method may further include classifying a subset of the directed graph so that nodes are grouped together with their neighbors. Classification may be performed after the subset of the directed graph is concretized. Classification may include determining a list of nodes in the subset of the directed graph and adding randomly selected nodes to a new list of nodes. For each node in the new list of nodes, connected nodes are determined. For each of the connected nodes, if the corresponding node is in the subset of the directed graph, the corresponding node is added to the list of nodes. The method may further include removing nodes from the list of nodes.

[0243] The pseudocode for the classification algorithm described in the previous paragraphs is provided below:

[0244] graphNodeSet = graph.getAllNodes()

[0245] backlog = []

[0246] WHILE graphNodeSet NOT EMPTY

[0247] IF backlog IS EMPTY

[0248] / / perform a random pick / jump in the graph

[0249] backlog.add(graphNodeSet.randomDraw())

[0250] END IF

[0251] FOR r IN backlog DO

[0252] / / cover the full neighborhood of n

[0253] FOR n IN r.connectedTriple DO

[0254] serialize(r, n.edge, n.node)

[0255] IF n.node IN graphNodeSet

[0256] backlog.add(n.node)

[0257] END IF

[0258] END FOR

[0259] graphNodeSet.remove(r)

[0260] END FOR

[0261] The serialize() function above can verbalize the corresponding triples ("r" - subject, "n.edge" - object, "n.node" - predicate). In other words, the serialize() function can convert statements from a subset of the directed graph into a serialization format such as RDF / XML, RDFa, Notation3 (.n3) or Turtle (.ttl), N-Triples, or JSON-LD.

[0262] The classification algorithm can be called a clustering algorithm and can ensure topicality, that is, the calculated strings are close to each other in the sense that they refer to similar topics or the same topic. In other words, adjacent calculated strings are semantically similar. This can increase the effectiveness of the strings when training a language model.

[0263] The method may also include training a language model using the calculated string.

[0264] For example, providing the calculated string as input to the language model may include training (e.g., further training) the language model using the calculated string. For example, the language model may be pre-trained or extensively trained, but the training may not include data in the subset of the knowledge graph. Thus, training the language model using the calculated string may expand the capabilities of the language model and enable the language model to assist with tasks related to data in the subset of the knowledge graph.

[0265] Training a language model using a subset of a directed graph can have the advantage of leveraging or extending a lot of the effort and expense that goes into language models. For example, training a large language model (such as ChatGPT, PaLM, Megatron, Titan, or Chinchilla) can take months and cost tens of millions of dollars or euros. Enabling a large language model to apply information from a subset of a directed graph can be a way to further leverage the effort and expense that has been invested in training a large language model.

[0266] Furthermore, training a language model using a subset of the directed graph can involve fine-tuning the language model (e.g., by applying low-rank adaptation) to optimize the language model for a task or domain (e.g., the domain of the subset of the knowledge graph).

[0267] For further information on low-rank adaptation, see "LoRA: Low-Rank Adaptation of LargeLanguage Models", Edward Hu et al., June 17, 2021.

[0268] In some cases, the language model is a probability distribution over a sequence of words. The language model can be a large language model, for example, having at least one million parameters or at least one billion parameters.

[0269] In some cases, the language model includes a neural network. The neural network can be a deep neural network, for example, a neural network with one or more hidden layers. The neural network can have at least one million parameters (e.g., weights and biases) or at least one billion parameters. The neural network can be trained on unlabeled (i.e., unannotated) text using unsupervised (i.e., self-supervised) learning.

[0270] Additionally or alternatively, the neural network can include a transformer that uses self-attention to differentially consider the importance of each portion of input data provided to the neural network. The input to the neural network can be parsed into tokens, and the tokens can be processed simultaneously by calculating weights for the tokens in successive layers of the neural network. The neural network can be designed to process sequential input data. The neural network can include weights (e.g., soft weights) that can be changed during runtime.

[0271] According to another aspect, a computer program (eg a computer program product) is provided. The computer program comprises instructions which, when executed by a computer, cause the computer to perform a method according to any one of the preceding claims.

[0272] According to yet another aspect, a computer readable medium stores a computer program. For example, the computer program may be tangibly embodied in a computer readable medium. In other words, the computer readable medium may be a non-transitory storage medium.

[0273] According to another aspect, a computer system for providing data from a directed graph to a language model is provided. The system includes a database storing the directed graph. The system also includes a software service configured to define a plurality of conditions and a plurality of patterns. Each condition has at least one corresponding pattern. The software service is also configured to receive a subset of the directed graph from the database. The subset of the directed graph includes a plurality of statements. Each statement includes a subject, an object, and a predicate that relates the subject to the object. For each statement in the subset of the directed graph, the software service is configured to perform the following operations: when one of the conditions matches the corresponding statement and the pattern corresponding to the condition can be applied to the corresponding statement, a string is calculated from the corresponding statement using the pattern. The software service is also configured to provide the calculated string as input to the language model.

[0274] The software service may be a network service. The network service may run on a server and listen for network requests on a port (eg, port 80).

[0275] The subject matter described in the present disclosure may be implemented as a method or on a device, and may be implemented in the form of one or more computer programs (eg, computer program products). Such computer programs may enable a data processing device to perform one or more operations described in the present disclosure.

[0276] The subject matter described in the present disclosure may be implemented in a data signal or on a machine-readable medium, wherein the medium is embodied in one or more information carriers, such as a CD-ROM, a DVD-ROM, a semiconductor memory or a hard disk. In particular, the disclosed subject matter may be tangibly embodied in a non-transitory machine (computer) readable medium.

[0277] In addition, the subject matter described in this disclosure can be implemented as a system including a processor and a memory coupled to the processor. The memory can encode one or more programs to cause the processor to perform one or more of the methods described in this application. The other subject matter described in this disclosure can be implemented using various machines.

[0278] The details of one or more embodiments are set forth in the exemplary drawings and description below.Other features will be apparent from the description and drawings.

[0279] Hereinafter, a detailed description of the examples will be given with reference to the accompanying drawings. Various modifications may be made to the examples. In particular, one or more elements of one example may be combined and used in other examples to form new examples.

[0280] Figure 1 A subset 100 of a directed graph is shown. More specifically, Figure 1 A subset 100 of a knowledge graph is shown. Nodes of a directed graph are shown as circles, and edges of a directed graph are shown as arrows. The subset 100 of the directed graph includes labels 102, 104, 106, 108, 110, where labels are URIs and are defined in the Resource Description Framework (RDF). Node labels 102 and 104 are objects, and edge labels 106, 108, 110 are predicates. The string "1972-01-01" can also be a node label (i.e., an object) with an xsd:date type.

[0281] The subset 100 of the directed graph includes a statement 112 (i.e., a triple statement) having a subject "dbr:SAP_SE", a predicate "dbo:foundationPlace", and an object "dbr:Germany", each of which is a URI defined in RDF. An exemplary serialization of the statement 112 is dbr:SAP_SE dbo:foundationPlace dbr:Germany. The schema of the directed graph may be defined via the RDF Schema (RDFS) from the World Wide Web Consortium (W3C) or the Web Ontology Language (OWL).

[0282] Figure 2 The domain 201 and range 203 of the attribute ":capitalOf" are shown. The domain 201 and range 203 may be defined as follows:

[0283] :capitalOf rdfs:domain :City .

[0284] :capitalOf rdfs:range :Country .

[0285] Figure 3 An exemplary SPARQL query 301 of a knowledge graph is shown. Query 301 is configured to determine an answer to the following question: What is the population of Chinese cities with more than 100,000 residents? The results of executing query 301 are also shown.

[0286] Figure 4 The block diagram representation used in the subsequent figures is shown. System component 401 is shown as a rectangle. System components can also be shown as ovals. Storage device 403 is shown as a cylinder with an oval outline. Requests between system components 405 are shown as arrows including circles. Writes between system components and storage devices are shown at 407. Reads between system components and storage devices are shown at 409. Reads and writes between system components and storage devices are shown at 411.

[0287] Figure 5 A duck-foot notation for annotating entities and cardinalities (eg, minimum and maximum multiplicity) is shown.

[0288] Figure 6 The architecture of a system for providing data from a directed graph to a language model is shown. The language model can be trained using data from a directed graph. Figure 4 In the representation, user interface and software components are shown as rectangles, and storage components are shown with rounded corners.

[0289] The system may take as input a subset 100 of a directed graph (or a reference to a subset 100), custom conditions and custom patterns, and configuration options. The subset 100 may be provided as a set of triple statements. The pattern and configuration storage 609 may store default patterns, while custom conditions and patterns are provided by the client 601.

[0290] Configuration options may include an indication of whether the question pattern should be used in addition to or exclusively to the text pattern. Configuration options may specify how multiple patterns corresponding to a condition will be handled:

[0291] RUN_ALL: If the condition is true, all patterns assigned to the condition are applied;

[0292] RUN_RANDOM: The patterns are ordered randomly, and the first pattern that can be applied to the statement is used.

[0293] Another configuration option can specify whether directed graphs should be materialized (default is true). Other ways of handling multiple modes and other configuration options are also available.

[0294] The directed graph may be materialized and the statements of the directed graph may be classified, for example, by the generation agent 611. Subsequently, a string may be calculated from the statements of the subset 100 of the directed graph, such as by combining Figure 7 discussed.

[0295] Client 601 can be used to interact with software service 603. Client 601 can interact with software service 603 via different user interfaces (UIs) 605 and 607 to maintain patterns and / or configurations in pattern and configuration storage 609, or to calculate strings from subset 100 of a directed graph via generation agent 611. The pattern and configuration storage can be accessed via pattern maintenance and access application programming interface (API) 613. Subset 100 can be the entire directed graph or a suitable subset of the directed graph identified via a query (e.g., a SPARQL query). The SPARQL query can be constructed via a user interface of an abstract query language (e.g., a low-code or no-code platform).

[0296] The directed graph may be stored in storage 615 , and strings computed from the statements of subset 100 of the directed graph may be stored in storage 617 .

[0297] Figure 7 An algorithm for computing strings from statements (i.e., triples) in a subset 100 of a directed graph is shown. In the depicted example, the directed graph is sorted before computing the strings. Before computing (i.e., generating) the strings from the default mode, it is first determined whether the conditions corresponding to the specific (i.e., custom) mode match the statements. If the conditions corresponding to the specific mode match the statements, the specific mode is used to compute the strings. Otherwise, the default mode is used to compute the strings.

[0298] When multiple specific and default patterns exist, then the specific patterns may be checked first, and only if no specific pattern applies may the string be evaluated using one of the default patterns.

[0299] After the string is calculated, the string can be reconstructed using a reconstruction language model. The reconstruction language model can be different from the language model to be trained. The reconstruction language model can be a language model with high or very high accuracy for f in the following reconstruction function f:

[0300] f(sentence) = sentence'

[0301] A reconstruction language model (also known as a paraphrase language model or encoder-decoder model) can be implemented using Google T5, FLAN-T5, or Quillbot. The reconstruction language model can have high accuracy (e.g., at least 90% correct) or very high accuracy (e.g., at least 99% or at least 99.9% correct). The reconstructed string can have a greater degree of language variation than the original calculated string. The reconstructed string can be provided to a user and can retain the user's attention better than the original calculated string. Alternatively, the reconstructed string can be provided to a language model to be trained. When training the language model, the language variation in the reconstructed string can produce better results than the original calculated string.

[0302] The reconstructed strings may sometimes be incorrect. Therefore, a human may have the option to accept or reject the reconstructed strings. The accepted and / or rejected reconstructed strings may be used to retrain the reconstruction language model. Additionally, the accepted and / or rejected reconstructed strings may be used to compare the quality of different reconstruction functions f.

[0303] Figure 8 An exemplary classification algorithm for classifying sentences of a directed graph is shown. Classification can prevent the calculated strings from being output in a random order. Once the sentences are classified, the same or similar subjects can be grouped together. The classification algorithm can have the following form:

[0304] List <triple>result=sort(Set <triple>)

[0305] Set <triple>corresponds to the statements in the directed graph before classification, and List <triple>result corresponds to the statements in the directed graph after sorting. "sort" calls the sorting function, such as Figure 8 Other classification functions may also be used.

[0306] The graphNodeSet variable initially holds the unclassified statements of the directed graph, and the backlog variable will contain the classified directed graph when the classification algorithm is completed.

[0307] Fig. 9 shows that the input can be provided to Figure 8 The directed graph of the classification algorithm. Assuming that node 1 is drawn first, the output will be the following statement:

[0308] 1 a 2

[0309] 1 b 3

[0310] 1 c 4

[0311] 2 d 6

[0312] 7 e 8

[0313] Each number above represents a node, and each letter represents an edge, such as Fig. 9 Thus, the classification algorithm ensures that nodes that are close together in the directed graph are also grouped together in the computed strings provided as output, rather than being output randomly. In other words, after computation, adjacent strings cover semantically similar content.

[0314] Fig.10 use Figure 5 The duck-foot notation is used to show that a condition has at least one but can have multiple corresponding patterns (also called text patterns). When provided with a statement (e.g., an RDF triple), the condition can evaluate to true or false. If the condition evaluates to true, i.e., the condition matches the corresponding statement, it is determined whether at least one pattern corresponding to the condition can be applied to the corresponding statement. If the pattern can be applied, a string is computed from the corresponding statement using the pattern.

[0315] Fig.11 Extensions Fig.10 to cover question modes other than text mode. Therefore, Figure 5 The duck-foot notation is used to show that a condition has at least one but can have multiple corresponding text patterns. Additionally, a condition can have zero or more question patterns. The syntax of question patterns can be the same as the syntax of text patterns.

[0316] The question pattern can enable the computation of conversation-like strings. Such conversation-like strings can be particularly useful for some language models (e.g., language models that require conversations).

[0317] Fig.12 Extensions Figure 7 to override the problem mode. In particular, if the configuration option that enables the use of the problem mode is set, a specific condition corresponding to the problem mode is checked. If the problem mode corresponding to the specific condition can be applied, the problem mode is used to calculate the string from the corresponding statement. Otherwise, the default mode is used to calculate the string from the corresponding statement.

[0318] The configuration options for specifying how multiple patterns corresponding to a condition will be handled can be extended to the following problem patterns. Specifically, the following three configuration options can be used to handle multiple patterns assigned to at least one condition:

[0319] RUN_ALL: If the condition matches the statement and multiple problem patterns are assigned to the condition, all the problem patterns that can be executed are executed. If there are multiple text patterns and problem patterns, the Cartesian product of the text pattern and the problem pattern is executed.

[0320] RUN_RANDOM: If the condition matches the statement, a random order of all available problem patterns assigned to the condition is determined. The execution of the problem patterns is then tested in a top-down manner. The first problem pattern that can be executed is executed, and the process stops.

[0321] RUN_ALL_QPATTERNS_RANDOM_TEXT_PATTERN: Use all question patterns, but if multiple text patterns exist, only use the random text pattern to generate the answer.

[0322] Other ways of handling multiple patterns assigned to a condition may also be used.

[0323] Fig.13 A remote service configuration and remote storage (i.e., a cloud implementation) of a method for providing data from a directed graph to a language model is shown. A network client 130 connected to an organization's local area network 132 can connect to a remote service, i.e., a graph service 134, via the Internet. The local area network can use Ethernet and / or Wi-Fi to provide network communications. The local area network can be contrasted with a wide area network. The graph service 134 can use a storage device 136 to perform the steps of the method for providing data from a directed graph to a language model. The network client 130 can initiate the method for providing data from a directed graph to a language model via a method call to the graph service 134. In response to the method call, the graph service 134 can connect to the storage device 136 and receive a subset 100 of the directed graph.

[0324] Then, the graph service 134 can apply the pattern to the corresponding statement of the subset 100 of the directed graph based on whether the condition corresponding to the pattern matches the corresponding statement, and can apply the pattern to the corresponding statement. If the pattern can be applied, the string is calculated from the corresponding statement using the pattern. After all the statements in the subset 100 of the directed graph have been processed, the graph service 134 can provide the calculated string to the network client 130.

[0325] Graph service 134 and storage 136 may be part of a cloud computing environment. Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with a service provider.

[0326] A cloud computing environment (i.e., a cloud environment or a cloud) can have one or more of the following characteristics: scalability, multi-tenancy, performance monitoring, virtual resources that can be dynamically allocated to different users based on demand, multiple redundant sites, multiple virtual machines, and network accessibility (e.g., via the Internet) from multiple locations (e.g., via a web browser) and devices (e.g., a mobile device or a PC).

[0327] A cloud computing environment can have a higher ratio of virtual resources to physical resources (e.g., a higher ratio of virtual machines to physical machines) than a local computing environment. For example, in a cloud computing environment, the ratio of virtual resources (e.g., machines) to physical resources can be at least 10:1, at least 20:1, or at least 30:1. In contrast, a local computing environment can have fewer than four virtual resources (e.g., machines) per physical resource.

[0328] A cloud environment can be a public cloud or a private cloud. Public cloud (computing) infrastructure can involve sharing hardware, storage and / or network resources between multiple organizations or tenants. Services can be accessed and managed using a web browser. Private cloud (computing) infrastructure can include resources used exclusively by one organization or a group of users. Compared to public cloud computing infrastructure, private cloud infrastructure can provide more flexibility and control, however, private cloud infrastructure can be more expensive. In both cases, public and private cloud computing infrastructure can be hosted by a service provider, for example, Microsoft (Azure), Amazon (AWS) or SAP Business Technology Platform.

[0329] Fig.14 A hybrid implementation of a method for providing data from a directed graph to a language model is shown. Thus, the network client 130 still accesses the graph service 134 over the Internet. However, the storage device 136 is located locally 132 on the organization. For example, the storage device 136 may be connected to the network client 130 via a local area network. Alternatively, the network client 130 and the storage device 136 may be connected to different local area networks. The graph service 134 may be configured as described above. Fig.13 Operate as described.

[0330] Fig.15 A local configuration is shown, where the network client 130, the graph service 134, and the storage device 136 are located locally 132 in the organization. In this case, the network client 132, the graph service 134, and the storage device 136 can be connected to the same local area network. Other network configurations are also possible. The graph service 134 can be as described in detail in relation to FIG. Fig.13 Operate as described.

[0331] Example 3 - Example Supplementation of User Input to a Large Language Model

[0332] Fig.16 Illustrations are provided of how the disclosed techniques can be used to influence the responses of a large language model, including by providing information unknown to the large language model or constraining the large language model to use specific information identified from another data source from user input, regardless of whether that information has typically been used in a training corpus for the large language model.

[0333] Consider user input 1610 in the form of a question about the meaning of a particular document code used by a particular company. At 1614, the user input 1610 is submitted to a large language model 1618 trained on a training dataset (which may also be referred to as a training corpus) 1622. Assume further that the information required to respond to the user input 1610 is not part of the training dataset 1622, such as because the information required to respond is located in a document internal to the company that is not available in the training dataset.

[0334] In many typical scenarios, large language model 1618 may indicate that it cannot answer the question, or it may attempt to answer the question but provide "made-up" / inaccurate information in the response, which may be referred to as hallucination. Specifically, response 1630 illustrates a scenario where large language model 1618 recognizes that it does not have enough information to answer the question of user input 1610.

[0335] Response 1634 illustrates a scenario where large language model 1618 does not recognize that it does not have enough information to answer the question. For example, assume that the document code D12345 provided in user input 1610 refers to a marketing campaign analysis. Large language model 1618 may use "creative fill" or other processes to incorrectly indicate that the document code is an internal financial report. In some cases, this may be a "reasonable" conclusion for large language model 1618 based on other information in training data set 1622, even if it is actually incorrect.

[0336] Fig.16 It is shown how the disclosed techniques can be used to supplement or emphasize the data available to the large language model 1618 so that the large language model can formulate an accurate and helpful response. In particular, consider another example of user input 1610, which is indicated as user input 1650. Rather than being submitted directly to the large language model 1618, the user input 1650 is first analyzed and used to identify additional information to be included in a modified version of the user input based on the information in the supplemental data set 1654.

[0337] In this particular example, supplemental dataset 1654 may represent a data source internal to a particular company. While big language model 1618 may also be company private, the big language model may be a publicly available big language model, or a version of a publicly available big language model specifically for use by the company. A private version of a publicly available big language model may be useful for reasons such as maintaining confidentiality of information in user input 1610, information in supplemental dataset 1654, and information in response to user input. In some cases, training dataset 1622 may also have information that is different from the publicly available version of big language model 1610 or other "private" versions of a publicly available big language model.

[0338] The supplemental data set 1654 may be maintained in a variety of formats, and various techniques may be used to search the supplemental data set and provide information from the supplemental data set for use with the user input 1650. A specific example, which will be further described, is the use of a named entity recognition service to identify entities in the user input 1650, where these entities are then used to search for supplemental information in a knowledge graph, such as a knowledge graph using a resource description framework (RDF). As will be further described, the RDF format expresses relationships in a knowledge graph in the form of triples representing a combination of a subject, an object, and a predicate. Searching for relevant information may also use an ontology, such as an ontology represented in the Web Ontology Language (OWL). An ontology may be used to identify relationships between different entities, as well as relationships between relationship (predicate) types.

[0339] Information from supplemental dataset 1654 may be added to user input 1650 to provide modified user input 1670. In some cases, information from supplemental dataset 1654 may be processed or formatted before being added to user input 1650. For example, rather than providing "raw" triples for results from a knowledge graph, triples may be "verbalized," such as using the techniques described in Example 2.

[0340] The modified user input 1670 is then provided to the large language model 1610 to provide a response 1674. The response 1674 is based at least in part on the supplemental information in the modified user input 1670. The response 1674 may optionally include / be based on information in the training data set 1622. For example, the response 1674 may include facts or explanations derived at least in part from the training data set 1622.

[0341] However, even if any factual or explanatory information in response 1674 is described solely based on supplemental information in modified user input 1670, use of large language model 1618 is still beneficial, such as because training data set 1622 provides information that can be used to process and respond to modified user input, such as grammar-based "understanding," which helps large language model 1618 "understand" modified user input 1670 and provide response 1674 in an easily understandable, grammatically correct format.

[0342] Note that the conversation represented in panel 1680 contains original user input 1650, rather than modified user input 1670, in addition to response 1674. The conversation does not contain any information from supplemental dataset 1654, except to the extent such supplemental information is incorporated into response 1674 using large language model 1610. Thus, the process of identifying supplemental information using user input 1650, generating modified user input 1670, and submitting the modified user input to large language model 1618 is "invisible" to the user. From the user's perspective, it is as if they simply received response 1674 from large language model 1618 based on their original user input 1650.

[0343] Example 4 - Verbalizing knowledge Figure 3 Example procedure for tuple complementing user input

[0344] Fig.17 Provides information about Fig.16 A more formal explanation of the described scenario is provided in a flow chart of a process 1710 for providing answers to user-posed questions in a prompt to a large language model. The description of the process uses notation 1704.

[0345] At 1714, a set of entities E related to a particular user input I is determined. For example, the set of entities E can be determined by submitting the user input I to a named entity recognition service. A named entity recognition (NER) service is a natural language processing technique that identifies and classifies specific entities within a given text, such as a person's name, location, organization, date, etc. The NER service uses machine learning and language patterns to extract these entities and classify them into predefined categories. Examples of NER services include spaCy, Stanford NER, OpenNLP, and Google Cloud Natural Language API. The identified entities (optionally using a specific ontology) can then be used to search data sources such as semantic frameworks (e.g., knowledge graphs).

[0346] At 1718, based on the results of searching a dataset (such as a knowledge graph), a set of triples (such as in the format of subject, object, predicate) are compiled based on the identified entities E for answering I. At 1722, the triples are verbalized, such as using the techniques described in Example 2. The verbalized representation of the triples can be more useful for a large language model in providing one or more grammatically correct fact sentences S.

[0347] At 1726, a prompt P is constructed using a set of grammatically correct fact sentences S. For example, the sentences S may be appended to the original user input I to provide the prompt P. As will be further described, additional instructions may be added to the prompt P, such as instructions that affect how the large language model answers, such as providing a short response or a lengthy response.

[0348] At 1730, an answer A is generated for prompt P, such as by submitting prompt P to a large language model. Optionally, at 1734, answer A can be processed to link words or phrases in the answer to relevant entities is a knowledge graph G (which can be the same knowledge graph used at 1718, or a different semantic graph (or, more generally, one or more semantic frameworks can be used to determine entity E in user input I, and one or more semantic frameworks can be used to determine entities in answer A, where all or a portion of the semantic frameworks can be different for these two usage scenarios). Linking can similarly be accomplished using a named entity recognition service in a manner similar to that described at 1714. At 1738, answer A' is provided to the user in response to input I. For example, answer A' can be displayed, or answer A' can be sent to a component for display to the user. If no linking operation occurs, or if no entity to be linked is found or the linking criteria is not met, then answer A' can be answer A.

[0349] Example 5 - Example construction of modified user input, large language model response, and linking of entities and supplementary information in the large language model response

[0350] Fig.18A Provided Fig.17 An example of certain aspects of process 1700 is provided. In particular, Fig.18A 18 shows how to generate a prompt 1810 corresponding to the modified user input. As shown, the prompt 1810 consists of a knowledge intent 1814, which can correspond to the user input provided initially (identifying what knowledge the user wants to gain by submitting the input to the large language model), Fig.17 1704. The prompt 1810 also includes an invisible fact task 1818. The fact task corresponds to the result of searching the knowledge graph using a named entity recognition service for entities identified in the processing of the knowledge intent 1814. In a specific example, the result of the invisible fact task 1818 is a set of ordered triples, such as in a subject, object, predicate format.

[0351] The triples of invisible fact task 1818 are processed to provide a set (or list) of linguisticized triples (such as using the techniques of Example 2), which are used as invisible facts 1822 that form part of prompt 1810. Optionally, prompt 1810 can include invisible commands, such as close command 1826. In particular, close command 1826 can be used to restrict what data the large language model uses when generating an answer to prompt 1810, or to provide input to guide how a result should be generated or presented.

[0352] FIG. 18 provides a specific example of a prompt 1810 and its components 1814, 1818, 1822, 1826. In particular, a panel 1840 provides an example knowledge intent 1842. The panel 1840 includes a specific set of invisible facts 1846 generated from triples identified in the invisible facts task of the knowledge intent 1842. Generating the set of invisible facts 1846 may involve using a named entity recognition service to identify entities in the knowledge intent 1842. An example pseudo code for performing this operation may be:

[0353] METHOD NER(String knowledgeIntent) RETURNS List <uri>

[0354] In this case, the value of knowledgeIntent will be knowledge intent 1842, and List will hold the URI (Unique Resource Identifier) ​​of the recognized entity "Customer Invoice Bill (W99)". The URI may correspond to the identifier of the node of the knowledge graph corresponding to "Customer Invoice Bill (W99)". As a specific example, assume that the invocation of the METHOD NER of the example shown provides the result:

[0355] http: / / www.signavio.com / opal / SAP / CPM / BPX / Customer Invoice Billing(W99)

[0356] The above URI can then be used in a query to the knowledge graph to retrieve the relevant triples. In particular, the query can be a SPARQL query of the knowledge graph represented in RDF. Note that various configuration information or constraints can be provided for the query. For example, in many cases, large language models are able to accept a limited number of tokens in a single input prompt. Therefore, the results of the knowledge graph can be subject to a threshold, where results exceeding the threshold are not provided in the prompt 1810, or various criteria are used to select specific results in the results to include in the prompt up to the threshold, such as by weighting specific entities or selection mechanisms.

[0357] One or more indirection parameters may also be defined for a query. For example, an ontology may be used to relate various entities or entity relationships in a knowledge graph. An ontology may be set to use no indirection or to use a specified level of indirection. Similarly, instead of retrieving information for only a specific URI from the knowledge graph, multiple "hops" may be specified. These configurations may be dynamic. For example, an initial number of results may be analyzed. If the number of results is less than a threshold, the level of indirection in one or both of the ontology or the knowledge graph may be increased. Fig.21 An example knowledge graph and the use of hops are further described.

[0358] The search process can be subject to other types of constraints or configurations. For example, portions of the knowledge graph can be prioritized for search based on metadata associated with the user, such as the user's job function or the project to which the user is currently assigned. The knowledge graph can also be subject to authorization requirements, and in at least some cases, triples in portions of the knowledge graph for which the user does not have appropriate access rights can be excluded from search results.

[0359] In addition to the original knowledge intent 1842 and the invisible fact 1846, the window 1840 representing the prompt P created from the original prompt including the original knowledge intent may include instructions 1852, 1854, 1856. In general, the instructions can be used to affect how the large language model responds to the modified prompt, and potentially affect the later interaction with the user. General types of instructions that can be provided include contextual instructions, content constraint instructions, formatting instructions, source simulation instructions, creative instructions, question clarification instructions, interpretation instructions, contrast instructions, synonym / paraphrase instructions, or humor instructions.

[0360] For example, instruction 1852 instructs the large language model to treat invisible fact 1846 as new information. Instruction 1852 may be useful so that the large language model does not provide a response such as "as you just told me" or "as you told me previously." That is, in at least some embodiments, the information injected into the modified prompt is intended to be invisible to the end user. That is, from the user's perspective, the large language model is responding to the user's original prompt, not the modified prompt. In other cases, the modified prompt is not hidden from the user.

[0361] Instructions 1854 direct the large language model to generate responses only from unseen facts 1846, rather than information that may be generated based on its training material. Instructions 1856 direct the large language model to provide brief responses. Instructions 1856 may further constrain the large language model to unseen facts 1846, which may reduce the chance of the model "hallucinating." Other instructions may be used to prevent users from trying to get the large language model to say negative things about a particular company, such as a company whose semantic framework is being used to supplement the original user input / prompt.

[0362] Panel 1870 represents the initial response provided by the large language model to the modification prompt of panel 1840. It can be seen that the response is generated only based on invisible facts 1846, which are described in an easily understandable form based on the knowledge of grammar, semantics, context understanding, natural language generation, and transfer learning of the large language model.

[0363] Optionally, the initial response can be further processed. For example, the initial response can be processed by a named entity recognition service to identify entities represented in the knowledge graph. The initial response can then be modified so that aspects of the response are linked to information in the knowledge graph for such entities, such as shown in panel 1880. In particular, "Customer Invoice Bill (W99)" and "Scope Project" are both underlined in panel 1880. By selecting those phrases, additional information can be provided to the user, such as information retrieved from the knowledge graph based on the entities associated with the corresponding phrases.

[0364] Fig.18B and 18C Provided are example user interface screens 1886, 1888 that may be displayed if the user selects "Customer Invoice Bill (W99)" in panel 1880. Specifically, Fig.18B User interface screen 1886 provides a textual summary of the related entities, including the attributes of the entities. User interface screen 1888 provides a visual representation 1890 of the immediate portion of the knowledge graph surrounding the node of the "Customer Invoice Bill (W99)" entity.

[0365] Note that the information linked to a particular entity in the response can be information in the knowledge graph or in addition to information in the knowledge graph. For example, a mapping between a particular entity in the knowledge graph and specific information to be linked to the entity can be provided. In other cases, specific information to be provided for an entity via a link in the response of a large language model can be defined in the knowledge graph. For example, an attribute of an entity can be defined, where the attribute specifies a URI to be used with the link.

[0366] Fig.19 An example query 1910 for this example is shown. As can be seen, query 1910 is limited to providing 20 results. Fig.19 The query results 1916 are also listed in the form of subjects 1920, objects 1924, and predicates 1928 resulting from the query (where each correspondingly numbered subject, object, and predicate corresponds to a discrete query result). A set 1934 of triples may be defined, as shown at 1932. In this example, the query results may represent results within a one-hop distance of the identified named entity.

[0367] Fig. 20 A linguistic function 2010 is shown that can be used to process the set 1934 of triples. The linguistic function 2010 can be implemented using the techniques described in Example 2. The output of the linguistic function 2010 is shown as a set 2020 of linguistic responses corresponding to each triple in the set 1934 of triples. The linguistic responses of the set 2020 convey the information of the triples in the set 1934 in a format that is more understandable to humans, which is also a format that can be more easily processed by a large language model.

[0368] Example 6 - Example Knowledge Graph

[0369] Fig.21 A simple knowledge graph 2100 is shown for demonstrating how to determine query results using initially defined entities (such as entities identified using a named entity recognition service) and a particular number of hops allowed for the query.

[0370] Graph 2100 is formed by nodes 2110 representing entities (which may be subjects or objects of triples), and where edges 2114 represent specific relationships (predicates) between two entities.

[0371] Nodes 2110 may include nodes that may be associated with more general information that is related to nodes that provide more specific information. For example, node 2110a may represent a book. A book may have specific characteristics or attributes, such as node 2110b representing an author attribute. Although not shown, a book may have other attributes, such as a title, a year of publication, or a category or classification (such as fiction or non-fiction). Edge 2114a may indicate the nature of the relationship between book node 2110a and author attribute 2110b. For example, the relationship may be "has attribute" from the "viewpoint" of node 2110a, or may be an opposite relationship from the viewpoint of node 2110b, such as "attribute of..."

[0372] Another type of general-specific relationship is in Fig.21 , because the book node 2110a and the author node 2110b can have specific instances. That is, the node 2110a can represent the general concept of a book, for example, while the nodes 2110c, 2110d, and 2110e are specific instances of a book. From the perspective of the book node 2110a, the corresponding edges 2114b, 2114c, 2114d can represent "having an instance", where the opposite relationship can be "an instance of...". Similarly, the nodes 2110f, 2110g of the author node 2210b can represent specific authors. The book instances 2110c, 2110d, 2110e can then be linked to their specific author instances 2110f, 2110g, where the corresponding edges 2114e, 2114f, 2114g can represent a "having an author" relationship from the perspective of the book instance, or a "author of..." relationship from the perspective of the author instance.

[0373] Now, assume that a query is received based on a named entity recognition search that identifies entity 2110d. A 1-hop limit may result in the identification of nodes 2110a and 2110f, so book instance 2110d may be identified as an instance of book entity 2110 and authored by the author of node 2110f. However, a one-hop limit would not identify that author 2110f also wrote book instance 2110c. If the hop limit were greater than one, this information would be identified.

[0374] Example 7 - Provides sample code for searching the knowledge graph for nodes related to a specified entity

[0375] Figures 22A to 22C Provides example pseudo code that can be used to implement the disclosed technology. In particular, Fig.22A Illustrated is a function 2204 that may be used to call a named entity recognition service to return the URI of an entity recognized in a particular user input intended for use with a large language model.

[0376] Code 2208 includes functions 2210 and 2212 designed to retrieve triples related to a specified entity identified by function 2204. Function 2210 collects triples where the entity is used as a subject, and function 2212 identifies triples where the entity is used as an object. Code 2208 corresponds to a one-hop search.

[0377] exist Fig. 22B , codes 2218 and 2220 show how triples that are more than one hop away from a specified entity can be identified. It does this by calling functions 2210 and 2212 from code 2208 during the white loop, which in turn identify triples of the initial entity and the entity within a specific number of hops. The "depth" variable in codes 2218 and 2220 controls the scope of the traversal. As shown, setting it to 1 results in a single iteration of the loop, effectively mirroring the functionality of functions 2210 and 2212. Larger "depth" values ​​provide more results by expanding the traversal.

[0378] Fig. 22C Code 2230 identifies both subject triples and object triples for a given set of entities. Code 2230 operates within a specific jump level determined by the value of the "depth" variable. Initially, code 2230 retrieves triples in which the entity is used as a subject. If the total number of triples collected so far is below a predefined "threshold", code 2230 continues to retrieve triples in which the entity is used as an object. At this stage, the jump level is set to 1.

[0379] Subsequently, code 2230 enters a loop that continues as long as the number of collected triples is below a threshold. Within the loop, the code iterates over each entity in the given set and retrieves triples in which these entities are used as subjects. If the threshold is still not met, the code retrieves triples in which the entity is used as an object. After each iteration, the depth variable is incremented, allowing traversal to deeper levels of the graph.

[0380] Example 8 - Example computing environment with an orchestrator for modifying user input to a large language model

[0381] Fig.23 An example computing environment 2300 is provided in which the disclosed technology can be implemented. The computing environment 2300 includes a client 2304 interacting with a computing system 2308 that is configured to provide a large language model response to input received from the client 2304.

[0382] Computing system 2308 includes orchestrator component 2312. Orchestrator component 2312 may be responsible for receiving user input from client 2304, providing answers back to the client, and invoking other components of computing system 2308 to generate answers, such as executing Fig.17 Components of at least some operations in process 1710.

[0383] In particular, orchestrator component 2312 can be configured to provide user input to named entity recognition service 2316. Named entity recognition service 2316 can provide the recognized entities to orchestrator component 2312. Orchestrator component 2312 can then query data store 2320 to obtain information in the data store related to the entities recognized by named entity recognition service 2316. For example, orchestrator component 2312 can call interface 2324 of data store 2320 to provide a query to knowledge graph 2328.

[0384] Although this disclosure generally describes the use of a single knowledge graph, it should be understood that multiple knowledge graphs may be available, and in at least some cases, multiple graphs may be searched for entities that may be used to supplement / modify user input to a large language model. Fig.23 Multiple knowledge graphs 2328 are shown in . If multiple graphs 2328 are used, search results from multiple graphs can be combined, such as using a union operation. The hop limit for the search can be the same for the graphs 2328 used in the search, or different graphs can have different search limits. When limiting the maximum number of results to accept, a separate limit can be provided for each graph 2328 being used, or an overall limit for all graphs can be specified.

[0385] Although the graph 2328 can be located on the computing system 2308, in other cases, the computing system can access the graph 2332 of the remote system 2330. When searching multiple graphs, the graphs can be located on the computing system 2308, at one or more remote systems 2330, or a combination of one or more graphs 2328 of the computing system 2308 and one or more graphs 2332 of one or more remote systems. In addition, as described, the knowledge graph is a specific semantic framework described to illustrate the disclosed technology. Some of all knowledge graphs 2328, 2332 can alternatively be other types of semantic frameworks, and a given use case can include all of the same type of semantic frameworks, or different types of semantic frameworks can be used.

[0386] Optionally, a particular use case may be configured to use a specified one or more knowledge graphs 2328. That is, for example, the functionality for performing the disclosed innovation may be relatively standardized, where a given use of the functionality is configured by providing an identifier of one or more relevant knowledge graphs 2328, 2332 for the scenario. Configuration may also include specifying any instructions that should be added to the modified user input, including modifying any default instructions that may be provided.

[0387] The results from the query may be provided by data store 2320 to orchestrator component 2312, which may then create updated prompts that are submitted to large language model 2336. In some cases, the results from the query may be verbalized, such as into a form that conforms to the grammar of a particular human language by verbalization component 2340 of data store 2320. In other cases, verbalization component 2340 may be part of orchestrator component 2312, or may be another component (including being a subcomponent of a larger component) that is otherwise useful to orchestrator component 2312.

[0388] Orchestrator component 2312 receives answers to updated prompts and can provide answers to client 2304. In some cases, before being provided to client 2304, the answers can be processed by named entity recognition service 2316 to identify entities in the response, such as using interface 2324. Named entity recognition service 2316 then identifies related entities in knowledge graph 2328 and can modify the answers to link to such entities. Alternatively, named entity recognition service 2316 provides the identified entities to another component, such as orchestrator component 2312, and such other component can access data store 2320 to identify related entities / information and modify the answers to include links to such related entities / information.

[0389] Submitting a response to the modified user input from the large language model 2336 to the named entity recognition service 2316 may be performed by a graph linking / mapping component 2350. The graph linking / mapping component 2350 may also be responsible for inserting link functionality in the response from the large language model 2336, and optionally for processing the request in response to the user's selection of a link. Links may be established using techniques such as hyperlink markup, CSS styling, or event handling (such as using a "clickable text" class defined in a language such as JAVASCRIPT or PYTHON, or techniques similar thereto).

[0390] Example 9 - Example Operation

[0391] Fig.24 A flow chart of a process 2400 of modifying user input to a large language model to include supplementary facts for use by the large language model before submitting the user input to the large language model is provided. At 2404, user input is received from a user. The user input includes multiple word-grams. At 2408, at least a portion of the multiple word-grams is analyzed. Based on the analysis, at 2412, one or more entities represented in at least a portion of the multiple word-grams of a semantic framework are determined. At 2416, one or more triples of a semantic framework are determined for at least a portion of one or more entities or associated entities. At 2420, at least a portion of the one or more triples or their representations are added to the user input to provide a modified user input. At 2424, the modified user input is submitted to the large language model. At 2428, the modified user input is processed using the large language model to provide a response. A response is returned in response to receiving the user input at 2432.

[0392] Example 10 - Calculation System

[0393] Fig.25 A generalized example of a suitable computing system 2500 is depicted in which the described innovations may be implemented. The computing system 2500 is not intended to suggest any limitation as to the scope of use or functionality of the present disclosure, as the innovations may be implemented in various general-purpose or special-purpose computing systems.

[0394] refer to Fig.25 , the computing system 2500 includes one or more processing units 2510, 2515 and memories 2520, 2525. Fig.25 , the basic configuration 2530 is included within the dashed line. Processing units 2510, 2515 execute computer executable instructions, such as for implementing the techniques described in Examples 1-9. The processing unit may be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power. For example, Fig.25 A central processing unit 2510 and a graphics processing unit or co-processing unit 2515 are shown. Tangible memory 2520, 2525 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both, accessible by the processing units 2510, 2515. The memory 2520, 2525 stores software 2580 implementing one or more innovations described herein in the form of computer-executable instructions suitable for execution by the processing units 2510, 2515.

[0395] The computing system 2500 may have additional features. For example, the computing system 2500 includes storage 2540, one or more input devices 2550, one or more output devices 2560, and one or more communication connections 2570. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system 2500. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing system 2500 and coordinates the activities of the components of the computing system 2500.

[0396] Tangible storage 2540 may be removable or non-removable and include disks, tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory manner and that can be accessed within computing system 2500. Storage 2540 stores instructions for software 2580 that implement one or more innovations described herein.

[0397] Input device 2550 may be a touch input device (such as a keyboard, mouse, pen, or trackball), a voice input device, a scanning device, or another device that provides input to computing system 2500. Output device 2560 may be a display, a printer, a speaker, a CD burner, or another device that provides output from computing system 2500.

[0398] Communication connection(s) 2570 enable communication with another computing entity over a communication medium. The communication medium transmits information such as computer executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in a manner that encodes the information in the signal. By way of example and not limitation, the communication medium may use electrical, optical, RF or other carriers.

[0399] The innovations may be described in the general context of computer executable instructions, such as those included in program modules that are executed in a computing system on a target real or virtual processor. Typically, program modules or components include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules may be combined or split between program modules as desired. Computer executable instructions for program modules may be executed within a local or distributed computing system.

[0400] The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. In general, a computing system or computing device may be local or distributed, and may include any combination of special-purpose hardware and / or general-purpose hardware and software that implements the functionality described herein.

[0401] In various examples described herein, a module (e.g., component or engine) may be "coded" to perform certain operations or provide certain functionality, indicating that computer executable instructions for the module may be executed to perform such operations, cause such operations to be performed, or otherwise provide such functionality. Although the functionality described with respect to a software component, module, or engine may be performed as discrete software units (e.g., procedures, functions, class methods), it need not be implemented as discrete units. That is, the functionality may be incorporated into a larger or more general program, such as one or more lines of code in the larger or general program.

[0402] For purposes of presentation, the detailed description uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer and should not be confused with actions performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation.

[0403] Example 11 - Cloud computing environment

[0404] Fig.26 An example cloud computing environment 2600 is depicted in which the described techniques may be implemented. Cloud computing environment 2600 includes cloud computing services 2610. Cloud computing services 2610 may include various types of cloud computing resources, such as computer servers, data repositories, network resources, etc. Cloud computing services 2610 may be centrally located (e.g., provided by a data center of an enterprise or organization) or distributed (e.g., provided by various computing resources located in different locations, such as different data centers and / or located in different cities or countries).

[0405] Cloud computing services 2610 are utilized by various types of computing devices (e.g., client computing devices), such as computing devices 2620, 2622, and 2624. For example, computing devices (e.g., 2620, 2622, and 2624) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smart phones), or other types of computing devices. For example, computing devices (e.g., 2620, 2622, and 2624) can utilize cloud computing services 2610 to perform computational operations (e.g., data processing, data storage, etc.).

[0406] Example 12 - Implementation

[0407] Although the operations of some of the disclosed methods are described in a particular sequential order for ease of presentation, it should be understood that this description encompasses rearrangement unless the specific language set forth below requires a particular ordering. For example, the operations described in sequence may be rearranged or performed simultaneously in some cases. In addition, for simplicity, the accompanying drawings may not show the various ways in which the disclosed methods can be used in conjunction with other methods.

[0408] Any disclosed method may be implemented as computer executable instructions or a computer program product stored on one or more computer readable storage media (such as tangible, non-transitory computer readable storage media) and executed on a computing device (e.g., any available computing device, including a smartphone or other mobile device that includes computing hardware). A tangible computer readable storage medium is any available tangible medium that can be accessed within a computing environment (e.g., one or more optical media disks such as DVDs or CDs, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as flash memory or a hard drive)). As an example and with reference to Fig.25 , computer-readable storage media include memories 2520 and 2525 and storage device 2540. The term computer-readable storage medium does not include signals and carrier waves. In addition, the term computer-readable storage medium does not include communication connections (eg, 2570).

[0409] Any computer executable instructions for implementing the disclosed technology and any data created and used during the implementation of the disclosed embodiments can be stored on one or more computer readable storage media. The computer executable instructions can be, for example, a dedicated software application or a part of a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment using one or more network computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network) or other such networks).

[0410] For the sake of clarity, only some selected aspects of software-based implementations are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any specific computer language or program. For example, the disclosed technology can be implemented by software written in C, C++, C#, Java, Perl, JavaScript, Python, R, Ruby, ABAP, SQL, XCode, GO, Adobe Flash, or any other suitable programming language, or in some examples, by a markup language such as html or XML or a combination of suitable programming languages ​​and markup languages. Similarly, the disclosed technology is not limited to any specific computer or hardware type. Some details of suitable computers and hardware are known and do not need to be elaborated in this disclosure.

[0411] In addition, any software-based embodiments (including, for example, computer-executable instructions for causing a computer to perform any disclosed method) can be uploaded, downloaded, or remotely accessed via suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.

[0412] The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Rather, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and sub-combinations with each other. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments require the presence of any one or more specific advantages or problems to be solved.

[0413] The technology from any example can be combined with the technology described in any one or more other examples. In view of the many possible embodiments to which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology. On the contrary, the scope of the disclosed technology includes what is covered by the scope and spirit of the appended claims.< / uri> < / triple> < / triple> < / triple> < / triple> < / sequenceid>

Claims

1. A computing system comprising: at least one memory; one or more hardware processing units coupled to the at least one memory; as well as One or more computer-readable storage media storing computer-executable instructions that, when executed, cause the computing system to perform operations comprising: receiving user input including a plurality of word-grams from a user; analyzing at least a portion of the plurality of word-grams; determining, based on the analyzing, one or more entities of a semantic frame represented in the at least a portion of the plurality of word-grams; determining one or more triples of the semantic frame for at least a portion of the one or more entities or associated entities; adding at least a portion of the one or more triples, or a representation thereof, to the user input to provide a modified user input; submitting the modified user input to the large language model; processing the modified user input using the large language model to provide a response; and The response is returned in response to receiving the user input. 2 . The computing system of claim 1 , wherein analyzing the at least a portion of the plurality of word-grams comprises providing at least a plurality of word-grams to a named entity recognition service.

3. The computing system of claim 1 , wherein adding at least a portion of the one or more triples or a representation thereof to the user input to provide a modified user input comprises: Triples of the at least a portion of the plurality of triples are submitted to a linguisticization function to provide the representation, the representation being linguisticized triples.

4. The computing system of claim 1, wherein: The semantic framework includes a knowledge graph.

5. The computing system of claim 1 , the operations further comprising: One or more related entities of the subset of the one or more entities are identified by traversing the semantic framework through one or more levels of indirection from each respective entity within the set of related entities. The computing system of claim 5 , wherein the identifying is performed up to a specified level of indirection.

7. The computing system of claim 5, wherein: The identifying is performed until a threshold number of entities have been identified.

8. The computing system of claim 5, wherein: The triples are of the form (subject, object, predicate), and identifying one or more associated entities is performed for relationships in which a corresponding entity of the one or more entities serves as a subject and for relationships in which a corresponding entity of the one or more entities serves as an object.

9. The computing system of claim 1, wherein: The modified input is not provided to the user.

10. The computing system of claim 1, wherein: The user input before modification is not provided to the large language model without the modified content.

11. The computing system of claim 1 , the operations further comprising: A length constraint is added to the modified user input.

12. The computing system of claim 1, the operations further comprising: An instruction is added to the modified user input.

13. The computing system of claim 1, the operations further comprising: A contextual instruction is added to the modified user input.

14. The computing system of claim 1, the operations further comprising: A content constraint instruction is added to the modified user input.

15. The computing system of claim 1, wherein: The one or more entities correspond to a first set of one or more entities, the operations further comprising: identifying a second set of one or more entities in the response; linking one or more entities of the second set of one or more entities to supplemental content, wherein displaying the response in response to the user input comprises displaying the response with one or more links to supplemental content; receiving user input selecting a linked entity; and Displays supplemental content specific to the linked entity.

16. The computing system of claim 1, wherein: Based on the analyzing, determining one or more entities of the semantic frame represented in the at least a portion of the plurality of word-grams includes analyzing a plurality of discrete semantic frames.

17. The computing system of claim 1, wherein: Based on the analysis, determining one or more entities of the semantic framework represented in the at least a portion of the plurality of word-grams includes sending an analysis request to be performed on the semantic framework located on a remote computing system.

18. A method implemented in a computing system, the computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, the method comprising: receiving user input including a plurality of word-grams from a user; analyzing at least a portion of the plurality of word-grams; determining, based on the analyzing, one or more entities of a semantic frame represented in the at least a portion of the plurality of word-grams; determining one or more triples of the semantic frame for at least a portion of the one or more entities or associated entities; adding at least a portion of the one or more triples, or a representation thereof, to the user input to provide a modified user input; submitting the modified user input to the large language model; processing the modified user input using the large language model to provide a response; as well as The response is returned in response to receiving the user input.

19. The method according to claim 18, wherein: Adding at least a portion of the one or more triples or a representation thereof to the user input to provide a modified user input comprises: The triples of the at least a portion of the plurality of triples are submitted to a linguistic function to provide a representation, the representation being the linguistically-defined triples.

20. One or more computer-readable storage media comprising: Computer executable instructions that, when executed by a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, cause the computing system to receive user input comprising a plurality of word-grams from a user; computer-executable instructions that, when executed by the computing system, cause the computing system to analyze at least a portion of the plurality of word-grams; computer-executable instructions that, when executed by the computing system, cause the computing system to determine, based on the analyzing, one or more entities of a semantic frame represented in the at least a portion of the plurality of word-grams; computer executable instructions that, when executed by the computing system, cause the computing system to determine one or more triples of the semantic frame for at least a portion of the one or more entities or associated entities; computer-executable instructions that, when executed by the computing system, cause the computing system to add at least a portion of the one or more triples, or a representation thereof, to the user input to provide a modified user input; computer-executable instructions that, when executed by the computing system, cause the computing system to submit the modified user input to a large language model; computer-executable instructions that, when executed by the computing system, cause the computing system to process the modified user input using the large language model to provide a response; as well as Computer-executable instructions that, when executed by the computing system, cause the computing system to return the response in response to receiving the user input.

Images