Automated ontology maintenance

Abstract

Techniques and solutions are provided for improved use of knowledge maps in document processing. The relevance of the attributes to the knowledge graph may vary over time. Although an attribute may occur, some time may be required before the attribute should be used in the knowledge graph apparently. Similarly, although the attribute may be correlated over a period of time, it may lose its correlation. The present disclosure provides techniques for tracking use of attributes over time and making or proposing attribute state changes. These changes may result in the attribute being visible or invisible in the knowledge graph, which in turn may affect how to process future documents. Further, in some cases, the attributes may be made active, and documents processed when the attributes are not present or inactive may be reprocessed to obtain information for the attributes.

Description

Technical Field

[0001] The present disclosure generally relates to automated processes for maintaining ontologies, such as managing state information of properties of an ontology. Background Art

[0002] Ontologies are important for a variety of computer-implemented processes. For example, ontologies can be used to link data in the Semantic Web, in natural language processing, in query processing (such as by translating concepts into SQL), and in data integration (integrating data with common semantic concepts). Furthermore, ontologies can be used in artificial intelligence systems, including large language models, the use of which is currently experiencing explosive growth.

[0003] Typically, ontologies are created and updated manually. These tasks can be very time-consuming, especially when there are a large number of concepts to represent in the ontology. Manually created ontologies can vary depending on the user who developed the ontology, including using different labels for common semantic concepts and whether the user happens to recognize specific concepts—that is, some users may identify ontology concepts that may be overlooked by other users.

[0004] This is especially true for relatively new concepts. That is, for example, when mainstream electric vehicles were first introduced, the individuals responsible for the ontology may not have been aware that concepts related to electric vehicles existed. Even if they did exist, they might not have been considered relevant enough to be included in the ontology. In a similar manner, some concepts may become outdated and no longer be particularly relevant to the ontology. For example, there was a time when cars designed to run on ethanol or biodiesel were more actively sought after, but this has largely subsided.

[0005] Therefore, current techniques for developing ontologies can be very time-consuming, may contain errors (including not identifying relevant semantic concepts or identifying ontology concepts as outdated), and therefore, there is room for improvement. Summary of the Invention

[0006] This summary is provided to introduce some concepts in a simplified form that will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] Techniques and solutions are provided for improved use of knowledge graphs in document processing. The relevance of an attribute to a knowledge graph can change over time. While an attribute may appear, it may take some time before it is apparent that the attribute should be used in the knowledge graph. Similarly, while an attribute may be relevant for a period of time, it may lose its relevance. The present disclosure provides techniques for tracking the use of an attribute over time and making or proposing changes to the state of an attribute. These changes may cause the attribute to be visible or invisible in the knowledge graph, which in turn may affect how future documents are processed. Furthermore, in some cases, an attribute can be made active, and documents that were processed when the attribute was not present or inactive can be reprocessed to obtain information about the attribute.

[0008] In one aspect, the present disclosure provides a process for changing a property state of a property of a knowledge graph based on a count of the property in an electronic document provided within a time period, and processing the electronic document using the updated property state.

[0009] A plurality of attributes are received. A plurality of electronic documents within a time period are processed. The processing includes: if the document being processed includes a corresponding attribute from the plurality of attributes, incrementing a corresponding count for the corresponding attribute. For at least one attribute from the plurality of attributes, a total count for the at least one attribute over at least a portion of the time period is determined. The total count is compared to a threshold. A determination is made that the total count satisfies the threshold. A state of the at least one attribute is changed based at least in part on the determination that the total count satisfies the threshold to provide an updated state of the attribute. At least another electronic document is processed using the updated state of the attribute.

[0010] The present disclosure also includes computing systems and tangible, non-transitory computer-readable storage media configured to perform the above-described methods or including instructions for performing the above-described methods. As described herein, various other features and advantages can be incorporated into the technology as desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An exemplary directed graph is shown.

[0012] Figure 2 Shows the domain and range of attributes in a directed graph's schema.

[0013] Figure 3 An exemplary SPARQL query and the results of the query are shown.

[0014] Figure 4 An exemplary system architecture for methods according to some embodiments is shown.

[0015] Figure 5 An algorithm for computing strings from a directed graph for input to a language model is shown.

[0016] Figure 6 A sorting algorithm for sorting statements of a directed graph is shown.

[0017] Figure 7 shows a directed graph that can be provided as input to a sorting algorithm.

[0018] Figure 8 Shows the relationship between a condition and one or more patterns.

[0019] Figure 9 Is a flowchart of the process of creating a domain data model, including mapping elements of the domain data model to elements of the core data model.

[0020] Figure 10 Shows the expansion to handle the problem pattern Figure 5 algorithm.

[0021] Figure 11 is a diagram showing the relationships between and components of knowledge graphs, ontologies, and meta-ontologies.

[0022] Figure 12 is a flow chart of a method for extracting attributes from a set of source documents.

[0023] Figure 13 is a diagram of a computing environment in which the disclosed techniques for attribute extraction may be performed.

[0024] Figures 14A-14C An example source document is shown having information from which attributes can be extracted.

[0025] Figure 15 An example user interface for selecting source files for analysis and for identifying instances associated with such source files is shown.

[0026] Figure 16 An example user interface is shown in which a user can define or edit a set of source files for a particular instance.

[0027] Figure 17 An example user interface is shown in which a user can view and edit an instance graph created from one or more source files for an instance, including viewing properties or property values ​​extracted from the source documents.

[0028] Figure 18 Example pseudocode is provided for aligning instance graphs, such as using a common vocabulary, and counting the occurrences of a particular attribute in a set of instance graphs.

[0029] Figure 19A An example user interface is shown for viewing identified attributes and editing an attribute list and selecting attributes to include in an ontology.

[0030] Figure 19B Shows the screen after selecting a control to edit a property, such as a label applied to a property. Figure 19A User interface.

[0031] Figure 20 Shows how properties associated with an instance can change over time.

[0032] Figure 21 Depicted is an example computing environment that can be used to monitor property usage and update property status information.

[0033] Figure 22 Example data objects are shown that can be used to store associations between attributes and documents having the attributes and to track attribute usage.

[0034] Figure 23 Example documents in an interval and information that can be used to determine membership in an interval are shown.

[0035] Figure 24 Example pseudocode is provided for calculating attribute usage within an interval and assigning status to an attribute based on such attribute usage.

[0036] Figure 25 An example user interface is shown that can be used to add instances to a knowledge graph and view attribute information associated with such instances, and optionally add attributes associated with the instances to an ontology associated with the instances.

[0037] Figure 26 is an example user interface that provides information about properties that may be out of date.

[0038] Figure 27 is an example user interface that provides summary information about attributes within an interval and provides detailed information for a selected attribute, including the values ​​observed for the selected attribute within the interval.

[0039] Figure 28 is an example user interface that provides detailed information for a selected property, including usage information for the property during the current interval, the property values ​​observed during the interval, the intervals associated with different states of the property, and an identifier for the document associated with the property.

[0040] Figure 29 is an example user interface that summarizes property status and usage over various intervals, as well as graphs that can be displayed using such information.

[0041] Figure 30 and Figure 31 Example property graphs are shown, demonstrating how usage information for a property can be presented for intervals, and how graphs for multiple intervals can provide insight into property trends.

[0042] Figure 32 An example electronic newsletter that may contain information updates relevant to a user is shown.

[0043] Figure 33 is a sample email that can contain information updates relevant to the user.

[0044] Figure 34 An example data model that may be used with the disclosed technology is shown.

[0045] Figure 35 is a flowchart of an example process for defining intents and analyzing a knowledge graph for intent-related updates.

[0046] Figure 36 It can be achieved Figure 35 Diagram of a simplified computing environment for a process.

[0047] Figure 37 It can be achieved Figure 35 A more detailed diagram of the computing environment for the process.

[0048] Figure 38 It shows how nodes of a knowledge graph can have different weights or relevance to a user's internet, including based on knowledge graph elements that are inferred to be relevant to the user's intent.

[0049] Figure 39 Shown is how weights assigned to nodes of a knowledge graph are propagated to related nodes in the process of identifying nodes to include in the inferred intent.

[0050] Figure 40 An example user interface is provided where users can define intents by selecting elements of a knowledge graph.

[0051] Figure 41 An example user interface is provided in which a user can define an intent by selecting text of an electronic document.

[0052] Figure 42 Shown is an example log from which knowledge graph update information can be extracted.

[0053] Figure 43 An example standardized format in which knowledge graph changes can be expressed is provided, as well as example changes expressed in such a format.

[0054] Figure 44 Components of an example prompt template for generating a message summarizing a knowledge graph change are shown.

[0055] Figure 45 is used Figure 44Example prompt for the Example prompt template.

[0056] Figure 46 is generated by the natural language generator in response to Figure 45 Example responses generated for the prompt.

[0057] Figure 47 is a flow chart of an example process for changing the attribute state of an attribute of a knowledge graph based on a count of the attribute in an electronic document provided within a time period and processing the electronic document using the updated attribute state.

[0058] Figure 48 is a diagram of an example computing system in which some described embodiments may be implemented.

[0059] Figure 49 is an example cloud computing environment that can be used with the techniques described herein. DETAILED DESCRIPTION

[0060] Example 1 - Overview

[0061] Ontologies are important for a variety of computer-implemented processes. For example, ontologies can be used to link data in the Semantic Web, natural language processing, query processing (such as by translating concepts into SQL), and data integration (integrating data with common semantic concepts). Furthermore, ontologies can be used in artificial intelligence systems, including large language models, the use of which is currently experiencing explosive growth.

[0062] Typically, ontologies are created and updated manually. These tasks can be very time-consuming, especially when there are a large number of concepts to represent in the ontology. Manually created ontologies can vary depending on the user who developed the ontology, including using different labels for common semantic concepts and whether the user happens to recognize specific concepts—that is, some users may identify ontology concepts that may be overlooked by other users.

[0063] This is especially true for relatively new concepts. That is, for example, when mainstream electric vehicles were first introduced, the individuals responsible for the ontology may not have been aware that concepts related to electric vehicles existed. Even if they did exist, they might not have been considered relevant enough to be included in the ontology. In a similar manner, some concepts may become outdated and no longer be particularly relevant to the ontology. For example, there was a time when cars designed to run on ethanol or biodiesel were more actively sought after, but this has largely subsided.

[0064] Therefore, current techniques for developing ontologies can be very time-consuming, may contain errors (including not identifying relevant semantic concepts or identifying ontology concepts as outdated), and therefore, there is room for improvement.

[0065] This disclosure provides techniques for maintaining ontologies, including adding or removing attributes from an ontology. In one aspect, when instances are added to an ontology, new attributes are identified. Usage counts for these new attributes are tracked over time. In some cases, attribute count information can be presented to a user, and a determination can be made as to whether the attribute should be added to the ontology. Alternatively, an automatic suggestion can be provided to the user as to whether the attribute should be added. In other cases, an attribute can be automatically added to an ontology when specific criteria are met.

[0066] In a similar manner, the usage of existing properties in an ontology can be tracked over time. By analyzing a specific time interval, such as a more recent time interval, it can be determined whether fewer instances include an existing ontology property. The user can be provided with data on the property's usage and, optionally, a recommendation on whether to consider marking the property as obsolete or removing it from the ontology. Alternatively, if threshold criteria are met, the property can automatically change its status to obsolete.

[0067] The ability to track attribute usage over time also helps track the evolution of an ontology. Tracking the evolution of an ontology can include tracking how the usage of a particular attribute (including related attributes) changes over time. This type of analysis can help users identify trends, including whether new trends are likely to be short-lived or persist over a longer period of time.

[0068] Updated ontologies enhance the utility of knowledge graphs and the instances they represent. By incorporating the latest domain knowledge and accommodating new attributes and concepts, updated ontologies help ensure that knowledge graphs remain relevant and comprehensive. Furthermore, they facilitate tracking of instance attributes that may not have been previously cataloged, resulting in a more accurate and complete representation of the data. This adaptability and responsiveness to evolving knowledge enables updated ontologies to help maintain the semantic clarity and usability of knowledge representations.

[0069] In addition to tracking data for new instances, the present disclosure provides techniques for reprocessing instances received before the ontology was updated. For example, if attributes related to electric vehicles are added to the ontology, previous instances of vehicles can be processed so that data for the instance's attributes can be added where appropriate.

[0070] Example 2 describes a knowledge graph, including a representation of the knowledge graph, and linguistic knowledge graph triples. Examples 3-7 describe the relationships between knowledge graphs, ontologies, and meta-ontologies, as well as techniques for identifying new properties of ontologies. Examples 8-13 describe techniques for changing the state of a property of an ontology, including changing between an active state and an outdated state, or changing a property to an active state that was observed but not previously used frequently enough to be active. Examples 14-23 describe techniques that can be used to register a user's intent and update the user when relevant changes are detected, where the intent indicates the portion of the knowledge graph that the user is interested in.

[0071] Example 2 - Example Linguisticization of Knowledge Graph Triples

[0072] An enterprise may have a variety of different products, services, and teams. An enterprise may also have a comprehensive knowledge graph that stores knowledge related to the skills, processes, experiences, capabilities, and insights that the enterprise relies on in its daily operations. The content of the knowledge graph may also include enterprise-specific acronyms, the enterprise's departments, and product specifications. This knowledge can enable the enterprise to respond to business situations in a fast, professional, and flexible manner. Building and maintaining a knowledge graph can be expensive and labor-intensive. A knowledge graph (i.e., a semantic web and / or linked data web) can be specified using the Resource Description Framework (RDF).

[0073] In some cases, users want to ask questions or provide tasks to a language model (e.g., a large language model based on a generative pre-trained transformer (such as ChatGPT)). However, language models are typically trained on unlabeled human-readable text in an unsupervised manner. Therefore, language models may not be able to directly process knowledge graphs or use knowledge graphs as input (e.g., for training).

[0074] Therefore, it may be desirable to maximize the usability of knowledge graphs, for example, using knowledge graphs as the basis for artificial intelligence applications, more specifically, training or otherwise improving language models. When training a language model, the language model can be used to answer questions or perform tasks based on the knowledge stored in the knowledge graph.

[0075] Additionally, it may be desirable to extract human-readable text from the knowledge graph (e.g., for interpreting answers provided by software that relies on the knowledge graph (e.g., a process advisor)).

[0076] According to one aspect, a computer-implemented method for providing data from a directed graph to a language model is provided. The method includes defining a plurality of conditions and a plurality of patterns, wherein each of the conditions has at least one corresponding pattern. The method also includes receiving a subset of the directed graph, wherein the subset of the directed graph includes a plurality of statements. Each statement includes a subject, an object, and a predicate that relates the subject to the object. For each of the statements in the subset of the directed graph, the following operations are performed: when one of the conditions matches the corresponding statement and the pattern corresponding to the condition can be applied to the corresponding statement, a string is calculated for the corresponding statement using the pattern. The calculated string is provided as input to the language model.

[0077] Providing data from the directed graph to the language model can include extracting or reading data from the directed graph and feeding or sending the data as input to the language model. Statements can be referred to as triples or triple statements, having subject, predicate, and object components. Conditions can correspond to corresponding patterns, in the sense that when a condition is determined to be true for the corresponding statement, it can be determined whether the corresponding pattern can be applied to the corresponding statement.

[0078] The clause "each of the conditions has at least one corresponding pattern" can be understood to mean that each of the multiple conditions has at least one corresponding pattern from the multiple patterns. Therefore, each of the multiple conditions can be assigned at least one pattern from the multiple patterns. In other words, when a condition has at least one corresponding pattern, the at least one corresponding pattern is assigned to the condition. Furthermore, each of the multiple conditions can be assigned multiple patterns from the multiple patterns.

[0079] The clause “when one of the conditions matches the corresponding statement and the pattern corresponding to the condition can be applied to the corresponding statement” may include determining whether at least one pattern corresponding to the condition can be applied to the corresponding statement. Thus, determining whether at least one pattern corresponding to the condition can be applied to the corresponding statement may involve determining whether the pattern corresponding to the condition matches the corresponding statement (i.e., testing whether the corresponding statement has the features or elements specified by the pattern).

[0080] For example, determining whether a pattern can be applied to a statement including a subject can be done as follows:

[0081] Multiple conditions can include the first condition:

[0082] <?s><?p><?o>.

[0083] BIND(sap:BusinessActivity AS<?s>)

[0084] BIND(rdf:type AS <?p>)

[0085] The first mode may correspond to the first condition:

[0086] The<s.rdfs:label> is a business activity.(<s.rdfs:label> It is a business activity.)

[0087] Since the first pattern includes a reference to a label of the subject s, if a statement including a subject satisfies the first condition (i.e., BusinessActivity), but the subject of the statement does not have the label required by the first pattern, the first pattern cannot be applied to (i.e., does not match) the statement.

[0088] Using a pattern to compute a string from a corresponding statement may involve directly outputting the text of the pattern and matching pattern operators with components of the statement.

[0089] Furthermore, when evaluating a string, not only the corresponding statement that matches the condition but also one or more additional statements of the digraph subset may be accessed by the pattern corresponding to the condition. In other words, the pattern may consider statements that do not match the condition. Thus, the condition may trigger the execution of the pattern for the corresponding statement, which in turn may trigger the processing of at least a portion of the digraph subset (or the entire digraph subset) using the pattern.

[0090] Subjects, objects, and predicates can be referred to as components of their respective statements. A subject and / or predicate can be an RDF resource (e.g., a subject and / or predicate can have RDF resource properties and can be a type or label). An object can be a literal (e.g., an RDF literal) with a defined data type, such as a string, integer, Boolean, or double-precision floating-point number (as defined in the Extensible Markup Language (XML) Schema Definition Language (XSD)). For more information about RDF, see the RDF specification, "Concepts and Abstract Syntax," https: / / www.w3.org / TR / rdf11-concepts / .

[0091] The computed strings can be provided directly to the language model (e.g., in the case of small strings, such as less than 1GB), or can be serialized into a text file before being provided to the language model (e.g., in the case of terabytes of data computed from a comprehensive directed graph). In summary, the method accepts a subset of a directed graph as input and generates grammatically correct sentences as computed strings. The method iterates over the input statements (i.e., triples) in the subset of the directed graph and uses conditional filters to determine which patterns are applicable to each of the statements. When a condition matches a statement and the statement has the elements required by the pattern that corresponds to (e.g., is assigned to) the condition, the pattern is applied to the condition. The result of the iteration can be a set of strings such that a string is computed for each statement in the input. The syntax and interpretation of the patterns are discussed in more detail below.

[0092] Directed graph representations can approximate human language.

[0093] Therefore, providing the calculated string as input to the language model can maximize the usefulness of the subset of the directed graph, for example, by using the subset of the directed graph as the basis for artificial intelligence applications. Once the language model has processed the calculated string, the language model can be used to answer questions or perform tasks based on the knowledge stored in the subset of the directed graph. Thus, the time, effort, and expense invested in constructing the directed graph can be utilized in other ways (e.g., answering questions or by using the language model).

[0094] Additionally or alternatively, it may be desirable to extract human-readable text from the subset of the directed graph (eg, for interpreting answers provided by software (eg, a process advisor) that relies on the subset of the directed graph).

[0095] In some cases, each of the conditions includes at least three condition variables (variables that appear in a condition may be referred to as condition variables). Each of the condition variables may correspond to (e.g., store) a different component of the statement. For example, a first of the condition variables matches a subject, a second of the condition variables matches a predicate, and a third of the condition variables matches an object. At least one of the condition variables may be bound to at least one value (e.g., to an RDF property). In other words, at least one of the condition variables may specify at least one value (e.g., an RDF property) that a component of the statement must have. Each of the condition variables may specify a literal (e.g., an RDF literal) or an instance of a class (e.g., an RDF class). An instance of a class may be referred to as an instance. An instance may relate to a specific concept and have a definite article, while a class may relate to a general concept and have an indefinite article.

[0096] Each condition can be applied to a statement and can evaluate to TRUE or FALSE. In other words, a condition can return a Boolean value. For example, if a condition evaluates to TRUE, the condition matches the corresponding statement, and a determination is made as to whether at least one pattern corresponding to the condition can be applied to the corresponding statement. The following are numbered examples of conditions that can be in multiple conditions:

[0097] 1.<?s><?p><?o>.

[0098] 2.<?s><?p><?o>.

[0099] BIND(rdfs:label AS <?p>)

[0100] 3.<?s><?p><?o>.

[0101] <? s>rdf:type sap:BusinessActivity.

[0102] BIND(rdfs:label AS <?p>)

[0103] 4.<?s><?p><?o>.

[0104] <? s>rdf:type sap:BusinessActivity.

[0105] <?o>sap:requires<?r>.

[0106] BIND(rdfs:label AS <?p>)

[0107] In the first condition, "<?s>" is a variable corresponding to the subject in the directed graph, "<?p>" is a variable corresponding to the predicate in the directed graph, and "<?o>" is a variable corresponding to the object in the directed graph. Therefore, the first condition specifies (i.e., in order for the first condition to evaluate to TRUE) that the statement must contain a subject, a predicate, and an object. The second condition requires that the statement contain a subject, a predicate, and an object, and that the predicate is an rdfs:label. The third condition requires that the statement contain a subject, a predicate, and an object, the subject has the property sap:BusinessActivity (more specifically, is of type sap:BusinessActivity), and the predicate is an rdfs:label. The fourth condition requires that the statement contain a subject, a predicate, and an object, the subject has the property sap:BusinessActivity (more specifically, is of type sap:BusinessActivity), the object has a "sap:requires" relationship with the object "<?r>" (i.e., a subject-object relationship), and the predicate is an rdfs:label. In this regard, "<?r>" is a variable bound to a requirement.

[0108] A condition among the multiple conditions can be used to prevent a pattern from being used to compute semantically incorrect strings. In other words, a condition can be used to ensure that the computed string is semantically and / or grammatically correct. Therefore, by assigning a pattern to a condition, the circumstances in which the pattern is applied can be restricted, thereby ensuring or facilitating the computation of semantically correct strings (i.e., sentences). In the absence of a condition, the pattern can be applied to compute an exemplary fantasy string, such as "Harry Potter is a business activity" or "Star Wars movies are business activities." However, the exemplary fantasy string is semantically incorrect; therefore, the exemplary simulated string will not be helpful as input to a language model and may prolong the training of the language model or even cause the language model to generate incorrect output.

[0109] In some cases, at least one of the conditions has multiple corresponding patterns. Thus, computing a string from the corresponding statement using the patterns may include computing multiple strings from the corresponding statement using each pattern corresponding to a condition applicable to the corresponding statement (i.e., a condition that matches the corresponding statement).

[0110] Alternatively, calculating the string from the corresponding statement using the patterns may include determining a random order of the patterns corresponding to the condition, and calculating the string from the corresponding statement using only the first pattern in the random order of the patterns that can be applied to the corresponding statement. For example, patterns 1 to 4 may be ordered as 2, 4, 3, 1, and pattern 2 can be applied to the corresponding statement; therefore, pattern 2 is applied to the corresponding statement.

[0111] For example, multiple conditions can include a sequenceID condition:

[0112] <? I1> <sequenceid><? L1>.

[0113] The sequenceID condition can correspond to the following pattern (A):

[0114] (A)The sequence identifier of<? I1.rdf:type.rdfs:label><? I1.rdfs:label>is<? L1>.

[0115] ((A) The ordinal identifier of <?I1.rdf:type.rdfs:label><?I1.rdfs:label> is <?L1>.)

[0116] Continuing with the example, a subset of the directed graph could include the following statements:

[0117] 1.Emissions Management isA BusinessCapability.

[0118] 2.BusinessCapability rdfs:label "Business Capability".

[0119] 3.EmissionsManagement SequenceID "5".

[0120] The SequenceID condition matches only statement (3) because statement (3) includes "SequenceID" while statements (1) and (2) do not include "SequenceID".

[0121] Pattern (A) can be applied to statements (1), (2), and (3) to compute the following string:

[0122] The sequence identifier of Business Capability is 5.

[0123] All three statements are needed to evaluate the string above because statements (1) and (2) provide context for statement (3).

[0124] Thus, as described above, when evaluating a string, not only the corresponding statement that matches the condition, but also one or more other statements in the subset of the directed graph that are accessible by the pattern corresponding to the condition. In other words, the pattern can consider statements that do not match the condition. Therefore, the condition triggers the execution of the pattern for the corresponding statement.

[0125] Furthermore, statements can be iteratively checked for subsets of the directed graph. Thus, in the above example, statements (1) and (2) will not cause the pattern to be triggered, but statement (3) will.

[0126] In this example, determining whether pattern (A) can be used for statement (3) may include determining context information items for pattern (A), namely:

[0127] the label of the type of variable<? I1|

[0128] the label of variable <? I1>

[0129] Both items of contextual information are not available in statement (3) itself, but are available elsewhere in a subset of the directed graph.

[0130] Given statement (3), context information exists in a subset of the directed graph, and pattern (A) can be executed for statement (3).

[0131] Whenever forward dot notation is used, it encloses additional information that is not present in the triple itself.

[0132] In some cases, each pattern includes one or more of the following:

[0133] - at least one variable, wherein the variable specifies (e.g., is bound to) a class, an instance of a class, a literal, or a predicate;

[0134] - text, such as one or more articles (e.g., definite or indefinite grammatical articles);

[0135] - At least one attribute to apply to the variable.

[0136] Each schema can also include language filters. Literals can specify numeric values ​​or text, where literals can conform to the RDF schema class for literal values.

[0137] Advantageously, a pattern may combine static text with variables that specify the structure of a subset of a directed graph (eg, the subject, object, and predicate of one of the statements), possibly supplementing information generated from reifying the subset of the directed graph.

[0138] In addition or alternatively, the pattern may include at least one specific pattern (i.e., a custom pattern) and a plurality of default patterns. When a condition corresponding to a specific pattern matches a corresponding statement and the specific pattern can be applied to the corresponding statement, calculating a string from the corresponding statement using the pattern may include using the specific pattern. When the condition corresponding to the specific pattern does not match the corresponding statement, the method may further include determining whether a condition corresponding to one of the default patterns matches the corresponding statement. When a condition corresponding to one of the default patterns matches the corresponding statement, calculating a string from the corresponding statement using one of the default patterns. Defining multiple conditions and multiple patterns may further include defining at least three conditions and at least three patterns, wherein at least one of the three patterns is a specific pattern, and at least one of the three conditions corresponds to the specific pattern.

[0139] The following are example patterns that can be included in multiple patterns:

[0140] Text<? I1.rdf:type.rdfs:label>Text

[0141] The variable <? I1> can be bound to sap:PrintReceipt. Patterns such as the example pattern above can use forward dot notation (also known as dot notation) to refer to fields, components, or subproperties of a property. This can provide an advantage for patterns over traditional SPARQL, which does not support forward dot notation because forward dot notation allows for more compact expression.

[0142] Thus, the above exemplary schema may be applied to (e.g., a subset of directed graphs may include) the following three statements:

[0143] sap:PrintReceipt rdf:type sap:Task.

[0144] sap:Task rdfs:label"Task"@en.

[0145] sap:Task rdfs:label"Process Task"@en.

[0146] In some cases, after a pattern is applied to a corresponding statement, the pattern is not applied to further statements that match the pattern (i.e., further statements in a subset of the directed graph that match the pattern may be skipped). Matching only one statement and skipping further statements may be configurable options. For example, as described below, a postfix operator may cause a Cartesian product to be computed.

[0147] For example, after applying the pattern to the corresponding statement that includes an rdfs:label of an object, further statements that include an rdfs:label of an object may be skipped (i.e., the pattern is not applied to further statements). Thus, computing the string from the statement above using the exemplary pattern would produce the following: "Text Task Text".

[0148] At least one of the patterns may include a filter condition and / or a post-operator. The filter condition may specify a language. The post-operator may cause a Cartesian product to be performed. As another example, a subset of a directed graph may include the following four statements:

[0149] sap:PrintReceipt rdf:type sap:Task.

[0150] sap:Task rdfs:label"Task"@en.

[0151] sap:Task rdfs:label"Process Task"@en.

[0152] sap:Task rdfs:label"Schritt"@de.

[0153] The following further exemplary modes may be included in the multiple modes and may be applied to the above four statements:

[0154] Text<? I1.rdf:type.rdfs:label(lang='en')*>Text

[0155] The above further exemplary pattern includes a filter condition for specifying a language and an asterisk (*) postfix operator for generating a Cartesian product. A Cartesian product can generate all possible combinations of the preceding elements. Therefore, the following string will be calculated by applying the further exemplary pattern including a Cartesian product, because the further exemplary pattern targets English tags, and there are two English tags in the above four statements:

[0156] Text Task Text

[0157] Text Process Task Text

[0158] Therefore, the above statement including "'Schritt'@de" will not be processed because the statement does not meet the filter condition in the further exemplary pattern (i.e., the statement is not in English language). Without the asterisk postfix operator in the above further exemplary pattern, only the first string "Text Task Text" will be evaluated.

[0159] The above exemplary patterns and further exemplary patterns may be specific patterns (ie, patterns applicable to a directed graph or a group of directed graphs).

[0160] Strings can be computed from the corresponding statements using only the default mode. However, the use of a specific mode can result in a computed string that more accurately and precisely describes the contents of the subset of the directed graph.

[0161] In addition to the asterisk, you can use other postfix operators (that is, operators provided at the end of the pattern, also called postfix operators). For example, additional postfix operators can limit the output of the Cartesian product to a specified number of combinations (for example, about 10 combinations).

[0162] As another example, the following requirement pattern can be applied to the four statements above:

[0163] The<? I1.rdf:type.rdfs:label(lang='en')*><? I1.rdfs:label>requires a<? I2.rdfs:label>.

[0164] In this example, <? I1> can be bound to sap:PrintReceipt and <? I2> can be bound to sap:Printer. Therefore, by applying the requirement pattern to the above four statements, the following string can be calculated:

[0165] -The Process Task Print Receipt requires a Printer.

[0166] -Task Print Receipt requires a Printer.

[0167] Without the asterisk postfix operator in the requirement pattern, only the first string would be evaluated (ie, "TheProcess Task Print Receipt requires a Printer.").

[0168] In some cases, each of the computed character strings is a grammatically correct sentence, where the condition and / or pattern may ensure that the computed character strings are grammatically correct sentences.

[0169] Additionally or alternatively, the at least one specific mode may include multiple specific modes. Each of the specific modes may be applicable to a set of directed graphs defined according to the Resource Description Framework or a set of knowledge graphs defined according to the Resource Description Framework. Each of the default modes may be applicable to any directed graph defined according to the Resource Description Framework or any knowledge graph defined according to the Resource Description Framework.

[0170] Thus, each specific pattern can be defined for a single local network and corresponding directed graph or a group of local networks and corresponding directed graphs, while the default pattern can be applied to any directed graph.

[0171] Modes can include one or more of the following five modes:

[0172] - Patterns that apply to instance-to-instance statements, including variables<I1,p,I2>

[0173] - Patterns for instance-to-class statements, including variables<I1,p,C1>

[0174] - Patterns that work with class-to-class statements, including variables<C1,p,C2>

[0175] - Mode for converting instances to literal statements, including variables<I1,p,L1>

[0176] - Modes that work with classes to literal statements, including variables<C1,p,L1>

[0177] By including the corresponding variables<I1,p,I2> 、<I1,p,C1> 、<C1,p,C2> 、<I1,p,L1> 、<C1,p,L1> The corresponding conditions make the above five patterns applicable to instance-to-instance statements, instance-to-class statements, class-to-class statements, instance-to-text statements, and class-to-text statements.

[0178] The five patterns mentioned above can be the default patterns because they are applicable to any directed graph, or more specifically, any knowledge graph.

[0179] For the five patterns above, "I1" and "I2" are variables referring to instances (i.e., instances of a class), "C1" and "C2" are variables referring to classes, "L1" is a variable referring to a literal, and "p" is a variable referring to a predicate. The first of the five patterns applicable to instance-to-instance statements can be implemented as follows:

[0180] The<? I1.rdf:type.rdfs:label><? I1.rdfs:label><? p.rdfs:label>the<? I2.rdf:type.rdfs:label><? I2.rdfs:label>.

[0181] The second of the five patterns applicable to instance-to-class statements can be implemented as follows:

[0182] The<? I1.rdf:type.rdfs:label><? I1.rdfs:label><? p.rdfs:label>a<? C1.rdfs:label>.

[0183] The third of the five patterns for class-to-class statements can be implemented as follows:

[0184] A<? C1.rdfs:label><? p.rdfs:label>a<? C2.rdfs:label>.

[0185] The fourth of the five patterns for converting instances to textual statements can be implemented as follows:

[0186] The<? I1.rdfs:label><? p.rdfs:label><? L1>.

[0187] The fifth of the five patterns for class-to-literal statements can be implemented as follows:

[0188] A<? C1.rdfs:label><? p><? L1>.

[0189] A user or administrator may define further default modes or change the exemplary default modes provided above.

[0190] In addition or alternatively, the plurality of patterns may include at least one text pattern and at least one question pattern. The text pattern and / or the question pattern may be a specific pattern. The text pattern and / or the question pattern may be a default pattern. Each condition may correspond to at least one text pattern, and at least one condition may correspond to at least one question pattern. For example, the conditions may be defined such that each condition must correspond to at least one text pattern, and each condition may correspond to at least one question pattern. In addition to or in place of the text pattern, a configuration option may be set to apply a question pattern. When one of the conditions matches a corresponding statement and the configuration option is set to apply a question pattern and the question pattern corresponding to the condition may be applied to the corresponding statement, the method may include calculating a string from the corresponding statement using the question pattern and / or calculating a further string from the corresponding statement using the question pattern in addition to calculating a string from the corresponding statement using the text pattern.

[0191] For example, a subset of a directed graph might include the following seven statements:

[0192] sap:PrintReceipt sap:requires sap:Printer.

[0193] sap:PrintReceipt rdf:type sap:Task.

[0194] sap:PrintReceipt rdfs:label"Print Receipt".

[0195] sap:Printer rdfs:label "Printer".

[0196] sap:Task rdfs:label"Task"@en.

[0197] sap:Task rdfs:label"Process Task"@en.

[0198] sap:Task rdfs:label"Schritt"@de.

[0199] Continuing with the example, the plurality of modes may include the following modes (eg, text modes):

[0200] The<I1.rdf:type.label(lang=‘en’)*><I1.rdfs:label> requires a<I2.rdfs:label> .

[0201] Additionally, multiple patterns may include the following question patterns that directly precede the above text patterns:

[0202] What is required by<I1.rdf:type.label(lang=‘en’)*><I1.rdfs:label> ?

[0203] The "*" (asterisk) operator (i.e., postfix operator) in the question pattern causes the Cartesian product to be calculated. In the text and question patterns above, <? I1> is bound to (i.e., holds the value of) sap:PrintReceipt, <? p> is bound to sap:requires, and <? I2> is bound to sap:Printer. Therefore, the configuration option can be set to apply both question and text patterns. Thus, by applying both question and text patterns to the seven statements above, the following question / answer string is calculated:

[0204] Q: What is required by Task Print Receipt?

[0205] A: The Task Print Receipt requires a Printer.

[0206] Q: What is required by Task Print Receipt?

[0207] A: The Process Task Print Receipt requires a Printer.

[0208] Q: What is required by Process Task Print Receipt?

[0209] A: The Task Print Receipt requires a Printer.

[0210] Q: What is required by Process Task Print Receipt?

[0211] A: The Process Task Print Receipt requires a Printer.

[0212] For clarity, the string above is preceded by "Q:" and "A:". Therefore, the question pattern can be used to simulate a question-answer interaction. As discussed above in the Cartesian product example, without the asterisk operator in this example, the string would be computed only from the first statement to which the question and text patterns can be applied (i.e., the first statement that matches the question and text patterns).

[0213] In some cases, the subset of the directed graph can be the entire directed graph. Alternatively, the subset of the directed graph can be a proper subset of the entire directed graph and can be determined by means of a query of the directed graph. The query can be a SPARQL Protocol and RDF Query Language (SPARQL) query.

[0214] In some cases, a subset of a directed graph includes multiple nodes connected by edges. Nodes can represent real-world entities, and edges can represent relationships between entities or between entities and their types (i.e., classes). Thus, predicates can be distinguished based on whether they connect two entities or an entity and an entity type. Entities can also be referred to as resources. For each statement, a subject can correspond to a node, an object can correspond to a (different) node, and an edge corresponding to the predicate can connect the subject node to the object node.

[0215] A node may have a corresponding class, such that each of the nodes has a corresponding class. The (corresponding) class may be part of (or organized in) a schema (i.e., a data schema or ontology). The schema may be defined in RDF or a web ontology language.

[0216] The following is an example of a class:

[0217] :State a rdfs:Class.

[0218] :EuropeanState a rdfs:Class.

[0219] :City a rdfs:Class.

[0220] Thus, ":State" is a resource that is a class, more specifically, an RDF class. The class ":EuropeanState" is another resource that is a class, more specifically, a subclass of ":State". Thus, class hierarchies are possible. Furthermore, multiple inheritance is also possible.

[0221] Additionally or alternatively, a directed graph can be labeled and multi-relational. Thus, both nodes and edges can have labels, and edges can have directions. The object of a statement can be a label of the directed graph. A directed graph can be multi-relational in the sense that edges have different labels. A node of a directed graph can be a subject or an object, and an edge can be a predicate.

[0222] Additionally or alternatively, a schema may include attributes. Each attribute may apply to at least one class of the schema. At least one of the attributes may have a domain and / or a range. Each of the attributes may be used by (or applied to) at least one statement. A domain (e.g., rdfs:domain) may specify the class to which a subject belongs, and a range (e.g., rdfs:range) may specify the class to which an object belongs. More specifically, a domain may specify the class to which a subject of a statement belongs, and a range may specify the class to which an object of a statement belongs. For information about RDF schemas, please refer to the W3C RDF Schema specification, https: / / www.w3.org / TR / rdf-schema / .

[0223] The following are examples of properties:

[0224] rdf:type a rdf:Property

[0225] dbo:foundationPlace a rdf:Property.

[0226] :EuropeanState rdfs:subClassOf:State.

[0227] :locatedIn a rdf:Property.

[0228] :capitalOf a rdf:Property.

[0229] :capitalOf rdfs:subPropertyOf:locatedIn.

[0230] Thus, ":locatedIn" and ":capitalOf" are properties. Furthermore, ":capitalOf" is a subproperty of ":locatedIn". Properties can thus form hierarchies. The property ":EuropeanState rdfs:subClassOf:State" indicates that ":EuropeanState" is a subclass of a class hierarchy that includes the class ":State" and the subclass ":EuropeanState".

[0231] Thus, a schema can provide a vocabulary for a directed graph (e.g., a knowledge graph). A directed graph can have predefined property prefixes that can indicate whether a node (i.e., a subject or object) is an instance of a class or a class (e.g., if a node has the prefix "dbo" which represents a DBpedia ontology, then the node can be a class, and if a node has the prefix "dbr" which represents a DBpedia resource, then the node can be an instance). In some cases, a directed graph can use URI design to distinguish between instances and classes. A directed graph can include statements that explicitly indicate that certain nodes are classes. In some cases, whether a particular node represents an instance or a class can depend on the underlying model. For example, whether a node is a class (and is included in the schema of the directed graph) or an instance (and therefore is not included in the schema of the directed graph) can be determined by inspecting the rdf:type property: if the type is owl:Class, then the node is a class and is included in the schema; otherwise, the node is an instance (i.e., an instance of a class) and is not included in the schema.

[0232] In some cases, the total number of patterns is greater than or equal to the total number of attributes.

[0233] Furthermore, for an ontology (ie, a schema) O with a set of classes C and a set of properties P, only |P| conditions are needed to compute a string from a complete directed graph without grammatical errors.

[0234] In addition or alternatively, the data from the directed graph covers multiple subject areas. Each statement can be identified by at least one uniform resource identifier (URI). At least one of the node and the edge can be identified by a URI or an international resource identifier (IRI). More specifically, the node and the edge can each be identified by a URI or an IRI. In some cases, one or more of the subject, object, and predicate can be a URI. Some nodes (e.g., nodes corresponding to objects) can be identified via text rather than URIs. RDF can be used to represent a directed graph. A directed graph can be a knowledge base and / or a knowledge graph. A statement can be referred to as a fact or a factual statement. Therefore, a directed graph can have a structure similar to a known knowledge graph (such as DBPedia, Wikidata, BabelNet, DBkWik, Freebase, and DBnaray).

[0235] Compared to relational databases, knowledge graphs have a more flexible data structure because the data types provided by knowledge graphs can vary. For example, the attributes associated with different instances can be different, even if these instances share the same class (for example, "SAP_SE" and "BASF_SE" can have different attribute data available, even though they share the same class "Company"). On the other hand, relational databases can be represented in the knowledge graph format (i.e., knowledge graphs can be a higher-level abstraction of relational databases).

[0236] In some examples, nodes in a directed graph (e.g., a knowledge graph) can be organized in a hierarchical structure, where lower-level nodes (representing more specific objects) can be connected to higher-level nodes (representing more general objects) through one or more edges. A lower-level node (or the lower-level object it represents) can be referred to as a descendant of a higher-level node (or the higher-level object it represents), and a higher-level node (or the higher-level object it represents) can be referred to as an ancestor of a lower-level node (or the lower-level object it represents).

[0237] The method may also include receiving one or more rules corresponding to the subset of the directed graph. The rules may be reasoning, logic, inference, or RDF pattern rules. The method may also include materializing the subset of the directed graph by applying the rules to a plurality of statements to compute additional statements. Materializing the subset of the directed graph may be described as adding contextual data or a reference to contextual data to the subset of the directed graph.

[0238] Materializing a subset of a directed graph may be achieved by applying inference or applying (inference) rules to the subset of the directed graph.

[0239] Examples of rule numbers are as follows:

[0240] 1. Each object of the predicate "dbo:foundationPlace" is a country

[0241] 2. Each subject of the predicate "capitalOf" is a city

[0242] 3. Every object of the predicate "capitalOf" is a country

[0243] The first rule can be implemented by setting the scope of the "dbo:foundationPlace" predicate so that its object must be an instance of the Country class. The second rule can be implemented by setting the domain of the ":capitalOf" predicate so that its subject must be an instance of the City class. Similarly to the first rule, the third rule can be implemented by setting the scope of the ":capitalOf" predicate so that its object must be an instance of the Country class.

[0244] The following is an example of reification (i.e., reasoning). The reification is based on the following statement:

[0245] :Madrid:capitalOf:Spain.

[0246] and the following properties:

[0247] :capitalOf rdfs:domain:City.

[0248] :capitalOf rdfs:range:Country

[0249] :capitalOf rdfs:subPropertyOf:locatedIn.

[0250] Thus, concretization can include combining statements with one or more attributes. More specifically, concretization can include combining statements with attributes that restrict the subject or predicate of the statement (e.g., attribute restrictions). Combinations can be used to determine further statements (e.g., classes of which the subject of the statement is an instance and / or classes of which the object of the statement is an instance). Concretization can be understood as determining statements that can be implicitly derived from a directed graph and adding the determined statements to the directed graph. Three numbered reasoning examples are as follows:

[0251] 1.:Madrid:capitalOf:Spain.

[0252] :capitalOf rdfs:domain:City.

[0253] →:Madrid a:City.

[0254] 2.:Madrid:capitalOf:Spain.

[0255] :capitalOf rdfs:range:Country

[0256] →:Spain a:Country.

[0257] 3.:Madrid:capitalOf:Spain.

[0258] :capitalOf rdfs:subPropertyOf:locatedIn.

[0259] →:Madrid:locatedIn:Spain.

[0260] Each of the three examples above combines the statement ":Madrid:capitalOf:Spain" with a different property to compute (i.e., derive) an additional statement. In the first example, the statement ":Madrid:capitalOf:Spain" is combined with the property ":capitalOf rdfs:domain:City" to compute ":Madrid a:City", which indicates that the subject of the statement ":Madrid" belongs to the class ":City" (i.e., is an instance of the class ":City"). In the second example, ":Madrid:capitalOf:Spain" is combined with the property ":capitalOf rdfs:range:Country" to compute ":Spain a:Country", which indicates that the object of the statement ":Spain" is an instance of the class ":Country". In the third example, ":Madrid:capitalOf:Spain" is combined with the property ":capitalOf rdfs:subPropertyOf:locatedIn" to compute ":Madrid:locatedIn:Spain", which indicates that the subject ":Madrid" has the property ":locatedIn" with respect to the object ":Spain".

[0261] Before receiving the subset of the directed graph and before computing the string, each of the additional computation statements (ie, the reified statements) may be added to the subset of the directed graph.

[0262] Continuing with the example, given the statement and property before reification, the following SPARQL query will return FALSE:

[0263] ASK{:Madrid a:City.}

[0264] After reification, the same SPARQL query will return TRUE. Given the additional reasoning provided and the logical connections created between statements, reifying the directed graph can increase the effectiveness of the computed strings in training a language model. Furthermore, the power of language model reasoning can increase with the level of detail in the input provided to the language model. Therefore, because reifying the directed graph increases the level of detail in the directed graph, strings computed from the reified directed graph can be more effective in training a language model than strings computed from a directed graph that has never been reified.

[0265] When statements are inserted into a directed graph (e.g., before defining multiple conditions and multiple patterns), the directed graph can be concretized. Thus, the steps of receiving one or more rules corresponding to a subset of the directed graph and the subsequent concretization steps can be performed before defining the multiple conditions and multiple patterns. This can result in faster string computation from the subset of the directed graph because concretization has already been performed.

[0266] Alternatively, the step of materializing the directed graph may be performed on a subset of the received directed graph. This may have the advantage of providing better performance without using materialization, or may improve the efficiency of creating the directed graph.

[0267] Additionally or alternatively, the method may further include sorting the subset of the directed graph such that nodes are clustered with their neighbors. The sorting may be performed after the subset of the directed graph is concretized. The sorting may include determining a list of nodes in the subset of the directed graph and adding a randomly selected node to a new list of nodes. For each node in the new list of nodes, connecting nodes are determined. For each of the connecting nodes, if the corresponding node is in the subset of the directed graph, the corresponding node is added to the list of nodes. The method may further include deleting a node from the list of nodes.

[0268] The pseudocode for the sorting algorithm described in the previous paragraph is provided below:

[0269] graphNodeSet=graph.getAllNodes()

[0270] backlog=[]

[0271] WHILE graphNodeSet NOT EMPTY

[0272] IF backlog IS EMPTY

[0273] / / perform a random pick / jump in the graph

[0274] backlog.add(graphNodeSet.randomDraw())

[0275] END IF

[0276] FOR r IN backlog DO

[0277] / / cover the full neighborhood of n

[0278] FOR n IN r.connectedTriple DO

[0279] serialize(r,n.edge,n.node)

[0280] IF n.node IN graphNodeSet

[0281] backlog.add(n.node)

[0282] END IF

[0283] END FOR

[0284] graphNodeSet.remove(r)

[0285] END FOR

[0286] The serialize() function above can verbalize the corresponding triples ("r" - subject, "n.edge" - object, "n.node" - predicate). In other words, the serialize() function can convert statements from a subset of the directed graph into a serialization format such as RDF / XML, RDFa, Notation3 (.n3) or Turtle (.ttl), N-triples, or JSON-LD.

[0287] The sorting algorithm can be called a clustering algorithm and can ensure topicality (i.e., the calculated strings are close to each other in the sense that they refer to similar topics or the same topic). In other words, adjacent calculated strings are semantically similar. This can increase the effectiveness of the strings when training a language model.

[0288] The method may further include training a language model using the calculated character string.

[0289] For example, providing the calculated string as input to the language model can include using the calculated string to train (e.g., further train) the language model. For example, the language model can be pre-trained or extensively trained, but the training may not include data in the subset of the knowledge graph. Therefore, using the calculated string to train the language model can expand the capabilities of the language model and enable the language model to assist in tasks related to data in the subset of the knowledge graph.

[0290] Training a language model using a subset of a directed graph can have the advantage of leveraging or expanding the significant effort and expense that goes into language models. For example, training a large language model (such as ChatGPT, PaLM, Megatron, Titan, or Chinchilla) can take months and cost tens of millions of dollars or euros. Enabling a large language model to apply information from a subset of a directed graph can be a way to further leverage the effort and expense that has already been invested in training a large language model.

[0291] Furthermore, training a language model using a subset of the directed graph can involve fine-tuning the language model (e.g., by applying low-rank adaptation) to optimize the language model for a task or domain (e.g., the domain of a subset of the knowledge graph).

[0292] For further information about low-rank adaptation, see “LoRA: Low-Rank Adaptation of Large Language Models” by Edward Hu et al., published on June 17, 2021.

[0293] In some cases, the language model is a probability distribution over a sequence of words.The language model can be a large language model (e.g., having at least one million parameters or at least one billion parameters).

[0294] In some cases, the language model includes a neural network. The neural network can be a deep neural network (e.g., a neural network having one or more hidden layers). The neural network can have at least one million parameters (e.g., weights and biases) or at least one billion parameters. The neural network may have been trained on unlabeled (i.e., unannotated) text using unsupervised (i.e., self-supervised) learning.

[0295] Additionally or alternatively, the neural network can include a transformer that uses self-attention to differentially assess the importance of each portion of input data provided to the neural network. The input to the neural network can be parsed into tokens, and the tokens can be processed simultaneously by computing weights for the tokens in successive layers of the neural network. The neural network can be designed to process sequential input data. The neural network can include weights that can be changed during runtime (e.g., soft weights).

[0296] According to another aspect, a computer program (eg a computer program product) is provided. The computer program comprises instructions which, when the program is executed by a computer, cause the computer to perform a method according to any one of the preceding claims.

[0297] According to another aspect, a computer-readable medium stores a computer program. For example, the computer program can be tangibly embodied in a computer-readable medium. In other words, the computer-readable medium can be a non-transitory storage medium.

[0298] According to another aspect, a computer system for providing data from a directed graph to a language model is provided. The system includes a database storing the directed graph. The system also includes a software service configured to define multiple conditions and multiple patterns. Each of the conditions has at least one corresponding pattern. The software service is further configured to receive a subset of the directed graph from the database. The subset of the directed graph includes multiple statements. Each statement includes a subject, an object, and a predicate that relates the subject to the object. For each of the statements in the subset of the directed graph, the software service is configured to perform the following operations: when one of the conditions matches the corresponding statement and the pattern corresponding to the condition can be applied to the corresponding statement, calculate a string from the corresponding statement using the pattern. The software service is further configured to provide the calculated string as input to the language model.

[0299] The software service may be a web service that runs on a server and listens for network requests on a port (eg, port 80).

[0300] The subject matter described in the present disclosure may be implemented as a method or on an apparatus, possibly in the form of one or more computer programs (eg, computer program products). Such computer programs may cause a data processing apparatus to perform one or more operations described in the present disclosure.

[0301] The subject matter described in this disclosure may be implemented in a data signal or on a machine-readable medium, wherein the medium is embodied in one or more information carriers, such as a CD-ROM, a DVD-ROM, a semiconductor memory, or a hard disk. In particular, the disclosed subject matter may be tangibly embodied in a non-transitory machine (computer) readable medium.

[0302] In addition, the subject matter described in this disclosure can be implemented as a system including a processor and a memory coupled to the processor. The memory can encode one or more programs to cause the processor to perform one or more of the methods described in this application. Various machines can be used to implement the further subject matter described in this disclosure.

[0303] The details of one or more embodiments are set forth in the exemplary drawings and description below.Other features will be apparent from the description and drawings.

[0304] Hereinafter, a detailed description of the examples will be given with reference to the accompanying drawings. Various modifications may be made to the examples. In particular, one or more elements of one example may be combined and used in other examples to form new examples.

[0305] Figure 1 A subset of a directed graph 100 is shown. More specifically, Figure 1 A subset 100 of a knowledge graph is shown. Nodes of a directed graph are shown as circles, and edges of a directed graph are shown as arrows. The subset 100 of the directed graph includes labels 102, 104, 106, 108, 110, where labels are URIs and are defined in the Resource Description Framework (RDF). Node labels 102 and 104 are objects, and edge labels 106, 108, 110 are predicates. The string "1972-01-01" can also be a node label (i.e., an object) with an xsd:date type.

[0306] Subset 100 of the directed graph includes statement 112 (i.e., a triple statement) with a subject "dbr:SAP_SE," a predicate "dbo:foundationPlace," and an object "dbr:Germany," each of which is a URI defined in RDF. An exemplary serialization of statement 112 is dbr:SAP_SE dbo:foundationPlace dbr:Germany. The schema of the directed graph can be defined using RDF Schema (RDFS) from the World Wide Web Consortium (W3C) or Web Ontology Language (OWL).

[0307] Figure 2 The domain 201 and scope 203 of the attribute ":capitalOf" are shown. The domain 201 and scope 203 may be defined as follows:

[0308] :capitalOf rdfs:domain:City.

[0309] :capitalOf rdfs:range:Country.

[0310] Figure 3 An exemplary SPARQL query 301 of a knowledge graph is shown. Query 301 is configured to determine an answer to the following question: What is the population of cities in China with more than 100,000 residents? The results of executing query 301 are also shown.

[0311] Figure 4 The architecture of a system for providing data from a directed graph to a language model is shown. The language model can be trained using data from the directed graph. The user interface and software components are shown as rectangles, and the storage component is shown with rounded corners.

[0312] The system can take as input a subset 100 of a directed graph (or a reference to a subset 100), custom conditions and custom patterns, and configuration options. The subset 100 can be provided as a set of triple statements. The pattern and configuration storage 409 can store default patterns, while custom conditions and patterns are provided by the client 401.

[0313] Configuration options may include an indication of whether question mode should be used in addition to text mode or exclusively. Configuration options may specify how multiple patterns corresponding to a condition will be handled:

[0314] RUN_ALL: If the condition is TRUE, all patterns assigned to the condition are applied;

[0315] RUN_RANDOM: The patterns are randomly ordered, and the first pattern that can be applied to the statement is used.

[0316] Another configuration option may specify whether directed graphs should be materialized (defaults to TRUE). Other ways of handling multiple modes and other configuration options may also be used.

[0317] The directed graph may be materialized and the statements of the directed graph may be ordered (e.g., by the generation agent 411). Subsequently, a string may be computed from the statements of the subset 100 of the directed graph, such as in combination with Figure 7 discussed.

[0318] Client 401 can be used to interact with software service 403. Client 401 can interact with software service 403 via different user interfaces (UIs) 405 and 407 to maintain schemas and / or configurations in schema and configuration storage 409 or to compute strings from subset 100 of a directed graph via generation agent 411. Schema and configuration storage can be accessed via schema maintenance and access application programming interface (API) 413. Subset 100 can be the entire directed graph or a proper subset of the directed graph identified via a query (e.g., a SPARQL query). SPARQL queries can be constructed via a user interface of an abstract query language (e.g., a low-code or no-code platform).

[0319] The directed graph may be stored in memory 415 , and the character strings computed from the statements of subset 100 of the directed graph may be stored in memory 417 .

[0320] Figure 5 An algorithm for computing strings from statements (i.e., triples) in a subset 100 of a directed graph is shown. In the depicted example, the directed graph is sorted before computing the strings. Before computing (i.e., generating) the strings from the default pattern, it is first determined whether the conditions corresponding to a specific (i.e., custom) pattern match the statements. If the conditions corresponding to the specific pattern match the statements, the specific pattern is used to compute the string. Otherwise, the default pattern is used to compute the string.

[0321] When multiple specific patterns and default patterns are present, then the specific patterns may be checked first, and only if no specific pattern applies may the string be evaluated using one of the default patterns.

[0322] After the string is calculated, a reformulation language model can be used to reformulate the string. The reformulation language model can be different from the language model to be trained. The reformulation language model can be a language model with high or very high accuracy for f in the following reformulation function f:

[0323] f(sentence)=sentence'(sentence, sentence)

[0324] The rewritten language model (also known as a paraphrase language model or encoder-decoder model) can be implemented using Google T5, FLAN-T5, or Quillbot. The rewritten language model can have high accuracy (e.g., at least 90% correct) or very high accuracy (e.g., at least 99% or at least 99.9% correct). The rewritten string can have a greater degree of linguistic variation than the original calculated string. The rewritten string can be provided to a user and can better capture the user's attention than the original calculated string. Alternatively, the rewritten string can be provided to a language model to be trained. When training the language model, the linguistic variation in the rewritten string can produce better results than the original calculated string.

[0325] Rewritten strings may occasionally be incorrect. Therefore, a human can have the option of accepting or rejecting the rewritten string. Accepted and / or rejected rewritten strings can be used to retrain the rewriting language model. Furthermore, accepted and / or rejected rewritten strings can be used to compare the quality of different rewriting functions f.

[0326] Figure 6 An exemplary sorting algorithm for sorting statements in a directed graph is shown. Sorting can prevent the calculated strings from being output in random order. Once the statements are sorted, the same or similar topics can be grouped together. The sorting algorithm can have the following form:

[0327] List <triple>result=sort(Set <triple>)

[0328] Set <triple>corresponds to the representation of the directed graph before sorting, and List <triple>The result corresponds to the representation of the directed graph after sorting. "sort" calls a sorting function, such as Figure 6 Other sorting functions may also be used.

[0329] The graphNodeSet variable initially holds an unsorted representation of the directed graph, and the backlog variable will contain the sorted directed graph when the sorting algorithm completes.

[0330] Figure 7 shows that can be provided as Figure 6 The input to the sorting algorithm is a directed graph. Assuming that node 1 is drawn first, the output will be the following statement:

[0331] 1a 2

[0332] 1b 3

[0333] 1c 4

[0334] 2d 6

[0335] 7e 8

[0336] Each number above represents a node, and each letter represents an edge, such as Figure 7 As shown in Figure 1. Therefore, the sorting algorithm ensures that nodes that are close together in the directed graph are also clustered together in the computed string provided as output, rather than being output randomly. In other words, after being computed, adjacent strings cover semantically similar content.

[0337] Figure 8 A condition is shown to have at least one but may have multiple corresponding patterns (also called text patterns). When provided with a statement (e.g., an RDF triple), the condition can evaluate to TRUE or FALSE. If the condition evaluates to TRUE (i.e., the condition matches the corresponding statement), a determination is made as to whether at least one pattern corresponding to the condition can be applied to the corresponding statement. If the pattern can be applied, a string is computed from the corresponding statement using the pattern.

[0338] Figure 9 Expanded Figure 8 To cover question patterns other than text patterns, and to show that a condition has at least one but may have multiple corresponding text patterns. In addition, a condition may have zero or more question patterns. The syntax of the question pattern may be the same as the syntax of the text pattern.

[0339] Question patterns can enable the computation of conversation-like strings, which can be particularly useful for some language models (e.g., language models that require conversation).

[0340] Figure 10 Expanded Figure 5 The algorithm overrides question mode. Specifically, if the configuration option to enable question mode is set, a specific condition corresponding to the question mode is checked. If the question mode corresponding to the specific condition can be applied, the question mode is used to calculate the string from the corresponding statement. Otherwise, the default mode is used to calculate the string from the corresponding statement.

[0341] The configuration options that specify how multiple patterns corresponding to a condition will be handled can be extended to the problem pattern as follows. Specifically, the following three configuration options can be used to handle multiple patterns assigned to at least one condition:

[0342] RUN_ALL: If the condition matches the statement and multiple problem patterns are assigned to the condition, all problem patterns that can be executed are executed. If there are multiple text patterns and problem patterns, the Cartesian product of the text patterns and problem patterns is executed.

[0343] RUN_RANDOM: If the condition matches the statement, a random order is determined for all available question patterns assigned to the condition. The question patterns are then tested in a top-down manner. The first question pattern that can be executed is executed and processing stops.

[0344] RUN_ALL_QPATTERNS_RANDOM_TEXT_PATTERN : Use all question patterns, but if multiple text patterns exist, only use the random text pattern to generate the answer.

[0345] Other methods of handling multiple patterns assigned to a condition are also available.

[0346] Example 3 - Example relationships between knowledge graphs, ontologies, and meta-ontologies

[0347] Figure 11 1108, an ontology 1110 for the knowledge graph, and a meta-ontology 1112 for the ontology. Ontology 1110 can be used to describe elements of knowledge graph 1108. Similarly, meta-ontology 1112 can be used to describe elements of ontology 1110. Knowledge graph 1108 can be considered an instance of ontology 1110 in some ways, and ontology can be considered an instance of meta-ontology 1112. Generally, concepts are described from a more specific perspective to a more conceptual perspective, moving from knowledge graph 1108 to ontology 1110 and then to meta-ontology 1112.

[0348] Aspects of the present disclosure are described with respect to the specific example of motor vehicles, which may be generally referred to as vehicles or more specific types of vehicles, such as automobiles and motorcycles.

[0349] Looking at ontology 1110, it can be seen that the ontology describes several classes 1118, such as vehicle 1118a, car 1118b, or motorcycle 1118c. Ontology 1110 also defines a plurality of properties 1122, such as properties that can be possessed by class 1118, or more specifically, properties that can be possessed by an instance of a class. In this example, property 1122a can be an isA property, such as a property that can be used to define a particular instance as an instance of a particular class 1118. Properties 1122b and 1122c describe specific properties of class 1118, such as the property of having a license plate or the property of having an owner, respectively. As described, ontology 1110 itself can be considered an instance of meta-ontology 1112, and instance identifier 1126b indicates that the ontology is an instance of the meta-ontology, being an instance of ontology concept class 1150a of class 1150 of the meta-ontology.

[0350] The knowledge graph 1108 is shown as including instances 1160 of the vehicle class 1118 and instances 1162 of the motorcycle class 1118c. Each instance 1160, 1162 includes values ​​1160a, 1160b, 1162a, 1162b for attributes 1122b, 1122c. The knowledge graph 1108 also defines various relationships 1170. For example, a relationship 1170 may associate a particular node with a particular class 1118 in the ontology 1110, or may associate a node with a particular attribute 1122 and its corresponding value.

[0351] As described, meta-ontology 1112 describes, at a higher conceptual level, the concepts used in ontology 1110. For example, ontology 1110 has specific classes 1118, and meta-ontology 1112 specifies the concepts of the classes. Similarly, meta-ontology 1112 defines concepts such as properties and relationships, including properties where one object can be an instance of another object.

[0352] For the purposes of this disclosure, "object" is used to refer to a specific instance of a class, where a class is represented by a set of instances. Classes and their instances can have attributes, including data type attributes and object attributes. For example, object attributes can refer to relationships between objects (which can also be called instances, entities, or resources), while data type attributes associate objects with literal values.

[0353] Example 4 - Example Attribute Identification Process

[0354] Figure 12 A flow chart of a process 1200 for automatically extracting attributes from a data set is shown. At 1210, a resource is presented. A resource corresponds to data representing one or more instances of a class. Examples of resources include electronic documents, such as text documents or tables. Resources contain electronic data, but need not be in any particular format. The data can be structured, such as in CSV format, or can be unstructured, such as free text in a word processing document. In some cases, rather than providing a resource directly, the resource location can be specified, such as using a URL, and the URL can be used to retrieve the resource's data.

[0355] In at least some implementations, data from a resource may be extracted and converted to one or more common formats. For example, a document (or more generally, a "file") having text may have the text extracted as a string. The string may be used to generate a graph of the concepts in the string, such as a graph in RDF format. Some documents (such as web pages) may natively include RDF descriptions (such as using JSON-LD, Microdata, or RDFa), and therefore the RDF description may be extracted in RDF format. Some documents may have some text already represented in RDF format and other text represented only in a non-RDF format. In this case, text elements that are not in RDF format may be extracted as strings, converted to RDF format, and optionally connected to an existing RDF description in the resource.

[0356] An instance graph is created at 1220. The input may be text extracted from the document at 1210. The input may be processed using information extraction techniques, such as using OpenNRE (github.com / thunlp / OpenNRE, where NRE stands for Neural Relation Extraction), which may extract relationships from the input text. In some cases, a knowledge graph may be generated using coreference resolution, named entity linking, and relationship extraction (such as described at neo4j.com / blog / text-to-knowledge-graph-information-extraction-pipeline / ). Typically, information extraction may be performed using one or more of pattern matching, machine learning, named entity recognition, or dependency parsing, including as known in the art of information extraction, natural language processing, and knowledge graph construction. Another suitable information extraction technique is disclosed in U.S. patent application No. 17 / 973,322, filed on October 25, 2022, which is incorporated herein by reference to the extent not inconsistent with the present disclosure.

[0357] Using the instance graph, a matching process is performed at 1230. The matching process at 1230 produces a list of suggested attributes, as well as a list of instances that contain such attributes. Note that attributes do not need to be associated with classes, and the list of attributes does not need to all use the same name for attributes that represent a common semantic concept. That is, the matching process can account for differences in the terminology used to describe the same attribute. For example, different instances might use "fuel economy," "mileage," or "fuel efficiency" to refer to the same attribute.

[0358] While any suitable matching technique may be used, suitable matching techniques include those described in U.S. Patent Nos. 11,487,721 and 11,263,187, which are incorporated herein by reference.

[0359] The results of the matching process 1230 are provided as input to the attribute confirmation and validation operations 1240. The operations at 1240 include selecting attributes to be used in the ontology, including resolving differences in the terminology / labels for specific attributes. If necessary, the instance graph obtained at 1220 can be converted to use the aligned vocabulary. In some cases, the operations at 1240 can be at least partially automated, while in other cases, at least some operations can be performed by a user.

[0360] Example 5 - Example computing environment implementing the attribute identification process

[0361] Figure 13 An example computing environment 1300 is provided in which the disclosed technology can be implemented. For example, the computing environment 1300 can be used to execute Figure 12 Operations of process 1200.

[0362] The computing environment 1300 includes a computing system 1304 in communication with a client 1308. The computing system 1304 may include a user interface 1312 that can be used by the client 1308 to perform various operations, as will be described.

[0363] The computing system 1304 also includes a source manager 1320. The source manager 1320 can be responsible for obtaining documents, including, in some cases, extracting data from the documents. The source manager 1320 can include one or more source adapters 1322, shown as 1322a, 1322b. The source adapter 1322 is configured to extract data from a specific source type, which can include formatting the data or removing formatting from the data. Consider a document generated by a word processor. The document can include text as well as formatting information for the text, such as field codes, font style, or spacing information. The source adapter 1322 can be configured for a document type generated by a word processor, which can parse the document to extract text while removing non-text information. Similarly, the source adapter 1322 can be configured to extract data from a database table, a JSON document, or a web page.

[0364] The source adapter 1322 may be configured to implement the interface INTERFACE extract(PARAMETER Source f) RETURNS ONE OF [RDF, String].

[0365] The source manager 1320 can access a source store 1326. The source manager 1320 can retrieve documents from the source store 1326, such as for processing the documents using the source adapter 1322. The source manager 1320 can also cause text extracted from a source to be saved in the source store 1326.

[0366] A user, such as client 1308, can cause source manager 1320 to perform various actions, such as through user interface 1312. For example, a user can cause documents to be loaded into source store 1326, and can request that a particular document be processed by source adapter 1322, or create groupings of sources in the source store (such as groupings of sources believed to represent related instances to be used in extracting properties of an ontology).

[0367] Computing system 1304 includes a graph generator 1332. Graph generator 1332 may execute Figure 12 At least part of operation 1220 of process 1200. Graph generator 1332 may include functionality for implementing named entity recognition and relationship extraction. Graph generator 1332 may access source storage 1326 and may store graph information, such as storing triples in a temporary triple store 1336.

[0368] Match executor 1344 can access temporary triple store 1336 and identify attributes that are the same but referred to using different terms, as in operation 1230 of process 1200. In some cases, mappings may already exist between different schemas, and previous mappings can be retrieved from mapping store 1348, which can reduce the time and computing resources required to perform the matching process. The matching results generated by mapping executor 1344 can be stored in triple store 1360. The triple store can also store triples associated with the attribute lists identified as a result of process 1200.

[0369] Example 6 - Example Source for Attribute Identification

[0370] 14A to 14C Example sources are shown, such as sources from which text may be extracted. 14A to 14C The sources provide information for one or more car models, where a given car model, for example, may represent an instance of a car, and where a particular car instance may be described by the one or more sources.

[0371] Figure 14A A source 1400 is shown, containing descriptions of a car model "SE" produced by the automotive company SAP. Note that source 1400 includes various keywords that a human could identify as relevant to properties of a car. However, the disclosed technology provides ways to automate the creation of an ontology / attribute identification, including in ways that would not be performed by a human. For example, a human could review instance data for one or two car models and then use intuition to extract properties, such as those of the car class. For example, a human would not create an instance graph from a single source or perform the matching process described.

[0372] exist Figure 14A Examples of words that can be associated with attributes include "electric power," "vehicle," "plug-in," "hybrid," "kW," "hp," "fuel efficiency," "electric power consumption," "range," and "emissions." Figure 14B Another source 1420 is shown, describing a SAP SE virtual car. Note that source 1420 includes some terms and attributes that correspond to those used in source 1400. For example, source 1420 also includes terms such as "kw," "hybrid," and "hp." Source 1420 also includes terms that are not present in source 1410 but could be attributes of a car, such as "gasoline engine" or "engine," "torque," "accelerates," "adaptive," "suspension," and "differential." Note that some of these terms can indicate attributes, while others can indicate attribute values. For example, "differential" can correspond to the "differential type" attribute, and "sport" can be an example literal value for "differential type."

[0373] Figure 14C Similar to Figure 14A and Figure 14B , has text for source 1430, where at least some of the text may correspond to attributes (or literal values) that are also present in source 1400 or source 1420, where the terms used for the attributes may be the same or different. Source 1430 may include attributes or example literal values ​​that are not in source 1400 or 1420, and may omit attributes or literal values ​​that are in source 1400 or source 1420. For example, source 1430 does not include "differential," which is a term included in source 1420.

[0374] Example 7 - Example user interface for performing attribute identification process and example attribute counting and alignment process

[0375] Figure 15 An example user interface 1500 is provided in which a user can, for example, Figure 12 Source files to be analyzed are selected during operation 1210 of process 1200. In panel 1510 of user interface 1500, a user can select specific files to be analyzed, such as by adding files using control 1512, adding all files from a specific folder using control 1514, or deleting specific files using control 1516. Field 1520 displays a list of currently selected source or input files 1522. Typically, a user can select files that are expected to contain instances, such as instances of one or more classes whose properties are to be extracted, such as for building an ontology.

[0376] User interface 1500 provides access to functionality that can be used to determine instances present in files in field 1520 via count instances control 1528. For example, selecting control 1528 can result in identification of instances in files 1522 present in field 1520. As discussed, multiple files can represent / describe instances.

[0377] Identifying instances can be performed in various ways. For example, the structure of file 1520 can be analyzed, such as to determine whether common elements exist. In the example shown, file 1520 is identified by a URL, and the URL has the form of a car manufacturer name followed by a car model name. In other cases, determining instances can be based on various heuristics, such as at least temporarily assigning files to common instances based on term usage counts. For example, if "SE" is used multiple times in a file or set of files, it can be determined that "SE" may represent a specific car instance.

[0378] Panel 1540 lists instances 1542 associated with input file 1522. Instances 1540 may be instances obtained by automatically analyzing the contents of input file 1522, or may correspond to instances manually added by a user. For example, the user may choose to delete instance 1540 using delete instance control 1546 or to add an instance using add instance control 1544.

[0379] Figure 16 A user interface 1600 is provided through which a user can initiate a knowledge graph discovery or update process, such as corresponding to Figure 12 Actions performed during operation 1210 of process 1200. The user interface 1600 provides a panel 1610 that lists instances 1614 to be used in knowledge graph creation or update. The instances 1614 may correspond to Figure 15 Instance 1540.

[0380] Panel 1620 lists input (or source) files 1624 corresponding to the selected instance 1614a of the instances 1614. Input files 1624 correspond to Figure 15 1614a. As previously described, multiple input files 1522 may describe the same instance 1614. User interface 1600 may provide controls 1628 that allow a user to add input files to the file set of instance 1614a, as well as controls 1632 that allow a user to delete such files. Panel 1636 provides a preview of a selected input file 1624a of input files 1624. A user may initiate the diagram creation / update process for one or more instances 1614 by selecting user interface control 1640.

[0381] Figure 17 A user interface 1700 is shown that allows a user to view and optionally modify the knowledge graph determined after selecting the user interface control 1640. The panel 1710 lists the instances 1714 for which the knowledge graphs are determined and may correspond to Figure 16 1614. Panel 1720 transfers an instance diagram of the selected instance 1714, such as instance diagram 1724 of instance 1714a.

[0382] Graph 1724 includes nodes and edges, where nodes may correspond to instance nodes 1728, property nodes 1730, and literal value nodes 1732. Edges may represent relationships, such as node 1740 indicating that an instance "hasProperty" or node 1742 that a property "hasValue."

[0383] Optionally, user interface 1700 includes controls that may allow a user to modify the graph, as shown in example graph 1724. User interface controls may include a control 1750 for adding a node, a control 1752 for deleting a node, a control 1754 for adding an edge, or a control 1756 for deleting an edge.

[0384] If targeted Figure 12 As described in operation 1230 of , a matching process may be used to determine properties associated with different instances of a particular ontology concept. The mapping software function may implement the interface: INTERFACE match (GraphSet G) RETURNS Alignment.

[0385] Some properties may be common to all or a large number of concept instances, while other properties may be less common, and some may even be specific to a single instance. When determining the properties of an ontology concept, it may be useful to determine the frequency of occurrence of a particular property. Often, it is desirable to determine which instances have properties with the same semantic meaning, even if the term / label used for the property differs between instances.

[0386] Figure 18 Example pseudocode 1800 is provided for performing instance alignment and counting the number of occurrences of a particular semantic attribute, regardless of the applied label. In loop 1810, pseudocode 1800 performs an alignment between a particular instance graph and the final graph constructed for the set of instance graphs. Specifically, line 1814 calls the "match" function, which performs the alignment. In line 1818, for an attribute in the instance graph being processed, if the attribute already exists in the alignment, the attribute's count is incremented by one. If the attribute does not yet exist in the alignment, its count is set to one.

[0387] Figure 19A A user interface 1900 is shown that presents the results of the matching process from the instance graph, including attribute counts. The user interface 1900 includes a table 1910 having a column 1914 for attributes identified during the graph analysis and matching process, a column 1916 indicating the number of times the attribute was observed in the instance graph, and a column 1920 that accepts user input as to whether a given attribute should be included in the "final" ontology generated from the instance data.

[0388] User interface 1900 includes a control 1930 that allows a user to add an attribute, a control 1932 that allows a user to modify an attribute, and a control 1934 that allows a user to delete an attribute. Modifying an attribute may include changing the name / label applied to the attribute. That is, the names of the attributes in column 1914 may be attribute names automatically selected from the instance data. In certain implementations, a default name is selected that corresponds to the name that occurs most frequently in the instance data, or the name that is most commonly used for a set of discrete instances (i.e., the number of instances in which the name is used at least once is counted).

[0389] Figure 19B User interface 1900 is shown after control 1932 is selected. In this case, window 1950 allows the user to select a name from a drop-down list 1954, where the options may correspond to names identified in the input set. The user may be allowed to enter a different attribute name that does not appear in the instance data. Window 1950 is also shown as including a field showing a range (such as a data type and, optionally, a range or list of values ​​observed in / consistent with the instance data).

[0390] In other implementations, instead of or in addition to having a user add attributes to an ontology, attributes can be added automatically, including whether they meet defined criteria. For example, you can define rules by which an attribute will be added to a set (for an ontology, for example, if it appears more than a threshold number of times in a set of instances, or appears in a percentage of instances above a threshold amount).

[0391] Furthermore, in some cases, the graph can be modified based on the final set of selected attributes, such as Figure 18 The finalGraph is defined in pseudo code 1800. For example, nodes that do not correspond to instances or attributes of the selected attribute set can be deleted from the graph.

[0392] Example 8 - Example Attribute Evolution

[0393] Figure 20 This diagram shows how the properties of instances of a particular class (or subclass) defined in an ontology evolve over time. Instance graph 2010 represents a particular car model 2012 (an instance of the Car Model class). When processing this instance, the ontology includes the following properties: empty weight 2018, engine type 2020, number of doors 2022, and an indication of whether the car has seatbelts 2024. Corresponding values ​​2028, 2030, 2032, and 2034 are associated with these properties 2018, 2020, 2022, and 2024 in instance graph 2010.

[0394] Instance graph 2010 may correspond to a portion of a knowledge graph, and thus values ​​2028-2034 may be associated with properties of the knowledge graph, regardless of whether some properties may not currently be represented in an ontology associated with the knowledge graph. Alternatively, instance graph 2010 does not correspond to a knowledge graph, and the instance graph may have properties that do not correspond to properties in the knowledge graph or properties in the ontology. That is, the disclosed technology may be applicable to instances similar to instance graph 2010. Figure 1 Even if the instance graph is not a formal knowledge graph or is not associated with a knowledge graph, it can be used as long as a technique is provided for tracking changes in attributes and attribute values ​​that are identified as being of interest to a particular user (or computing process).

[0395] It is noteworthy that when instance graph 2010 is initially created, the ontology does not include attributes related to electric vehicles, such as the ability of an electric vehicle to recharge its battery using regenerative braking or deceleration, i.e., regenerative charging (recuperation). However, at a later time, another instance is processed to generate instance graph 2050 for a particular car model (instance) 2052. In addition to the existing attributes 2018, 2020, 2022, 2024 and their associated values ​​2058, 2060, 2062, 2064, instance graph 2050 introduces a "recuperation" attribute 2070 and its corresponding value 2072, indicating that vehicle instance 2052 has regenerative charging capabilities. Note that, despite being present in instance graph 2050, recuperation attribute 2054 is not present in the ontology of the vehicle class.

[0396] The disclosed technology can identify the occurrence of new instance attributes and can track the occurrence of attributes, including adding the attribute to a related ontology or prompting a user to consider adding the attribute to the ontology.

[0397] Now consider attribute 2024 indicating that a car has seat belts. Over time, instances (such as instances generated from a particular electronic document) may no longer refer to that attribute, even though the car reflected by the instance includes seat belts. As an example, seat belts are currently mandatory for inclusion on all vehicles sold in the United States. Before the law mandated the inclusion of seat belts, including seat belts in a vehicle was optional. Thus, documents describing a particular vehicle may have mentioned whether the vehicle included seat belts. After the law mandated the presence of seat belts in all cars sold, the use of the term "seat belts" in documents describing cars may decline, including because the public assumes that new cars have seat belts to some extent. Maintaining the seat belt attribute may cause user confusion because it may not be clear whether the attribute is optional. Including outdated attributes in an ontology can also make the ontology more complex and more difficult to maintain, update, and understand.

[0398] Maintaining outdated attributes can also lead to data integrity and consistency issues. For example, an aggregation operation performed on instances with an outdated value for an attribute and instances without that value may provide inaccurate results—such as identifying a smaller percentage of vehicles with seatbelts because, even if newer vehicles include seatbelts, that information will not be present in the documents processed to generate the vehicle instances.

[0399] The disclosed techniques can be used to identify when a property is potentially outdated and either mark the property as outdated or prompt the user to consider marking the property as outdated. While in some cases, outdated properties are deleted from the ontology, in other cases, the outdated property remains in the ontology but is marked as outdated. In the latter case, retaining the property but marking it as outdated allows it to be used in historical analysis but can be "ignored" in scenarios where only active properties are of interest.

[0400] Example 9 - Example Compute Environment for Property Management

[0401] Generally, the disclosed techniques involve tracking the use of attributes in instances over time. This tracking includes new attributes that appear in instances. That is, in at least some implementations, the disclosed techniques assume that upstream processes can be used to identify new attributes in instances. In some cases, these attributes can be manually added as instance attributes, while in other cases, the attributes can be determined automatically. For example, the techniques described in Examples 4-8 can be used to identify new attributes.

[0402] Figure 21 Shown as Figure 13 The computing environment 2100 is a modified version of the computing environment 1300. Figure 13 Elements of computing environment 2100 that are the same as or at least substantially similar to elements of computing environment 2100 are identified by reference numerals.

[0403] Compared to computing environment 1800, computing environment 2100 adds a feature information store 2110. Feature information store 2110 can maintain information about attribute usage by instances over time. As will be discussed further, feature information store 2110 can store information about attributes, instances that include the attributes (such as from processed documents), and one or more dates associated with the use of the attributes in the instances. Optionally, feature information store 2110 can include aggregate information, such as a total count of attribute usage over a specific time interval. Information can also include specific attribute values ​​associated with instances that include the attributes.

[0404] Matching and inserting actuator 2114 can be similar to Figure 18 2100 , but may also include functionality for writing information about new attributes or attribute usage to the feature information store 2110. In other implementations, the feature information store 2110 may be used in a computing environment that does not include components of the computing environment 2100. In such cases, another process may be used to store new attributes and attribute usage information in the feature information store 2110. In some cases, aggregate values ​​are calculated using components other than the feature information store 2110 and written to the feature information store. In other scenarios, the feature information store 2110 includes functionality for performing aggregation. In still other scenarios, the feature information store 2110 does not store aggregate values, but rather stores information from which aggregate values ​​can be calculated.

[0405] The computing environment 2100 also includes an attribute manager 2130. The attribute manager 2130 can retrieve information from the feature information store 2110 and, at least in some cases, can also write information to the feature information store. For example, the attribute manager 2130 can write attribute state information to the feature information store 2110, including changes to the state of the attribute. In scenarios where the attribute manager 2130 performs aggregations or calculations, this information can optionally be written to the feature information store 2110.

[0406] The attribute manager 2130 can perform operations such as tracking the use of attributes and making or suggesting attribute state changes based on such tracking. In at least some cases, the user is alerted when an attribute is added or an attribute state change is made, or when such an addition or attribute change is suggested but the addition or change requires user approval. The user interface 1312 can cause an appropriate user interface to be rendered to provide this functionality and can also allow the user to perform actions such as viewing information in the feature information store 2110 (including performing a query on the feature information store and displaying the query results).

[0407] The computing system 1304 can communicate with multiple clients 1308. In performing operations, the property manager 2130 can, in certain implementations, access data from multiple clients 1308. Evaluating data from multiple clients 1308 can provide a more comprehensive view of property usage. In some cases, evaluating data from multiple clients 1308 can result in changes to the client's ontology. In other cases, evaluating data from a single client 1308 that results in a change or suggested change to the ontology can be used to change or suggest an ontology change to another client using the same ontology or an ontology that includes at least some overlapping properties.

[0408] Example 10 - Example storage of attribute usage data

[0409] Figure 22 An example data object that can be used to store characteristic information of an attribute is shown in Figure 21 The characteristic information is stored in 2110 for use.

[0410] Data object 2210 tracks the use of a particular attribute 2212a, including through the use of a business timestamp 2212b and a system timestamp 2212c associated with a particular document having a document identifier 2212d. Data object 2210 can be implemented in various ways, including as a relational database table or in a key-value store. Data object 2210 can be implemented as a CSON / JSON object. A single data object 2210 can be used for a single attribute, or a single data object can track information for multiple attributes.

[0411] One advantage of the structure of data object 2210 is that it allows documents to be associated with specific attributes. Suppose initially a document has an attribute, but the attribute is not active in the ontology, so no data for the attribute is recorded (such as in a data object, such as a table or view). Later, if the attribute is changed to an active state, data object 2210 can be used to identify documents with that attribute, and the document can be reprocessed so that the value of the now active attribute can be identified and stored.

[0412] Data object 2220 tracks the state of a property over time. Data object 2220 can be implemented in a similar manner to data object 2210 and can also be used in feature information store 2110. Data object 2220 can track information for a single property or multiple properties.

[0413] Data object 2220 is shown with three possible values ​​for a state attribute 2222b for a particular attribute 2222a. The states are active, present, and outdated. An active attribute refers to an attribute that is active in the ontology. When processing an instance / document, it is determined whether the instance / document has the attribute and the appropriate value can be recorded.

[0414] An attribute is emerging when it has been identified as an attribute but has not yet been promoted to active status. For example, the attribute may not be used frequently enough to qualify for active status. The disclosed technology can track the use of attributes over time. If the attribute is used more frequently, such as within a specific time interval, the attribute can be changed to active status, or the user can be prompted to consider changing the attribute to active status.

[0415] Deprecated attributes are those that were once active, but whose recent usage has become relatively rare. In some cases, the threshold for deprecating an attribute may be the same as the threshold for making it active. In other cases, the thresholds may be different. For example, the threshold for deprecating an attribute may be higher than the threshold for changing an attribute from present to active. In other words, once an attribute is active, its usage must be below the threshold that first made it active.

[0416] Other status types can be used in the present disclosure. For example, a property can have a status of "propose active" or "propose obsolete." These status types can be useful when user input is required for a status change based on at least a set of criteria. A user can approve a change from "propose active" or "propose obsolete," which then results in the property having an "active" or "obsolete" status. In some cases, a combination of methods can be used, such as proposing a change to active or obsolete at one threshold, or automatically making the change if a higher threshold is met.

[0417] For each state, the data object 2220 also tracks the total count 2222c of instances with the attribute, the count 2222d of instances with the attribute over a particular interval, the start date 2222e and end date 2222d of the interval, and the value 2222e observed for the attribute during the interval. The value 2220e can be used to determine why the attribute may have changed state between intervals.

[0418] In the case of cars, the "Engine Type" attribute may have shown increased usage. Analyzing the values ​​for "Engine Type" may reveal that the most recent interval has a new value of "Electric" for Engine Type compared to earlier intervals, which only had values ​​of "Gas" or "Diesel." A user or process trying to decide, for example, whether to change "Engine Type" from Obsolete back to Active or from Present to Active can use the new "Electric" value to help determine why this increase in usage is occurring and how this may influence the decision to make a state change.

[0419] As suggested by the above discussion, a given attribute can have multiple intervals for a given state. An attribute can be new and have an Appearing state, be used frequently enough to have an Active state for a period of time, after which the attribute becomes Obsolete. Circumstances can change such that the attribute is used again or more frequently and promoted to the Active state again, optionally after again proceeding to the Appearing state. The increased frequency of use may (or may not) coincide with a new value for the attribute. Thus, an attribute has two intervals associated with the Active state. These types of cycles can similarly result in multiple intervals of the Obsolete or Appearing states, or other states such as Suggested Active or Suggested Obsolete.

[0420] Total count 2220c and interval count 2220d can be implemented in different ways as needed. In one implementation, total count 2220c represents all time periods in which an attribute has a given state, and the intervals associated with interval count 2220d may represent specific intervals in which the attribute has a given state. That is, when an attribute has a particular state, data object 2220 may have a "record" for each time period. In other implementations, when an attribute has a particular state, multiple intervals may exist within the entire continuous time period. In this case, total count 2220c may represent the total time period, and interval count 2220d may represent specific intervals within the total time period.

[0421] Example 11 - Example Intervals and Interval Membership Criteria

[0422] Aggregations (such as over a specific interval or "window") can be used to determine interval counts (how often an attribute is used over a time period). Intervals can be defined in terms of a date and a time period starting from that date. Typically, windows are defined in terms of a specific period before a given date (e.g., two weeks, a month), but intervals can alternatively be based on a time period after a given date. Time periods can be expressed in any suitable way, such as a fixed number of years, months, weeks, or days. In other cases, intervals can be specified using specific start and end dates.

[0423] Figure 23 An interval 2310 of attributes is shown. Interval 2310 is defined by a date D 2314 and an interval size. The start date 2318 of interval 2310 is determined by subtracting the interval size from date D. Interval 2310 includes multiple documents 2322 associated with the instance, as shown for document 2322a. Multiple documents 2322 include a business timestamp 2322a, a system timestamp 2322b, and a document ID 2322c for the instance. Note that interval 2310 does not necessarily include all documents processed during the interval. Rather, typically, interval 2310 only includes documents with relevant attributes.

[0424] In general, intervals can be calculated in different ways. For example, the interval can be calculated using the date the document was processed, the system timestamp, or the business date (the actual date the document was created, the event occurred, etc.). If available, it is generally desirable to use the business date when determining the date of the document because this represents the "real word" use of the attribute. The system date can be much later than the business date. Consider the record of a birth or wedding. A birth certificate may not be issued, or an update may be made in the system to indicate that the individual has become married until after the actual birth or marriage has occurred. However, the system date can be used if desired, particularly in scenarios where there is little to no difference between the business date and the system date. In other scenarios, the business date is used to determine whether a document should be included in the interval (if available), otherwise the system date is used.

[0425] Example 12 - Example Automatic Instance Counting and Attribute Classification

[0426] The present disclosure provides techniques for periodically calculating counts of attributes within an interval and, based on such counts, maintaining state, suggesting changes to state, or making changes to state. For example, a threshold can be set for a number of instances of an attribute within an interval that would result in a change from present to active or suggested active, or from active to obsolete or suggested obsolete. The threshold can be set manually or automatically.

[0427] Automatically determining the threshold value can include monitoring the number of occurrences of an attribute with an active status in an interval. The threshold for promotion from the occurring state to the active state can be set to the number of occurrences of the least frequently occurring active attribute. The mean or median count can also be used to set the threshold value. In addition, statistical techniques (including distributions) can be used to determine the threshold value. For example, the threshold value can be set to the number of standard deviations of the mean occurrence count of an attribute with an active status. Values ​​for other status types, such as obsolete, recommended obsolete, or recommended active, can be determined in a similar manner.

[0428] As described, in some scenarios, the user approves a state change. The threshold value can be determined with reference to the occurrence value that causes the user to initiate the state change. The threshold value can be calculated in a similar manner as described above, including by using the median, average, or statistical calculation using values ​​associated with user-initiated state changes.

[0429] Figure 24 Example pseudocode 2400 is shown for performing an aggregation calculation on a property within an interval and then using the result to determine whether the state of the property should be changed. Aggregate function 2410 has parameters for the location of property feature information (or an object with feature information) and a value for the interval duration. In example pseudocode 2400, an interval (such as a start date and an end date) is determined by subtracting the interval duration from the current date. In other implementations, aggregate function 2410 may have parameters including the interval duration and a reference date based on which the interval will be calculated, or may have parameters including a start date and an end date for defining the interval.

[0430] In loop 2414, pseudo-code 2400 loops through all attributes in the feature information and calculates the sum of instances with the attribute in the interval.

[0431] Function 2430 assigns a state to an attribute of the feature information during the interval. The state is determined with respect to a threshold, which can be set manually or determined as described above. If the count for the attribute exceeds the threshold and the attribute does not currently have an active state, conditional statement 2434 applies. Conditional statement 2434 sets the state of the attribute to "suggest active"

[0432] If the threshold is met and the attribute currently has an active state, conditional statement 2436 applies. In this case, the attribute remains set to the active state. If the threshold is not met and the attribute currently has an active state, conditional statement 2438 applies. Conditional statement 2438 sets the attribute state to "suggested obsolete". If the threshold is not met and the attribute currently has no state, conditional statement 2440 applies. In this case, the attribute may be set to the "appearing" state. Although not shown, function 2430 may include a conditional statement that retains the "appearing" state for attributes that do not meet the threshold but were previously assigned the "appearing" state.

[0433] According to function 2430, the remaining attributes are assumed to be out-of-date, so conditional statement 2442 sets the "out-of-date" status for any attributes that do not meet the previous conditional statement.

[0434] Pseudo code 2400 can be modified in various ways. For example, function 2430 uses a single threshold value. As described, multiple threshold values ​​can be set, including having different threshold values ​​for when an attribute is promoted to an active state and when an attribute is degraded to an obsolete state. Pseudo code 2400 can also be modified to comprise other state changes, for example, state is directly changed from "appearing" to "active", changed to "active" from "suggesting activity", changed to "appearing" from "active", changed to "obsolete" from "appearing", changed to "obsolete" from "suggesting activity" or changed to "obsolete" from "active". These additional state changes can be associated with additional threshold values.

[0435] Example 13 - Example UI screen for viewing property information and modifying property status

[0436] Figures 25 to 31 An example user interface is provided that can be used to perform the disclosed techniques. Figure 25 A user interface 2500 is shown that can be used to add newly emerging properties to a knowledge graph or other set of properties and ontologies associated with the knowledge graph.

[0437] User interface 2500 provides information for a particular instance, such as a document identified in field 2508. A user may choose to open a document by selecting user interface element 2512.

[0438] As described, there may be multiple documents describing an instance. User interface 2500 provides user interface controls 2516, 2518 that allow a user to indicate that the document is a new instance and to enter an identifier for the instance. User interface 2500 also provides user interface controls 2520, 2522 that allow a user to associate a document with an existing instance and the identifier of the existing instance.

[0439] A table 2526 is provided of the attributes 2528 identified in the instance, along with an identification value 2530 for the attribute and an identifier 2532 indicating whether the attribute already exists in an ontology, knowledge graph, or other set of tracked attributes.

[0440] A user can select a property 2528 and obtain additional information about the property by selecting user interface element 2536. Additional details are presented in panel 2540. Panel 2540 includes a table 2542 that lists feature details 2544 of the selected property and its value 2548. The feature details include a current state 2544a, the number of counts for the property within the current interval 2544b, and a total number 2544c of counts for the property, such as over multiple intervals (including all intervals, at least since the time the property assumed the current state).

[0441] Panel 2540 also provides information 2546 about the type of value (such as a data type), such as whether the value is a numeric value, a Boolean value, a string, or is associated with a category with enumerated values.

[0442] Table 2550 provides links to documents / instances having the attribute, which can be selected and viewed by activating user interface element 2560. Table 2550 includes a link or identifier 2554 of the document, a date 2556 on which the document was entered into the system, and a count 2558 associated with the instance. That is, as described, an instance can be described by multiple documents, and the number of counts 2556 can indicate the number of documents that include instances of the selected attribute. The number of counts 2556 can reflect the total number of counts or the number of counts within an interval. Table 2550 can optionally include additional information, such as a business date associated with the instance or document.

[0443] The user can choose to add the attribute to the knowledge graph or other tracked attribute collection by selecting user interface element 2570. The user can select user interface element 2574 to add the values ​​of other instances having the attribute. That is, as discussed, once an attribute is marked as active, previous instances or documents having the attribute can be reprocessed so that their attribute values ​​are captured.

[0444] Figure 26 A user interface 2600 is shown in which a user can view information about attributes that may be considered for transition to an obsolete state. In panel 2610, table 2614 lists attributes 2616, the current state of the attribute relative to the ontology 2618, the number of occurrences of the attribute within an interval 2620, and the start date of the most recent interval 2622. The user can select attribute 2616 and obtain more information about the attribute by selecting user interface element 2624.

[0445] Panel 2630 provides additional information for the selected attribute 2616. In panel 2630, table 2634 lists various feature details 2636 and corresponding values ​​2638 for the attribute. Feature details 2636 include the information provided by the attribute manager (such as Figure 26 To illustrate, a property may have a current state 2636a with respect to the ontology, represented as current state 2618, and a state associated with information analyzed for the property within a particular interval, such as by Figure 24 The feature details 2636 also include the number of counts within the current interval 2636b of the attribute and the total number of counts 2636c of the attribute (which may have a relative Figure 25 the meaning discussed).

[0446] and Figure 25 As with user interface 2500, user interface 2600 may provide data type information 2650 for the values ​​of the attribute observed for instances within the interval, as well as information 2660 about the specific instance / document in which the attribute was observed. A user may change the state of a selected attribute to an obsolete state by selecting user interface element 2664.

[0447] Figure 27 A user interface 2700 is shown that provides information about a plurality of attributes, including details about values ​​associated with such attributes. The user interface 2700 includes a panel 2708 that provides a table 2710, which includes a column 2712a identifying a particular attribute, a column 2712b providing the current state of a given attribute, a column 2712c providing the number of instances that included the attribute within the most recent interval, and columns 2712d and 2712e providing the start and end dates of the current state. Note that for attributes with an active state, column 2712e may not include a value, may use a null value, or may use another value indicating that the attribute state is not associated with an end date.

[0448] A user can select a particular attribute and obtain further details about the attribute by selecting user interface element 2720. The further details are displayed in panel 2730. Similar to user interface 2500, panel 2730 includes feature details 2734 and data type information 2738. In addition, panel 2730 provides a table 2746, which includes columns 2748a and 2748b, where column 2748a lists the values ​​observed for the attribute during the current interval and column 2748b lists the number of occurrences of a particular attribute value within the interval. As described, this information can provide insight into the appropriate status of the attribute, including assessing why the attribute may be recommended for promotion to active status or demotion to obsolete status.

[0449] Additional details for a particular property may be obtained by selecting user interface element 2754, which enables Figure 28 The user interface 2800 is rendered. The user interface 2800 can provide comprehensive information about a particular attribute, including information displayed on other user interface screens. For example, the user interface 2800 displays feature information 2734, data type information 2738, and a table 2746.

[0450] The user interface 2800 displays information about the various states that the attribute has been in in the form of a table 2824, which includes the interval information shown in a column 2826a providing the interval start date, a column 2826b providing the interval end date (if the interval has ended), and a column 2826c providing the state of the attribute within the interval.

[0451] Information about the source describing a particular instance is also displayed in user interface 2800 in the form of table 2840. Table 2840 includes a column 2842a identifying the source, a column 2842b providing the system entry date of the source, and a count of the number of occurrences of the attribute relative to the source.

[0452] The user can choose to change the state of the property to "obsolete" by selecting user interface element 2850. Note that the property of user interface element 2850 can change depending on the current state of the property as viewed on user interface 2800. For example, if the property is currently in an "obsolete" state, user interface element 2850 can allow the user to set the property to an "active" state. In the case of a property in a state such as "appearing," "suggested active," or "suggested obsolete," user interface 2800 can optionally provide a number of user interface elements that allow the user to make state changes. For example, in the case of an "appearing" property, the user can be provided with the option to make the property active or delete the property. For "suggested active," options can be provided to make the property active, change the state to "appearing," or delete the property.

[0453] Figure 29 A user interface 2900 is provided that provides more detailed historical information about the state of a property over time. Specifically, the user interface 2900 provides a table 2910 of usage intervals. Table 2910 includes columns 2914a, 2914b that provide the start date and end date (if applicable) of the interval. Column 2914c provides the state of the property within the interval, while column 2914d provides a total count of instances with the property within the interval. Instance values ​​within the interval are provided in column 2914e, while column 2914f may include comments about the reason for a particular state change.

[0454] As can be seen from table 2910, the active state interval is associated with a significantly higher count than the obsolete state interval. As described, these values ​​can be used in automated processes to set or recommend state changes. That is, if the count observed within the interval is closer to the count of the interval with the obsolete state, the attribute state can be changed from active to obsolete, or such a change can be recommended. If the count is closer to the count of the active state interval, the attribute can remain in the active state, or if it is in the obsolete state, it can be changed from the obsolete state to the active state, or such a change can be recommended.

[0455] The count in column 2914d and the instance value in column 2914f can be used to explain why a state change occurred. For example, in an earlier interval, cars powered by alcohol fuel might have been considered an alternative to gasoline-powered cars. Over time, this fuel source became less likely to find widespread commercial use, so the use of the term "fuel type" may have decreased as gas became the dominant fuel type. At a later date, a similar scenario might have occurred with respect to diesel-powered vehicles, where the attribute changed from obsolete to active, but then changed from active to obsolete again, such as because diesel-powered vehicles were found to have higher-than-expected emissions. New technologies using electricity as the fuel type could make the attribute active again.

[0456] Automated techniques for assigning states to attributes can take into account the appearance or disappearance of specific attribute values. If new attribute values ​​appear, then a stronger recommendation to make the attribute active or recommend active status can be made. If existing attribute values ​​appear less frequently, then a recommendation to make the attribute obsolete or recommend obsolete status can be made.

[0457] User interface 2900 may provide visual aids to help the user understand the usage of an attribute over time, such as a graph 2950 of counts of an attribute over time. Graph 2950 may be provided for a single interval, multiple intervals, or the complete history of an attribute. Figure 29 , graph 2950 shows a sharp increase in the use of an attribute, which can be associated with an active state. Over time, the attribute is used less, but is still commonly used. At a later time period, the attribute is less commonly used, which can correspond to a transition to an obsolete state.

[0458] Figure 30 A user interface 3000 is shown that can provide information about the usage of an attribute over time (such as within a specific interval). The user interface 3000 includes a graph 3010 formed by nodes 3014 representing attributes. Edges 3018 between nodes 3014 indicate relationships between attributes. Attribute relationships can be based on attributes being part of a common class or subclass, having a mutual relationship, an attribute / subattribute relationship, being part of a common attribute chain, or having other logic or constraints that define relationships between attributes.

[0459] Graph 3010 shows attributes and their relationships, but does not provide information about how often the attributes are used. The disclosed technology allows graph 3010 to be modified, as in graph 3020, to provide usage information, such as information about the count of the attribute within an interval. Nodes 3014 in graph 3020 are shown with different types of shading, which correspond to general usage levels. For example, attributes with high usage counts may be shaded more heavily than attributes with lower usage counts. Thus, graph 3020 allows a user or computing process to understand how attributes are "trending."

[0460] Figure 31 A user interface 3100 is provided that shows changes in usage of properties over multiple intervals, as well as capturing information about properties that may have been added or deleted from the ontology / set of properties being tracked.

[0461] User interface 3100 shows various states of graph 3110 in three intervals. Graph 3100 is similar to Figure 30 3106. Similar to graph 3020, graph 3110 uses shading to show information about usage / counts of attributes within a given interval.

[0462] From the version of the graph 3110 at interval 3130 to the version of the graph at interval 3132, it can be seen that node 3136 represents an attribute whose usage has increased, while node 3138 represents an attribute with decreased usage. Graph 3110 at interval 3132 also reflects the addition of a new attribute represented by node 3142.

[0463] Graph 3110 at a later interval 3146 further shows node 3150 whose usage has increased and node 3154 whose usage has decreased.

[0464] Example 14 - Example source for changing information

[0465] Figure 32 An example electronic newsletter 3200 is shown that may have information relevant to at least some users. Newsletter 3200 includes text. The text may be processed, such as using named entity recognition techniques and entity linking, to associate entities in the text with a knowledge graph or ontology.

[0466] The text includes links 3210. In some cases, the disclosed technology can follow links 3210 to other data sources, which can be processed to determine whether they can provide additional instances of knowledge graph elements, or additional information about newly identified instances (such as instances in email 3200).

[0467] Figure 33 Another type of electronic document is shown, email 3300, which may include information relevant to at least some users. Similar to text message 3200, email 3300 includes text that can be processed and linked to elements of a knowledge graph or ontology. The text of email 3300 includes links 3310 that can be followed to other documents that may contain related information.

[0468] As discussed in Example 1, information documents such as newsletters tend to include information that may be relevant to various stakeholders. The relevant information may be just a specific paragraph of a longer text, or it may be available through a link in the document (such as links 3210, 3310). In some cases, information related to a particular topic may be included in the same area of ​​the document, such as Figure 32 and Figure 33 3230, 3330. In the case of links 3210, 3310, this further demonstrates the difficulty for users to manually locate relevant information.

[0469] In a similar manner to how different information may be relevant to different users, the timeliness requirements for users presented with updated information may vary between different users. That is, some users may not need to know the information at all, some users may want to view the information but without particular urgency, and other users may have an urgent need for the information. As will be described, the disclosed technology can consider not only whether the information is relevant to a particular user, but also the priority / relevance of the information if it is relevant to the user.

[0470] The scope of information that is relevant to users can also vary. For example, some users may be satisfied with general information about a particular topic, but other users may want detailed information. In terms of a knowledge graph, this can be expressed as whether one or more specifically identified nodes of the graph are relevant or whether nodes that are related to such nodes should also be considered relevant. When related nodes are relevant, a degree of indirection can be used to limit the scope of what nodes are considered relevant. As will be described, when defining what types of information are relevant to users, users can weight particular topics as more or less interesting, which can be used to determine other information that may be of interest to users even if they have not explicitly identified such information.

[0471] Example 15 - Example data model for intent definition

[0472] Figure 34 An example data model 3400 that can be used in the disclosed technology is shown. The data model 3400 includes a user element 3410. The user element 3410 can represent a specific user or class of users. A given user or class of users associated with the user element 3410 can be associated with zero or more intent elements 3414.

[0473] Intent element 3414 defines a specific topic / collection of one or more knowledge graph or ontology elements that the user is interested in and for which they would like to receive updates regarding changes to the knowledge graph. Intent element 3414 is associated with one or more intent nodes 3418. Intent nodes 3418 correspond to graph nodes 3424, nodes of a knowledge graph or ontology. When a graph node 3424 corresponds to a node of an ontology, the node is also linked to the corresponding node of the knowledge graph, so that when an attribute value in the knowledge graph is added, deleted, or changed, it can be associated with intent element 3414.

[0474] Intent nodes 3418 differ from graph nodes 3424 in that intent nodes can be subject to restrictions or parameters of the particular intent associated with intent element 3414. For example, intent node 3418 can be associated with zero or more restrictive attributes 3432. Restrictive attributes 3432 can be defined as value restrictions on graph attributes 3436. Graph attributes 3436 are typically limited to those attributes defined for the particular class associated with graph node 3424 (where graph node 3424 can correspond to an instance of that class).

[0475] A user may express an intent in terms of a particular graph node 3424, either directly or by associating user-provided input with a corresponding knowledge graph node. However, in some cases, it may be desirable to broaden the intent to include other graph nodes 3424 that the user may be interested in. For example, the user may not be aware that such graph nodes 3424 exist, and therefore may not be aware that they will be relevant to the intent being defined.

[0476] A user may choose to associate a priority element 3444 directly with an intent node 3418, or may assign a priority to an intent element 3414, where the priority is then applied to all intent nodes defined for that priority. Providing the user with the ability to weight particular intent nodes 3418 may provide a more precise expression of the inferred intent. In some cases, including if a user does not explicitly define a priority element 3444 for one or more intent nodes 3418, a default priority may be assigned, including a default priority specified relative to the intent element 3414, and applied to all intent nodes defined for the intent element. In other cases, different priorities may be assigned to particular intent nodes 3418 even if a user does not manually assign a priority element 3444. For example, the system may track usage of intent nodes 3418 relative to intent elements 3414, and may assign a higher priority to intent nodes that are more commonly used or have a higher priority associated with them during automation.

[0477] An update interval 3448 is also defined for the intent element 3414. In some cases, a default update interval may be used if no update interval 3448 is specified. The update interval 3448 may define how often the knowledge graph should check for updates related to the intent element 3414, or at least how frequently such updates should be collected and communicated to the user.

[0478] Although not in Figure 34 , but the data model may include additional components, such as a component that associates intent element 3414 with one or more communication types that should be used to convey update information to the user.

[0479] While the data model 3400 can be implemented in any suitable manner, in a particular implementation, the components of the data model can be implemented as one or more objects in a relational database, such as tables or views. For example, a table can be defined having columns corresponding to the elements of the data model 3400. However, the data of the data model 3400 can be maintained in other formats, including in a key-value store, such as JSON (JavaScript Object Notation) or CSN (Core Schema Notation).

[0480] In a further example, one or more elements of the data model 3400 can be defined as abstract data types or member variables of an abstract data type. An intent element 3414 can, for example, be an abstract data type that can have a user 3410, a priority 3444, or an update interval 3448 as member variables. An intent node 3418 can also be defined as an abstract data type, where the abstract data type of intent 3414 can include an instance of an intent node data type as a data member. The intent node data type can have restrictive attributes 3432 and member variables of a graph node 3424, where optionally, the restrictive attributes can also be implemented as abstract data types.

[0481] Example 16 - Example Intent Registration and Update Process

[0482] Figure 35 is a flow chart of a process 3500 for registering a specific area of ​​the knowledge graph that a user is interested in (called an intent), monitoring the system for new information about that intent, and then alerting the user to such new information. The information used in process 3500 can be stored in Figure 34 In the data object of the data model 3400.

[0483] At 3508, the user registers a definition of their intent, such as through a user interface. The intent can be specified as a node of a knowledge graph or associated with a node of a knowledge graph. In some implementations, the user can be presented with a list of available topics (corresponding to knowledge graph nodes) or a depiction of the knowledge graph, and the user can select a topic / node from there. In another implementation, the user can be presented with documents with content linked to knowledge graph nodes, and the user can select such content, where the corresponding knowledge graph node is added to the definition of the user's intent.

[0484] Also at 3508, in some implementations, the user may provide input to restrict the intent of one or more selected knowledge graph nodes in the definition of the intent. The node selected by the user may represent a class that has attributes for which restrictions may be specified. The value of the attribute set in the restriction may be used to restrict the related results (instances). The restriction may be a positive restriction (the attribute has a given value) or a negative restriction (the attribute does not have a specific value).

[0485] At 3512, the user can assign a priority and an update interval to the intent. Setting either or both of the priority and update interval may be optional. For example, the priority or update interval may not be user configurable, or a default priority or update interval may be used if the user does not override them with input at 3512. As described in Example 15, the priority can be used to determine the inferred intent, as will be further described. Additionally, rather than assigning a priority to the intent as a whole, priorities can be assigned to specific nodes within the intent, where different nodes may be assigned different priorities (and some nodes may have no assigned priorities).

[0486] At 3516, defined intents are registered for the user. A given user can register multiple intents, including intents with overlapping knowledge graph nodes. The same set of nodes can even be used in different intents, such as where in one intent the set of nodes is restricted and in another intent the set of nodes has different restrictions or no restrictions. Different priorities or intervals can be defined for different intents.

[0487] For a registered intent, a listener is registered at 3520. A listener is computer code that implements functionality for monitoring a knowledge graph or other information source about a knowledge graph, as the term "listener" is commonly used in computer science. When an event that triggers a listener is detected, the event can be associated with one or more related intents. An example listener and the message that triggers the listener are defined in the following code:

[0488] import kg_library #Assume this is a knowledgegraph library

[0489] def node_change_listener(node_id,old_data,new_data):

[0490] print(f"Node{node_id}has been updated!")

[0491] print(f"Old data:{old_data}")

[0492] print(f"New data:{new_data}")

[0493] def start_kg_listener():

[0494] kg = kg_library.connect_to_kg() #Connect to the knowledgegraph

[0495] kg.subscribe_to_node_changes(node_change_listener)#Subscribe to node change events

[0496] #Example message that might trigger the listener

[0497] #This could be triggered by an update operation performed on a KG node

[0498] example_message={

[0499] "node_id":"12345",

[0500] "old_data":{"name":"Old Name","age":30},

[0501] "new_data":{"name":"New Name","age":35}

[0502] }

[0503] #Simulate triggering the listener with the example message

[0504] node_change_listener(example_message["node_id"],

[0505] example_message["old_data"],example_message["new_data"])

[0506] In a particular implementation, the changes that trigger the listener are those that involve adding or removing a property value of an instance of a class from the knowledge graph, or a change in a property value. Adding a value to a property may include adding a value for the instance that was not previously extracted from the document describing the instance, or the added value may correspond to a property that was recently added to the class. Similarly, a deleted property value may correspond to a value that was deleted because the property was determined to be obsolete.

[0507] At 3524, the detected changes are collected. Collecting changes can occur at the interval specified in the intent. Alternatively, changes can be collected at other times, typically at more frequent intervals, and processed for a specific intent based on the intent interval. When changes are collected, they can also be associated with a specific priority specified for the intent.

[0508] In a particular implementation, the changes are captured as a set of triples, such as those described in Example 2. The triples can then be verbalized at 3528, also as described in Example 2. At 3532, the triples are submitted to a natural language generator, and a summary is received by the natural language generator. A notification, such as a text message alert, email, or app message, is generated at 3536 and sent to the relevant user at 3535.

[0509] Notifications can be created and sent based on the priority associated with the intent. For example, the selected priority can determine the communication channel used to send the notification. SMS messages can be used for urgent priorities, while emails can be used for lower priorities.

[0510] Example 17 - Example computing environment for intent registration and processing

[0511] Figure 36 It is used to implement Figure 35 FIG36 is a diagram of a computing environment 3600 for process 3500. Computing environment 3600 includes a client 3610 in communication with a subscription service 3614. Client 3610 can define one or more intents stored in a user configuration repository 3618. Subscription service 3614 processes the intents, including instantiating an appropriate listener 3622. Listener 3622 monitors a knowledge graph 3630 stored in a knowledge graph repository 3626.

[0512] The knowledge graph 3630 can be associated with an ontology 3634. As previously described, the ontology 3634 can be used to identify new attributes, which can then be added to the knowledge graph 3630. Information about these attributes in the knowledge graph 3630 can be associated with knowledge graph nodes in the user's intent, thereby enabling notification of changes to be sent to the user. For example, if the user has identified a class instance node as being of interest, a message can be generated if a property value for a new attribute is added to the class instance.

[0513] Subscription service 3614 may include a language component 3640. Language component 3640 may perform actions when processing user requests (such as requests to define intents) and when processing messages to users regarding such intents. In the case of intent definitions, language component 3640 may convert user-provided information into a standardized format for expressing intents, such as Figure 34 When sending information about changes related to an intent to a user, the verbalization component 3640 can convert the "raw" change information into a format that is more easily understood by the user, including providing contextual information that can remind the user of the context that led to the intent being registered (which can include, for example, identifying relevant knowledge graphs or identifying knowledge graph elements selected by the user for the intent).

[0514] The following scenario illustrates how computing environment 3600 can be used in the disclosed technology. A user can express an intent through client 3610. As an example, user Sandra can express the intent as "Please notify me of changes to SAP's LLM usage via email once a week." This "free text" input can be processed, such as using named entity recognition and entity linking (which can be part of linguistic component 3640), to map the intent to a knowledge graph node. Knowledge graph nodes and intervals, as well as preferred communication channels, can be registered in user configuration repository 3618.

[0515] In certain embodiments, user intent may be specified as one or more triples (such as described in Example 2). Given the example intent, the intent may be expressed as:

[0516] Change update on[SAP; uses; LLM]

[0517] Setting: once a week

[0518] Setting: via email

[0519] (Change [SAP; Use; LLM] Update

[0520] Setting: Once a week

[0521] Setup: via email)

[0522] Listener 3622 can check for relevant updates based on an interval. Changes can be represented as triples. An example change set for an example update can be:

[0523] Change Update on[SAP,uses;LLM]:

[0524] {(insert node:ODM; uses; LLM);

[0525] (change sub node:[P1; uses; LLM;]

[0526] →[P1;ConceptVersion;ACD_P1v2,0])}

[0527] (Change [SAP, use; LLM] update:

[0528] {(insert node: ODM; use; LLM);

[0529] (Change child nodes: [P1; Use; LLM;]

[0530] →[P1;Concept version;ACD_P1v2,0])})

[0531] The above changes can be submitted to a natural language generator (which can be part of the verbalization component 3640), which composes an email message to be sent to the user. An example email message is:

[0532] Dear Sandra,

[0533] Heere is the information you requested regarding LLM use in SAP forKW 34:

[0534] -ODM started using LLMs(25.08.022); click here for more information

[0535] -P1 created new Concept(27.08.2022),click here for more information

[0536] Regards,

[0537] You KG Subscription Service

[0538] (Dear Sandra,

[0539] Here is the information you requested regarding the SAP Usage LLM for KW 34:

[0540] -ODM starts using LLM (25.08.022); click here for more information

[0541] -P1 creates new concept (27.08.2022), click here for more information

[0542] respect,

[0543] Your KG subscription service)

[0544] Example 18 - Example detailed computing environment for intent registration and processing

[0545] Figure 37 A more detailed computing environment 3700 is shown in which the disclosed technology can be implemented. The computing environment 3700 includes a client 3708 in communication with a notification system 3712.

[0546] The notification system 3712 provides one or more user interfaces 3716 that a user can access to define, modify, or delete intents, including as previously described. Intents can be processed by a request coordinator 3720. The request coordinator 3720 can store the intent information in a settings store 3724.

[0547] Intent reasoner 3728 can perform various actions with respect to an intent. For example, as described in Example 17 for verbalization component 3640, intent reasoner 3728 can parse the intent and express it in a standardized format (such as a triple format). The more standardized version of the intent can then be stored in settings storage 3724.

[0548] In at least some scenarios, expressed intents can be supplemented. That is, for example, knowledge graph nodes that have relationships with explicitly identified knowledge graph nodes can be added to the intent. In some cases, whether additional nodes are added or the extent to which they are added can be related to user input. For example, a user-provided priority or another expression of how interested the user is in a particular topic can be used for this purpose. This "extended" intent definition can be stored in settings storage 3724.

[0549] The listener service 3732 can instantiate a listener 3734 based on the intent stored in the settings storage 3724. The listener 3734 created by the listener service 3732 can monitor the knowledge graph 3740 in the knowledge graph system 3738 in various ways. Figure 37 In the scenario shown, listener 3734 can monitor change log 3746 stored in change log storage 3744.

[0550] Change log agent 3750 can write a change log to change log storage 3744. For example, when the knowledge graph is updated to add, delete, or change a value associated with a class instance, a log entry can be written to change log storage 3744. In some scenarios, intent reasoner 3728 can also access change log storage 3744. When the expressed intent is modified to include at least some additional nodes, the addition or deletion of nodes in knowledge graph 3740 can result in a change to the extended intent.

[0551] When the listener 3734 detects changes, such as from the change log store 3744, the listener may notify the request coordinator 3720 of any changes. The request coordinator 3720 may then submit the changes to the natural language generator verbalization module 3754. The request coordinator 3720 may use the natural language generator verbalization module 3754 to specify parameters for the verbalization of the changes, such as selecting a particular template instruction for a particular type of communication to be sent.

[0552] The message 3756 generated by the natural language generator verbalization module 3754 is provided by the request coordinator 3720 to the notification module 3760. The notification module 3760 may have communicators 3764 (shown as 3764a and 3764b) for various communication types, such as an email communicator 3764a or a collaboration software communicator 3764b. The communicator 3764 is configured to package and send the message 3756. For example, the email communicator 3764a may paste the contents of the message 3756 into an email template, add a subject line and recipients, and contact the email server to send the message. In some cases, the communicator 3764 may access the settings storage 3724 to retrieve the contact information of the specific user to whom the message is to be sent. In other scenarios, the request coordinator 3720 retrieves the contact information from the settings storage 3724 and may provide this information along with the message 3756 to the notification module 3760.

[0553] Example 19 - Example Determination of Inferred Intent

[0554] As described, in some cases, instead of strictly limiting intents to specific topics specifically identified by a user as reflected by knowledge graph nodes, intents can be built based on those topics and expanded to identify other related topics that the user may also be interested in. For example, consider a scenario where a user expresses interest in nodes belonging to the "movies" class. Under a narrow interpretation, a user might only receive alerts when attribute values ​​for a given movie instance are changed, added, or deleted. However, an instance of the "movies" class may have connections to instances of the "actors" class. In some cases, a user may receive alerts to changes to movie attributes, but the user may also be presented with changes made to actors associated with the movie.

[0555] Optionally, when expressing an intent, the user can weight various concepts (e.g., assigning "priorities" to the overall intent or to specific nodes of the intent), and therefore the knowledge graph nodes to which they correspond. Priorities can be used to determine which nodes are included in the inferred intent, and optionally how often updates will be provided. That is, for example, in some scenarios, updates may be provided at user-specified intervals for a specific intent expressed by the user. Inferred intents can also be generated that include more knowledge graph nodes than the user explicitly specified. Inferred intents can be analyzed at the same frequency as the user expressed them, or at a different frequency. In general, the fewer direct relationships a knowledge graph node has to explicitly specified knowledge graph nodes, the less frequently the knowledge graph will be analyzed for these changes. If the relationship between nodes is too attenuated, the node is not included in the inferred intent.

[0556] exist Figure 38 The concepts of node priority and update frequency are illustrated in the knowledge graph 3800 of FIG. The knowledge graph 3800 includes a plurality of nodes 3810 (shown as nodes 3810a-3810c) connected by edges 3814. Assume that a user provides an intent that directly specifies node 3810a. Due to the assigned priority or assuming that node 3810a is explicitly identified in the intent, updates related to the node are frequently provided.

[0557] Node 3810b represents a node that is not directly specified in the intent, or a node that is specified but has a lower priority than node 3810a. Because node 3810b is inferred or explicitly identified as having a lower priority, updates related to such a node are also provided to the user, but less frequently than node 3810a. Node 3810c represents a node that has a more decaying relationship with the specified node 3810a, or a node that has been assigned a lower priority than node 3810b. Node 3810c has a priority less than the threshold set for "inferred intent," and therefore no updates are provided to the user.

[0558] In some cases, the relevance of a node 3810 can be affected by the number of connections to other nodes, not just the degree of indirection. Figure 39 Graph 3900 includes a plurality of nodes 3910 (shown as nodes 3910a-3910n) connected by edges 3914.

[0559] In the scenario of diagram 3900, assume that nodes 3910a and 3910b are directly identified as related by the user in their expression of intent. The inferred intent can be determined by assigning values ​​to related nodes, where half of the node's value is propagated to related nodes. Given this, half of the value of node 3910b, 0.5, is propagated to nodes 3910c and 3910d, which are directly connected to node 3910b. A similar situation occurs for node 3910a, where 0.5 is propagated to directly connected nodes 3910e and 3910f.

[0560] Now consider node 3910g, which is connected to nodes 3910c and 3910f. Half the value of nodes 3910c and 3910f (0.25 for each node) is assigned to node 3910g. Therefore, node 3910g also has a value of 0.5. Values ​​can be assigned to nodes 3910h through 3910n in a similar manner.

[0561] The technique for propagating node values ​​can be implemented in various ways. For graph 3900, propagation is constrained to be unidirectional. For example, the value associated with node 3910e is propagated to node 3910g, but not further to node 3910c because node 3910c propagates its value to node 3910g.

[0562] In some scenarios, a constraint may be applied where a node can have a maximum value, or where the value of an "inferred node" cannot exceed a maximum value. In one example, a constraint may be applied where an inferred node cannot have a value higher than a directly specified node (1 in the case of diagram 3900). This constraint is consistent with the assumption that the user has directly identified the nodes they believe to be most relevant.

[0563] In a similar manner, it may be useful to normalize node values, such as to fall between 0 and 1. Other constraints may include not propagating values ​​that exceed a set indirection limit. Furthermore, graph 3900 assumes that all edges 3914 have equal weight. It may be that some nodes may be more strongly related than others, and thus propagation between nodes may take into account edge weights and rules that determine how much of a node's value is propagated to related nodes.

[0564] Once the values ​​of the nodes in graph 3900 are determined, they can be compared to a threshold value, where the threshold value determines whether the inferred intent is strong enough to include the inferred node in the intent or to create a new inferred intent that includes the specified node from the expressed intent and the inferred intent. Assume that a threshold value of t≥0.70 is set. Using this criterion, node 3910i will be added to the inferred intent.

[0565] While a "distance factor" of 0.5 is used in the example of knowledge graph 3900, other distance factors may be used. In some cases, rather than using the same distance factor for all nodes, the factor may vary. For example, the distance factor itself may decrease as the number of indirections increases. Different nodes in a knowledge graph may have different relationship strengths or confidences, and this information may also be used as or in conjunction with the distance factor. For example, the total weight used to propagate a value may be determined as the product of the strength of the relationship between two nodes and a defined distance factor.

[0566] Example 20 - Example Intent Definition User Interface

[0567] Figure 40 and Figure 41 Example user interfaces 4000, 4100 are provided that allow a user to define an intent. Figure 40 , user interface 4000 includes a panel 4010 that displays a knowledge graph 4014, which can be an entire knowledge graph or a subset of a larger knowledge graph. Graph 4014 includes nodes 4018 corresponding to instances of classes or subclasses and nodes 4020 representing values ​​(literals) of attributes of nodes 4018. Edges 4026 between nodes 4018 represent relationships between class instances (where one class instance can correspond to the attribute value of another instance), while edges 4030 represent attributes with literal values, thereby linking the instance represented by node 4018 to the attribute value represented by node 4020.

[0568] The user may define an intent by selecting user interface element 4032. The user may provide an identifier for the intent in user interface element 4038 of panel 4034. User interface element 4042 may be used to specify an update frequency for the intent.

[0569] The user can then add node 4018 to the intent by selecting user interface element 4046. For example, user interface element 4046 can be selected, and then the user can select node 4018. For a given node, the user can optionally specify a priority using user interface element 4050. The user can also select user interface element 4054 to add one or more restrictions for the node. The restrictions can be specified as attributes, such as edge 4030 displayed in user interface element 4058, and one or more values ​​displayed in user interface element 4062. In other scenarios, the user can specify a specific combination of attributes and values ​​by selecting node 4020. The technology of the present disclosure can provide updated information to the user regardless of whether the attribute value is a literal or a class instance.

[0570] For some nodes 4018, the user may wish to receive updates more frequently than for the overall intent defined. The user interface 4000 provides a user interface element 4066 that the user can select to be immediately notified of any changes affecting the node 4018 (subject to any value constraints that may have been defined).

[0571] Once the user has finished defining their intent, they can register / activate the intent by selecting user interface element 4070.

[0572] User interface 4100 is generally similar to Figure 40 41. However, in panel 4110, instead of displaying a knowledge graph, content 4114 is displayed. Content 4114 may be a document containing information related to one or more instances of one or more classes of a knowledge graph. In some cases, a user may select specific text to be added to the content of an intent, such as by "right-clicking" after selecting the relevant text. When text is selected, the computing system may confirm whether the text corresponds to an element of a knowledge graph. In other scenarios, and as shown, the document may be pre-processed, and text that corresponds to a knowledge graph element and is selectable for an intent may be visually distinguished. For content 4114, the optional text is shown with a dotted underline. The described text may be selected, and a dialog window 4120 may be presented, which includes an option 4122 for adding a knowledge graph element to the intent, or an option 4124 for deselecting the intent.

[0573] Otherwise, intents can be defined as described with respect to user interface 4000 , where user interface 4100 includes controls that are similarly labeled as in user interface 4000 .

[0574] Example 21 - Example log of knowledge graph changes

[0575] As about Figure 37 As described, in some implementations, changes can be determined by examining log entries. Figure 42 An example log 4200 is provided having log entries 4210 (shown as 4210a-4210e). Each log entry 4210 is associated with a date 4220a and a time 4220b at which the log entry 4210 was written. The log entry 4210 also includes a user identifier 4220c of the user associated with the recorded activity and an identifier of the operation 4220d performed.

[0576] Example log 4200 includes entries 4210a and 4210e corresponding to updates to a knowledge graph. More specifically, the updates correspond to updating a value associated with a particular attribute of a particular instance of a class. Update entries 4210a and 4210e include a triple 4230 expressing the previous state of the knowledge graph and a triple 4234 expressing the updated state of the knowledge graph.

[0577] Log entry 4210b corresponds to deprecating or deleting information from the knowledge graph. Triple 4238 reflects the deprecated information in the knowledge graph.

[0578] Log entries 4210c and 4210d reflect adding information to the knowledge graph. Log entries 4210c, 4210d include a triple 4242 containing the added information. In this case, triple 4242 reflects adding a value to a specific attribute of an instance of a specific class in the knowledge graph.

[0579] A log, such as example log 4200, may be scanned to determine changes associated with a particular intent expressed by one or more users. For example, Figure 37 Listener 3734 can scan the log periodically, such as according to an interval defined for an intent. In other cases, listener 3734 can register information about nodes related to an intent, and components such as change log agent 3750 can notify listener 3734 of related activities. This configuration can be particularly beneficial when a user requests an immediate update for a specific intent or one or more nodes of a specific intent.

[0580] Listener 3734 or another component (such as request coordinator 3720) can extract the log information into a specified format, such as Figure 43 . Format 4300 includes field 4304, which lists the subject of the intent or information associated with the intent. Format 4300 also includes fields 4308, 4316 for added or deleted triples, respectively. Field 4312 for modified triples includes both the old triple and the new triple.

[0581] Figure 43 An example update message 4350 in format 4300 is also provided.

[0582] Example 22 - Example Generation of NLG Prompts and NLG Responses

[0583] although Figure 43 The message format 4300 conveys information about knowledge graph changes, but it is a somewhat difficult to read format. Therefore, the present disclosure provides techniques for conveying information in a more user-friendly manner, which may include customizing the information for the specific communication type (such as email) that will be used to provide the message.

[0584] Generally, the technique involves providing information in message format 4200 to a natural language generator, where the natural language generator is provided with the information in the message format and a standard set of instructions for generating a response. Figure 44 Included are example components of a prompt template 4400 that may be populated and provided to a natural language generator.

[0585] The input includes commands 4404, which are typically static commands. That is, commands are used regardless of the intent being processed and provide general instructions to the natural language generator. The input also includes changelog metadata 4408 and changelog details 4412. These components are specific to a particular update (aggregating information about changes detected for the intent). Optionally, additional custom constraints 4416 can be included in the prompt template 4400. A static closing instruction 4420 for all instructions to the natural language generator can also be included.

[0586] exist Figure 45 4 shows an example prompt 4500 generated using prompt template 4400. Text 4510 corresponds to command 4404. In this case, command 4404 provides general context information, such as interpreting the subsequent changelog data, the task to be performed (generating a "better" summary of the log details), and specifying the output format.

[0587] The text 4514 corresponds to the changelog metadata 4408 of the prompt template 4400. The text 4514 provides information about the intent, the user for whom the intent was defined, and information about the knowledge graph, such as the individual responsible for maintaining the knowledge graph.

[0588] Text 4518 corresponds to change log details 4412. Text 4518 is Figure 44 The format is 4400.

[0589] The text 4522 corresponds to the custom constraints 4416 of the prompt template 4400. In this case, the text 4522 specifies who should be contacted for questions about the message and also generates a summary of the changes. The example prompt 4500 is not shown as including a close instruction 4420.

[0590] While implementations of prompt template 4500 may be standardized in the sense of being used for multiple updates and even multiple intents, multiple implementations of prompt template 4500 may be used. For example, different knowledge graphs, or different uses of the same knowledge graph, may include different instructions as to how information should be presented or generated. As a very simple example, the content of the example prompt 4500 may be modified to generate a response suitable for sending via text message or via in-app messaging. In some cases, a knowledge graph may be assigned to one or more specific implementations of prompt template 4400, and an appropriate template selected based on criteria such as the user or user role, the knowledge graph element selected, the interval selected, or the priority selected for a particular knowledge graph node. In a further example, a user may be able to select which knowledge graph to use with a particular intent. Figure 1 A specific implementation of prompt template 4400 used together.

[0591] Figure 46 Shows the Figure 45 Example response 4600 for example prompt 4500 is shown. As can be seen, example response 4600 is formatted as an email and sent to the user specified in text 4514. Text 4612 provides a summary of the changes, as specified in the custom constraints of text 4522. Text 4616 provides the contents of text 4518, the changelog details, in a more readable format. Consistent with text 4514, the changelog metadata, and text 4522, the custom constraints, text 4620 explains who the user / recipient should contact if they have questions about the changes.

[0592] In some implementations, communications such as example response 4600 can be annotated to further help users understand the change or reasons for the change. For example, the text of the communication can be processed using techniques such as named entity recognition and entity linking to identify knowledge graph elements in the text, such as class instances. These knowledge graph elements can then be highlighted or annotated, or can be associated with links to additional information.

[0593] For example, example response 4600 includes links 4650. These links can be selected and, in some examples, can take the user to additional information describing the change. In some cases, one or more documents can be associated with the change, and link 4650 can be selected to cause such documents to be displayed. In other cases, link 4650 can be used to provide metadata about the change, such as the time / date the change was made or an identifier of the user who made the change.

[0594] Example 23 - Example Operation

[0595] Figure 47 is a flow chart of a process 4700 for changing a property state of a property of a knowledge graph based on a count of the property in an electronic document provided within a time period and processing the electronic document using the updated property state.

[0596] At 4710, a plurality of attributes are received. At 4720, a plurality of electronic documents within a time period are processed. The processing includes, if the document being processed includes a corresponding attribute from the plurality of attributes, incrementing a corresponding count for the corresponding attribute. At 4730, for at least one attribute from the plurality of attributes, a total count for the at least one attribute over at least a portion of the time period is determined. At 4740, the total count is compared to a threshold. At 4750, it is determined that the total count satisfies the threshold. At 4760, a state of the at least one attribute is changed based at least in part on the determination that the total count satisfies the threshold to provide an updated attribute state. At 4770, at least another electronic document is processed using the updated attribute state.

[0597] Example 24 - Computing System

[0598] Figure 48 A general example of a suitable computing system 4800 is depicted in which the described innovations may be implemented. The computing system 4800 is not intended to suggest any limitation as to the scope of use or functionality of the disclosure, as the innovations may be implemented in various general-purpose or special-purpose computing systems.

[0599] refer to Figure 48 , the computing system 4800 includes one or more processing units 4810, 4815 and memories 4820, 4825. Figure 48 , this basic configuration 4850 is included within the dashed line. Processing units 4810, 4815 execute computer-executable instructions, such as for implementing the techniques described in Examples 1-14. A processing unit may be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example, Figure 48 A central processing unit 4810 and a graphics processing unit or co-processing unit 4815 are shown. Tangible memory 4820, 4825 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both, accessible by the processing unit(s) 4810, 4815. The memory 4820, 4825 stores software 4880 that implements one or more innovations described herein in the form of computer-executable instructions suitable for execution by the processing unit(s) 4810, 4815.

[0600] The computing system 4800 may have additional features. For example, the computing system 4800 includes storage 4840, one or more input devices 4850, one or more output devices 4860, and one or more communication connections 4870. An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects the components of the computing system 4800. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing system 4800 and coordinates the activities of the components of the computing system 4800.

[0601] Tangible storage 4840 may be removable or non-removable and include magnetic disks, magnetic tapes or cartridges, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory manner and that can be accessed within computing system 4800. Storage 4840 stores instructions for software 4880 implementing one or more innovations described herein.

[0602] The input device(s) 4850 may be a touch input device (such as a keyboard, mouse, pen, or trackball), a voice input device, a scanning device, or another device that provides input to the computing system 4800. The output device(s) 4860 may be a display, a printer, a speaker, a CD burner, or another device that provides output from the computing system 4800.

[0603] Communication connection(s) 4870 enable communication with another computing entity via a communication medium. The communication medium carries information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more characteristics set or changed in a manner that encodes the information in the signal. By way of example, and not limitation, the communication medium may utilize electrical, optical, RF, or other carriers.

[0604] Innovations may be described in the general context of computer-executable instructions, such as those included in program modules, executed in a computing system on a target real or virtual processor. Typically, program modules or components include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The functionality of program modules can be combined or split between program modules as desired in various embodiments. Computer-executable instructions for program modules can be executed within a local or distributed computing system.

[0605] The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. In general, a computing system or computing device can be local or distributed and can include any combination of specialized and / or general-purpose hardware and software that implements the functionality described herein.

[0606] In various examples described herein, a module (e.g., component or engine) can be "coded" to perform certain operations or provide certain functionality, indicating that computer-executable instructions for the module can be executed to perform such operations, cause such operations to be performed, or otherwise provide such functionality. Although functionality described with respect to a software component, module, or engine can be performed as discrete software units (e.g., procedures, functions, class methods), it need not be implemented as discrete units. That is, functionality can be incorporated into a larger or more general program, such as one or more lines of code in the larger or general program.

[0607] For convenience, the detailed description uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer and should not be confused with actions performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation.

[0608] Example 25 - Cloud Computing Environment

[0609] Figure 49 An example cloud computing environment 4900 is depicted in which the described techniques may be implemented. Cloud computing environment 4900 includes cloud computing services 4910. Cloud computing services 4910 may include various types of cloud computing resources, such as computer servers, data repositories, network resources, etc. Cloud computing services 4910 may be centrally located (e.g., provided by a data center of an enterprise or organization) or distributed (e.g., provided by various computing resources located in different locations, such as different data centers and / or located in different cities or countries).

[0610] Cloud computing services 4910 are used by various types of computing devices (e.g., client computing devices), such as computing devices 4920, 4922, and 4924. For example, computing devices (e.g., 4920, 4922, and 4924) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smartphones), or other types of computing devices. For example, computing devices (e.g., 4920, 4922, and 4924) can utilize cloud computing services 4910 to perform computing operations (e.g., data processing, data storage, etc.).

[0611] Example 26 - Implementation

[0612] Although the operations of some disclosed methods are described in a particular sequential order for ease of presentation, it should be understood that this description encompasses rearrangement unless specific language set forth below requires a particular order. For example, operations described sequentially may in some cases be rearranged or performed simultaneously. Furthermore, for simplicity, the accompanying drawings may not illustrate the various ways in which the disclosed methods can be used in conjunction with other methods.

[0613] Any of the disclosed methods may be implemented as computer-executable instructions or a computer program product stored on one or more computer-readable storage media (such as tangible, non-transitory computer-readable storage media) and executed on a computing device (e.g., any available computing device, including a smartphone or other mobile device that includes computing hardware). A tangible computer-readable storage medium is any available tangible medium that can be accessed within a computing environment (e.g., one or more optical media disks (such as DVDs or CDs), volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as flash memory or a hard drive)). For example, and with reference to Figure 48 , computer-readable storage media include memories 4820 and 4825 and storage 4840. The term computer-readable storage media does not include signals and carrier waves. In addition, the term computer-readable storage media does not include communication connections (e.g., 4870).

[0614] Any computer-executable instructions for implementing the disclosed technology and any data created and used during the implementation of the disclosed embodiments can be stored on one or more computer-readable storage media. The computer-executable instructions can be, for example, a dedicated software application or a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be executed, for example, on a single local computer (e.g., any suitable commercial computer) or in a network environment using one or more network computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network) or other such network).

[0615] For the sake of clarity, only certain selected aspects of software-based implementations are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented by software written in C, C++, C#, Java, Perl, JavaScript, Python, R, Ruby, ABAP, SQL, XCode, GO, Adobe Flash, or any other suitable programming language, or in some examples, a markup language such as HTML or XML, or a combination of suitable programming languages ​​and markup languages. Similarly, the disclosed technology is not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are known and need not be elaborated in this disclosure.

[0616] Furthermore, any software-based embodiment (including, for example, computer-executable instructions for causing a computer to perform any disclosed method) may be uploaded, downloaded, or remotely accessed via any suitable communication means, including, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communications means.

[0617] The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Rather, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and subcombinations with one another. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments require the presence of any one or more specific advantages or problems to be solved.

[0618] The techniques from any example can be combined with the techniques described in any one or more other examples. In view of the many possible embodiments to which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology. Instead, the scope of the disclosed technology includes that covered by the scope and spirit of the appended claims.< / triple> < / triple> < / triple> < / triple> < / sequenceid>

Claims

1. A computing system comprising: at least one memory; one or more hardware processing units coupled to the at least one memory; and One or more computer-readable storage media storing computer-executable instructions that, when executed, cause the computing system to perform operations comprising: Receive multiple attributes; processing a plurality of electronic documents within a time period, the processing comprising: if the document being processed includes a corresponding attribute from the plurality of attributes, increasing a corresponding count of the corresponding attribute; determining, for at least one attribute of the plurality of attributes, a total count of the at least one attribute over at least a portion of the time period; comparing the total count to a threshold; determining that the total count satisfies the threshold; changing a state of the at least one attribute based at least in part on determining that the total count satisfies the threshold to provide an updated attribute state; and At least one other electronic document is processed using the updated property state.

2. The computing system of claim 1, wherein: Changing the state of the at least one attribute includes changing the state of the at least one attribute from an inactive state to an active state.

3. The computing system of claim 2, wherein the operations further comprise: The at least one attribute is added to the ontology.

4. The computing system of claim 3, wherein: The multiple attributes are defined for the knowledge graph.

5. The computing system of claim 4, wherein the operations further comprise: Receiving a query for the knowledge graph; processing the query using the ontology; as well as A query response is returned, wherein the query response includes an attribute value of the at least one attribute in the knowledge graph, wherein the attribute value is identified using the ontology. The computing system according to claim 2 , wherein: Changing the state of the at least one attribute from an inactive state to an active state comprises: providing a recommendation to the user on a user interface to change the at least one attribute to an active state; and A user input is received through the user interface to change the state of the at least one attribute to an active state.

7. The computing system of claim 2, wherein: The threshold is a first threshold, and when a second threshold requires a higher total count than the first threshold, the state of the at least one attribute is automatically changed from an inactive state to an active state in response to determining that the total count satisfies the second threshold.

8. The computing system of claim 2, wherein: Processing at least another electronic document using the updated property state includes storing a value for the at least one property, wherein values ​​for the at least one property of the plurality of electronic documents are not stored.

9. The computing system of claim 1, wherein: Changing the state of the at least one attribute includes changing the state of the at least one attribute from an active state to an inactive state.

10. The computing system of claim 9, wherein: Processing at least another electronic document includes not storing the value of the at least one property.

11. The computing system of claim 9, wherein: The plurality of attributes are defined for a knowledge graph, and the operations further include: receiving a query for the knowledge graph, the query specifying an attribute with an inactive state in response to the query; processing the query using the ontology; and A query response is returned, wherein the query response includes an attribute value of the at least one attribute in the knowledge graph, wherein the attribute value is identified using the ontology.

12. The computing system of claim 9, wherein: The plurality of attributes are defined for a knowledge graph, and the operations further include: receiving a query for the knowledge graph, wherein the query does not specify an attribute with an inactive state in response to the query; processing the query using the ontology; and A query response is returned, where the query response does not include an attribute value of the at least one attribute in the knowledge graph.

13. The computing system of claim 1, wherein: Attributes having an active state among the plurality of attributes are included in the ontology.

14. The computing system of claim 13, the operations further comprising: A display of at least a portion of the ontology is rendered, wherein the display provides an indication of an extent to which respective attributes of the plurality of attributes of the at least a portion of the ontology are used.

15. The computing system of claim 1, wherein: Changing the state of the at least one attribute includes changing the state from an inactive state to an active state, and the operation further includes: While processing the plurality of electronic documents, when a corresponding electronic document among the plurality of electronic documents includes the at least one attribute, storing an association between the corresponding electronic document and the at least one attribute; and In response to changing the state of the at least one property to an active state, at least a portion of the plurality of electronic documents including the at least one property is reprocessed, the reprocessing including storing respective values ​​of the at least one property in association with corresponding electronic documents of the at least a portion of the plurality of electronic documents.

16. The computing system of claim 1, the operations further comprising: In response to determining that the total count satisfies the threshold, changing the state of the at least one attribute to a recommended state; providing an indication of the status of the recommendation to the user; as well as User input is received to change the state of the at least one attribute to the recommended state, wherein changing the state of the at least one attribute is performed in response to the user input.

17. The computing system of claim 1, wherein: The total count is determined for a plurality of clients, and the operations further include: A state change is suggested to a client of the plurality of clients based at least in part on the total count.

18. The computing system of claim 1, wherein: Changing the state of the at least one attribute is based at least in part on a change in a value of the at least one attribute during the time period compared to a value of the at least one attribute during an earlier time period.

19. A method implemented in a computing system, the computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, the method comprising: Receive multiple attributes; processing a plurality of electronic documents within a time period, the processing comprising: if the document being processed includes a corresponding attribute from the plurality of attributes, increasing a corresponding count of the corresponding attribute; determining, for at least one attribute of the plurality of attributes, a total count of the at least one attribute over at least a portion of the time period; comparing the total count to a threshold; determining that the total count satisfies the threshold; changing a state of the at least one attribute based at least in part on determining that the total count satisfies the threshold to provide an updated attribute state; and At least one other electronic document is processed using the updated property state.

20. One or more non-transitory computer-readable storage media comprising: Computer-executable instructions that, when executed by a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, cause the computing system to receive a plurality of attributes; Computer-executable instructions that, when executed by the computing system, cause the computing system to process a plurality of electronic documents within a time period, the processing comprising: if the document being processed includes a corresponding attribute from the plurality of attributes, incrementing a corresponding count for the corresponding attribute; computer-executable instructions that, when executed by the computing system, cause the computing system to determine, for at least one attribute of the plurality of attributes, a total count of the at least one attribute over at least a portion of the time period; computer-executable instructions that, when executed by the computing system, cause the computing system to compare the total count to a threshold value; computer-executable instructions that, when executed by the computing system, cause the computing system to determine that the total count satisfies the threshold; Computer-executable instructions that, when executed by the computing system, cause the computing system to change the state of the at least one attribute based at least in part on determining that the total count satisfies the threshold to provide an updated attribute state; and Computer-executable instructions, when executed by the computing system, cause the computing system to process at least another electronic document using the updated property state.

Images