Automated ontology maintenance

Abstract

To provide a computing system, method and storage medium for tracking the use of properties over time, and making or proposing property status changes.SOLUTION: In a computing environment 3700 including a client that is in communication with a notification system, the notification system includes one or more user interfaces that a user can access to define, modify, or delete intents. The intents are processed by a request orchestrator that stores intent information in settings storage. An intent reasoner parses an intent and expresses it in a standardized format, such a triple format. A listener service instantiates listeners based on the intents stored in the settings storage. A listener created by the listener service monitors a knowledge graph in a knowledge graph system in various ways.SELECTED DRAWING: Figure 37

Description

[Technical Field]

[0001] This disclosure generally relates to automated processes for maintaining ontologies, including managing status information about properties of the ontologies. [Background technology]

[0002] Ontologies are important to a variety of computer-implemented processes. For example, ontologies can be used in linking data in the Semantic Web, natural language processing, query processing (such as by translating concepts into SQL), and data integration (integrating data that have common semantic concepts). Furthermore, ontologies can be used in artificial intelligence systems involving large-scale language models, the use of which is currently exploding.

[0003] Typically, ontologies are created and updated manually. These tasks can be prohibitively time-consuming, especially when a large number of concepts are to be represented within the ontology. Manually created ontologies can vary depending on the user developing the ontology, including the use of different labels for common semantic concepts and whether users happen to identify particular concepts—i.e., some users may identify ontological concepts that may be overlooked by other users.

[0004] This may be especially true for relatively new concepts. That is, for example, when mainstream electric vehicles were first introduced, the ontology staff may not have realized that concepts related to electric vehicles existed. Even if those concepts existed, they may not have been deemed relevant enough to include in the ontology. Similarly, a concept may become outdated and no longer be particularly relevant to the ontology. For example, cars designed to run on ethanol or biodiesel were at one time more actively pursued for consideration, but that has largely disappeared.

[0005] Thus, current techniques for developing ontologies can be very time-consuming and can contain errors (including not identifying relevant semantic concepts or not identifying ontological concepts as outdated), and therefore there is room for improvement. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent Application No. 17 / 973,322 [Patent Document 2] U.S. Patent No. 11,487,721 [Patent Document 3] U.S. Patent No. 11,263,187 [Non-patent literature]

[0007] [Non-Patent Document 1] Webpage: https: / / www.britishairways.com / en-gb / business-travel / articles / self-service-bag-drop [Non-patent document 2] RDF specification "Concepts and Abstract Syntax", https: / / www.w3.org / TR / rdf11-concepts / [Non-patent document 3] W3C's RDF Schema specification, https: / / www.w3.org / TR / rdf-schema / [Non-patent document 4] "LoRA: Low-Rank Adaptation of Large Language Models," Edward Hu et al., June 17, 2021 [Non-Patent Document 5] neo4j.com / blog / text-to-knowledge-graph-information-extraction-pipeline / Summary of the Invention [Means for solving the problem]

[0008] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0009] Techniques and solutions are provided for improved use of knowledge graphs in document processing. The relevance of properties to a knowledge graph may change over time. A property may emerge, but it may take time before it becomes clear that the property should be used in the knowledge graph. Similarly, a property may be relevant for a period of time but lose its relevance. This disclosure provides techniques for tracking property usage over time and making or proposing changes to property status. These changes may result in the property becoming visible or invisible in the knowledge graph, which in turn may affect how future documents are processed. Furthermore, in some cases, properties can be activated, and documents that were processed when the property was not present or active can be reprocessed to obtain the property's information.

[0010] In one aspect, the present disclosure provides a process for changing the property status of a property in a knowledge graph based on a count of the property in a provided electronic document over a period of time and processing the electronic document using the updated property status.

[0011] A plurality of properties are received. A plurality of electronic documents are processed over a period of time. The processing includes incrementing a respective count for each of the plurality of properties if the document being processed includes the respective property. For at least one property of the plurality of properties, a total count of the at least one property over at least a portion of the period of time is determined. The total count is compared to a threshold. It is determined that the total count meets the threshold. A status of the at least one property is changed based at least in part on a determination that the total count meets the threshold to provide an updated property status. At least another electronic document is processed using the updated property status.

[0012] The present disclosure also includes computing systems and tangible, non-transitory computer-readable storage media configured to perform or containing instructions for performing the above-described methods. As described herein, various other features and advantages may be incorporated into the technology as desired. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 illustrates an exemplary directed graph. [Figure 2] FIG. 1 is a diagram illustrating the domain and range of properties in a schema of a directed graph. [Figure 3] FIG. 1 illustrates an exemplary SPARQL query and the results of the query. [Figure 4] FIG. 1 illustrates an exemplary system architecture for a method according to some embodiments. [Figure 5] FIG. 1 illustrates an algorithm for computing strings from a directed graph for input into a language model. [Figure 6] FIG. 1 illustrates a sorting algorithm for sorting statements in a directed graph. [Figure 7]FIG. 1 illustrates a directed graph that may be provided as input to a sorting algorithm. [Figure 8] FIG. 1 illustrates the relationship between a condition and one or more patterns. [Figure 9] 1 is a flow diagram of a process for creating a domain data model, including mapping elements of the domain data model to elements of a core data model. [Figure 10] FIG. 6 shows the algorithm of FIG. 5 extended to handle query patterns. [Figure 11] FIG. 1 illustrates the relationships between and components of a knowledge graph, an ontology, and a meta-ontology. [Figure 12] 1 is a flow diagram of a method for extracting properties from a set of source documents. [Figure 13] FIG. 1 is a diagram of a computing environment in which the disclosed techniques for property extraction may be implemented. [Figure 14A] FIG. 2 illustrates an exemplary source document having information from which properties can be extracted. [Figure 14B] FIG. 2 illustrates an exemplary source document having information from which properties can be extracted. [Figure 14C] FIG. 2 illustrates an exemplary source document having information from which properties can be extracted. [Figure 15] FIG. 1 illustrates an exemplary user interface for selecting source files for analysis and for identifying instances associated with such source files. [Figure 16] FIG. 2 illustrates an exemplary user interface that allows a user to define or edit the set of source files for a particular instance. [Figure 17] FIG. 1 illustrates an exemplary user interface that allows a user to view and edit an instance graph created from one or more source files of an instance, including viewing properties or property values ​​extracted from the source documents. [Figure 18]FIG. 10 provides exemplary pseudocode for aligning instance graphs that use a common vocabulary and counting the occurrences of a particular property in a set of instance graphs. [Figure 19A] FIG. 10 illustrates an exemplary user interface for viewing identified properties and editing the list of properties to select properties to be included in the ontology. [Figure 19B] 19B illustrates the user interface of FIG. 19A after selection of a control for editing a property, such as a label applied to the property. [Figure 20] FIG. 1 illustrates how properties associated with an instance can change over time. [Figure 21] FIG. 1 illustrates an exemplary computing environment that can be used to monitor property usage and update property status information. [Figure 22] FIG. 2 illustrates an exemplary data object that may be used to store associations between properties and documents that have the properties and track property usage. [Figure 23] FIG. 1 illustrates exemplary documents within an interval and information that can be used to determine membership within the interval. [Figure 24] FIG. 10 illustrates exemplary pseudocode for calculating property usage over an interval and assigning status to properties based on such property usage. [Figure 25] FIG. 1 illustrates an exemplary user interface that can be used to add instances to a knowledge graph, view property information associated with such instances, and optionally add properties associated with the instances to an ontology associated with the instances. [Figure 26] 10A-10C are diagrams of exemplary user interfaces that provide information about properties that may be obsolete. [Figure 27]10A-10C are diagrams of example user interfaces that provide summary information about a property over an interval and provide detailed information about a selected property, including observed values ​​for the selected property over the interval. [Figure 28] FIG. 10 is a diagram of an exemplary user interface that provides detailed information for a selected property, including usage information for the property over the current interval, the property values ​​observed during the interval, intervals associated with different statuses of the property, and identifiers of documents associated with the property. [Figure 29] 1A-1C are diagrams of exemplary user interfaces summarizing property status and usage over various intervals and graphs that may be displayed using such information. [Figure 30] FIG. 10 illustrates an example property graph showing how property usage information may be presented with respect to intervals. [Figure 31] FIG. 10 illustrates an example property graph showing how a graph over multiple intervals can provide insight into property trends. [Figure 32] FIG. 1 illustrates an exemplary electronic newsletter that may include updates on information relevant to a user. [Figure 33] FIG. 1 is a diagram of an exemplary email that may contain updates on information relevant to a user. [Figure 34] FIG. 1 illustrates an exemplary data model that may be used in the disclosed technology. [Figure 35] 1 is a flow diagram of an example process for defining an intent and analyzing a knowledge graph for updates relevant to the intent. [Figure 36] FIG. 36 is a diagram of a simplified computing environment in which the process of FIG. 35 may be implemented. [Figure 37] FIG. 36 is a more detailed diagram of a computing environment in which the process of FIG. 35 may be implemented. [Figure 38]FIG. 1 illustrates how nodes in a knowledge graph can have different weights or relevance to a user intent, including based on knowledge graph elements that are inferred to be relevant to the user intent. [Figure 39] FIG. 10 illustrates how weights assigned to nodes in a knowledge graph can be propagated to associated nodes in the process of identifying nodes for inclusion in an inferred intent. [Figure 40] FIG. 1 illustrates an exemplary user interface that allows a user to define an intent by selecting elements of a knowledge graph. [Figure 41] 1 is a diagram of an exemplary user interface that allows a user to define an intent by selecting text in an electronic document. [Figure 42] FIG. 1 illustrates an example log from which knowledge graph updates can be extracted. [Figure 43] FIG. 1 is a diagram of an exemplary standardized format in which knowledge graph changes may be expressed, and exemplary changes expressed in such a format. [Figure 44] FIG. 10 illustrates components of an example prompt template for generating a message summarizing changes to a knowledge graph. [Figure 45] FIG. 45 is a diagram of an example prompt using the example prompt template of FIG. 44. [Figure 46] FIG. 46 is a diagram of an exemplary response generated by a natural language generator in response to the prompt of FIG. 45. [Figure 47] 1 is a flow diagram of an example process for changing property statuses of properties in a knowledge graph based on counts of properties in provided electronic documents over a period of time and processing the electronic documents using the updated property statuses. [Figure 48] FIG. 1 illustrates an exemplary computing system in which some described embodiments may be implemented. [Figure 49]FIG. 1 is a diagram of an exemplary cloud computing environment that may be used in conjunction with the technologies described herein. DETAILED DESCRIPTION OF THE INVENTION

[0014] Example 1--Overview Ontologies are important to a variety of computer-implemented processes. For example, ontologies can be used in linking data in the Semantic Web, natural language processing, query processing (such as by translating concepts into SQL), and data integration (integrating data that have common semantic concepts). Furthermore, ontologies can be used in artificial intelligence systems involving large-scale language models, the use of which is currently exploding.

[0015] Typically, ontologies are created and updated manually. These tasks can be prohibitively time-consuming, especially when a large number of concepts are to be represented within the ontology. Manually created ontologies can vary depending on the user developing the ontology, including the use of different labels for common semantic concepts and whether users happen to identify particular concepts—i.e., some users may identify ontological concepts that may be overlooked by other users.

[0016] This may be especially true for relatively new concepts. That is, for example, when mainstream electric vehicles were first introduced, the ontology staff may not have realized that concepts related to electric vehicles existed. Even if those concepts existed, they may not have been deemed relevant enough to include in the ontology. Similarly, a concept may become outdated and no longer be particularly relevant to the ontology. For example, cars designed to run on ethanol or biodiesel were at one time more actively pursued for consideration, but that has largely disappeared.

[0017] Thus, current techniques for developing ontologies can be very time-consuming and can contain errors (including not identifying relevant semantic concepts or not identifying ontological concepts as outdated), and therefore there is room for improvement.

[0018] This disclosure provides techniques for maintaining ontologies, including adding or removing properties from an ontology. In one aspect, new properties are recognized when instances are added to the ontology. Counts of the use of these new properties are tracked over time. In some cases, a user can be presented with property count information and can determine whether the property should be added to the ontology. Optionally, a user may be provided with automated suggestions for whether the property should be added. In other cases, a property may be automatically added to an ontology when certain conditions are met.

[0019] In a similar manner, the use of existing properties within an ontology can be tracked over time. By analyzing a particular time interval, such as a more recent time interval, it can be determined whether fewer instances contain the existing ontological property. A user can be provided with data regarding the property's usage and, optionally, a suggestion on whether to consider marking the property as obsolete or deleting it from the ontology. Alternatively, a property can have its status automatically changed to obsolete if threshold criteria are met.

[0020] The ability to track property usage over time also facilitates tracking the evolution of an ontology. Tracking the evolution of an ontology may include tracking how usage of a particular property, including related properties, changes over time. This type of analysis can assist users in identifying trends, including identifying whether new trends are likely to be short-lived or likely to persist over the long term.

[0021] Updated ontologies increase the usefulness of knowledge graphs and the instances they represent. By incorporating the latest domain knowledge and accommodating new properties and concepts, updated ontologies help ensure that knowledge graphs remain relevant and comprehensive. Furthermore, they facilitate tracking of instance properties that may not have been previously listed, leading to more accurate and complete representations of data. This adaptability and responsiveness to evolving knowledge to create updated ontologies helps maintain the semantic clarity and usability of knowledge representations.

[0022] In addition to tracking data for new instances, this disclosure provides techniques for reprocessing instances received prior to an ontology update. For example, if properties related to electric vehicles are added to an ontology, previous instances of the vehicle can be processed so that data can be added regarding the instance's properties, where appropriate.

[0023] Example 2 describes a knowledge graph, including a representation of the knowledge graph and verbalizing knowledge graph triples. Examples 3-7 describe the relationship between a knowledge graph, an ontology, and a metaontology, and techniques for identifying new properties in an ontology. Examples 8-13 describe techniques for changing the status of properties in an ontology, including between active and obsolete status, or changing a property that has been observed but not used frequently enough to previously be in active status to active status. Examples 14-23 describe techniques that can be used to register a user's intent, where the intent indicates the portion of the knowledge graph that the user is interested in, and to update the user when relevant changes are detected.

[0024] Example 2 – An example verbalization of a Knowledge Graph triple A company may have a variety of different products, services, and teams. A company may also have a comprehensive knowledge graph that stores knowledge about the skills, processes, experience, capabilities, and insights relied upon in the company's daily operations. The content of the knowledge graph may also include company-specific acronyms, company divisions, and product specifications. The knowledge may enable a company to respond quickly, professionally, and flexibly to business situations. Knowledge graphs may be expensive and labor-intensive to build and maintain. Knowledge graphs (i.e., the Semantic Web and / or the Web of Linked Data) may be specified using the Resource Description Framework (RDF).

[0025] In some cases, users want to ask questions or give tasks to language models (e.g., large-scale language models based on generative pre-trained transformers, such as ChatGPT). However, language models are typically trained unsupervised on unlabeled human-readable text. Therefore, language models may not be able to directly process knowledge graphs or use knowledge graphs as input (e.g., for training).

[0026] Thus, for example, it may be desirable to maximize the usability of a knowledge graph as a basis for artificial intelligence applications, and more particularly, to use the knowledge graph to train or otherwise improve language models, which, once trained, may be used to answer questions or perform tasks based on the knowledge stored in the knowledge graph.

[0027] Additionally, it may be desirable to extract human-readable text from a knowledge graph (e.g., for use in explaining answers provided by software (e.g., a process advisor) that relies on the knowledge graph).

[0028] According to an aspect, a computer-implemented method is provided for providing data from a directed graph to a language model. The method includes defining a plurality of conditions and a plurality of patterns, each of the conditions having at least one corresponding pattern. The method further includes receiving a subset of the directed graph, the subset of the directed graph including a plurality of statements. Each of the statements includes a subject, an object, and a predicate that associates the subject with the object. For each statement in the subset of the directed graph, when one of the conditions matches the respective statement and a pattern corresponding to the condition can be applied to the respective statement, computing a string for the respective statement using the pattern. Providing the computed string as input to the language model.

[0029] Providing data from the directed graph to the language model may include extracting or reading data from the directed graph and supplying or sending the data as input to the language model. The statements may be referred to as triples or triple statements having a subject component, a predicate component, and an object component. The conditions may correspond to the respective patterns in the sense that when the conditions are determined to be true for the respective statements, it may be determined whether the respective patterns can be applied to the respective statements.

[0030] The phrase "each of the conditions has at least one corresponding pattern" may be understood to mean that each of the conditions of the plurality of conditions has at least one corresponding pattern of the plurality of patterns. Thus, each of the conditions of the plurality of conditions may be assigned at least one pattern of the plurality of patterns. In other words, when a condition has at least one corresponding pattern, at least one corresponding pattern is assigned to the condition. Furthermore, each of the conditions of the plurality of conditions may be assigned more than one pattern of the plurality of patterns.

[0031] The clause "when one of the conditions matches the respective statement and a pattern corresponding to the condition may be applied to the respective statement" may include determining whether at least one pattern corresponding to the condition may be applied to the respective statement. Thus, determining whether at least one pattern corresponding to the condition may include determining whether the pattern corresponding to the condition matches the respective statement (i.e., testing whether the respective statement has a property or element defined by the pattern).

[0032] For example, determining whether a pattern can be applied to a statement containing a subject may be performed as follows:

[0033] The multiple conditions are: first condition, <?s><?p><?o> . BIND(sap:BusinessActivity AS<?s> ) BIND(rdf:type AS<?p> ) May include:

[0034] The first pattern may correspond to a first condition. The<s.rdfs:label> is a business activity.

[0035] Because the first pattern contains a reference to the label of the subject s, if a statement containing the subject satisfies the first condition (i.e., being a BusinessActivity), but the subject of the statement does not have the label required by the first pattern, then the first pattern may not apply to the statement (i.e., it does not match).

[0036] Using a pattern to compute a string from each statement may involve directly outputting the text of the pattern, or matching a pattern operator against components of the statement.

[0037] Furthermore, when computing the string, not only each statement that matches the condition but also one or more additional statements of the subset of the directed graph may be accessed by the pattern corresponding to the condition. In other words, the pattern may take into account statements that do not match the condition. Thus, the condition may trigger the execution of the pattern for each statement, which in turn may trigger the processing of at least a portion of the subset of the directed graph (or the entire subset of the directed graph) using the pattern.

[0038] The subject, object, and predicate are sometimes called the components of their respective statements. The subject and / or predicate may be an RDF resource (e.g., the subject and / or predicate may have RDF resource properties or may be a type or label). The object may be a literal (e.g., an RDF literal) with a defined datatype such as string, integer, Boolean, or double (defined in the Extensible Markup Language (XML) Schema Definition Language (XSD)). Regarding RDF, see the RDF specification "Concepts and Abstract Syntax", https: / / www.w3.org / TR / rdf11-concepts / .

[0039] The computed strings may be provided directly to the language model (e.g., for small strings, such as less than 1 GB), or may be serialized to a text file before being provided to the language model (e.g., for terabytes of data computed from a comprehensive directed graph). In essence, the method accepts a subset of the directed graph as input and generates grammatically correct sentences as computed strings. The method is applied iteratively to the input statements (i.e., triples) in the subset of the directed graph, using condition filters to determine which patterns may be applicable to each of the statements. When a condition matches a statement and the statement has elements required by the pattern corresponding to the condition (e.g., assigned to it), the pattern is applied to the condition. The result of the iteration may be a set of strings, such that a string is computed for each statement of the input. The syntax and interpretation of patterns are described in more detail below.

[0040] Digraph statements can be close to human language.

[0041] Thus, providing the computed strings as input to a language model may maximize the usability of the subset of the directed graph, for example, by using the subset of the directed graph as the basis for an artificial intelligence application. Once the language model processes the computed strings, the language model may be used to answer questions or perform tasks based on the knowledge stored in the subset of the directed graph. Thus, the time, effort, and money invested in constructing the directed graph may be leveraged in additional ways (e.g., to answer questions or by using the language model).

[0042] Additionally or alternatively, it may be desirable to extract human-readable text from a subset of the directed graph (e.g., for use in explaining answers provided by software (e.g., a process advisor) that relies on the subset of the directed graph).

[0043] In some cases, each condition contains at least three condition variables (variables that appear in a condition are sometimes called condition variables). Each of the condition variables may correspond to (e.g., store) a different component of the statement. For example, the first condition variable matches the subject, the second condition variable matches the predicate, and the third condition variable matches the object. At least one of the condition variables may be bound to at least one value (e.g., an RDF property). In other words, at least one of the condition variables may specify at least one value (e.g., an RDF property) that a component of the statement must have. Each of the condition variables may specify an instance of a class (e.g., an RDF class) or a literal (e.g., an RDF literal). An instance of a class is sometimes called an instance. An instance may relate to a specific concept and have a definite article, while a class may relate to a generic concept and have an indefinite article.

[0044] Each condition may be applied to a statement and may evaluate to TRUE or FALSE. In other words, a condition may return a Boolean value. For example, if a condition evaluates to TRUE, the condition matches the respective statement, and it is determined whether at least one pattern corresponding to the condition can be applied to the respective statement. Below are numbered examples of conditions that may be present within multiple conditions: 1.<?s><?p><?o> . 2.<?s><?p><?o> . BIND(rdfs:label AS<?p> ) 3.<?s><?p><?o> . <?s> rdf:type sap:BusinessActivity. BIND(rdfs:label AS<?p> ) 4.<?s><?p><?o> . <?s> rdf:type sap:BusinessActivity. <?o> sap:requires<?r> . BIND(rdfs:label AS<?p> )

[0045] In the first condition,<?s> " is a variable corresponding to the subject of the directed graph, and "<?p> " is a variable corresponding to the predicate of the directed graph, and "<?o> " is a variable that corresponds to the object of the directed graph. Thus, the first condition specifies that the statement must contain a subject, a predicate, and an object (i.e., for the first condition to evaluate to TRUE). The second condition requires that the statement contains a subject, a predicate, and an object, and that the predicate is an rdfs:label. The third condition requires that the statement contains a subject, a predicate, and an object, and that the subject has the property sap:BusinessActivity (more precisely, is of type sap:BusinessActivity) and the predicate is an rdfs:label. The fourth condition requires that the statement contains a subject, a predicate, and an object, and that the subject has the property sap:BusinessActivity (more precisely, is of type sap:BusinessActivity) and the object is the object "<?r> " and "sap:requires" (i.e., a subject-object relationship), and requires that the predicate be rdfs:label.<?r> " is a variable bound to the requirement.

[0046] A condition of the plurality of conditions may function to prevent the pattern from being used to compute semantically incorrect strings. In other words, the condition may be used to ensure that the computed string is semantically and / or grammatically correct. Thus, by assigning a pattern to a condition, the cases in which the pattern is applied can be restricted, thereby ensuring or facilitating the computation of semantically correct strings (i.e., sentences). Without the condition, the pattern could be applied to compute example unrealistic strings such as "The Harry Potter Book is a business activity." or "The Star Wars Movie is a business activity." However, the example unrealistic strings would be semantically incorrect, and therefore the example unrealistic strings would be useless as input to a language model, prolong the training of the language model, or even cause the language model to generate incorrect outputs.

[0047] In some cases, at least one of the conditions has multiple corresponding patterns. Thus, computing a string from each statement using a pattern may include computing multiple strings from each statement, with each pattern corresponding to a condition that may apply to (i.e., a condition that matches) each statement.

[0048] Alternatively, computing a string from each statement using the patterns may include determining a random order of patterns corresponding to the conditions, and computing a string from each statement using only the first pattern in the random order of patterns that can be applied to each statement. For example, patterns 1 through 4 may be ordered as 2, 4, 3, 1, and pattern 2 may be applied to each statement, and therefore pattern 2 is applied to each statement.

[0049] For example, multiple conditions can be a sequenceID condition, <?I1> <sequenceid><?L1>. May include:

[0050] The sequenceID condition may correspond to the following pattern (A). (A) The sequence identifier of<?I1.rdf:type.rdfs:label><?I1.rdfs:label> is<?L1> .

[0051] Continuing with the example, a subset of the directed graph might include the following statements: 1. Emissions Management isA Business Capability. 2. BusinessCapability rdfs:label "Business Capability". 3. EmissionsManagement SequenceID "5".

[0052] Since statement (3) contains "SequenceID" and statements (1) and (2) do not contain "SequenceID", the sequenceID condition matches only statement (3).

[0053] Pattern (A) can be applied to statements (1), (2), and (3) to compute the following sentence: The sequence identifier of Business Capability is 5.

[0054] Statements (1) and (2) provide context information for statement (3), so all three statements are needed to compute the string above.

[0055] Thus, as shown above, when computing a string, not only each statement that matches a condition, but also one or more further statements of the subset of the directed graph may be accessed by the pattern corresponding to the condition. In other words, the pattern may take into account statements that do not match the condition. Thus, the condition triggers the execution of the pattern for each statement.

[0056] Furthermore, statements in a subset of the directed graph may be checked iteratively. Thus, in the above example, statements (1) and (2) do not cause the pattern to trigger, but statement (3) does.

[0057] In this example, determining whether pattern (A) can be applied to statement (3) depends on the context information of pattern (A), i.e., variable<?I1> Type of label variable<?I1> Label may include determining:

[0058] Both contextual information is not available in statement (3) itself, but instead is available elsewhere in a subset of the directed graph.

[0059] Considering statement (3), there is contextual information in the subset of the directed graph, and pattern (A) can be executed for statement (3).

[0060] Whenever forward dot notation is used, additional information not present in the triple itself is included.

[0061] In some cases, each pattern includes one or more of the following: - At least one variable that specifies (e.g., is bound to) a class, an instance of a class, a literal, or a predicate - Text, including one or more articles (e.g., grammatical articles that are definite or indefinite) - At least one property applies to the variable

[0062] Each pattern may further include a linguistic filter. The literals may specify numbers or text, and may conform to an RDF Schema class for literal values.

[0063] Advantageously, a pattern may allow for a combination of static text and variables that specify the structure of a subset of a directed graph (e.g., the subject, object, and predicate of one of the statements), possibly supplemented with information resulting from materializing the subset of a directed graph.

[0064] Additionally or alternatively, the pattern may include at least one specific pattern (i.e., a custom pattern) and multiple default patterns. When a condition corresponding to the specific pattern matches the respective statement and the specific pattern can be applied to the respective statement, calculating a string from the respective statement using the pattern may include using the specific pattern. When a condition corresponding to the specific pattern does not match the respective statement, the method may further include determining whether a condition corresponding to one of the default patterns matches the respective statement. When a condition corresponding to one of the default patterns matches the respective statement, calculating a string from the respective statement using one of the default patterns. Defining multiple conditions and multiple patterns may further include defining at least three conditions and at least three patterns, where at least one of the three patterns is the specific pattern and at least one of the three conditions corresponds to the specific pattern.

[0065] The following are exemplary patterns that may be included in multiple patterns: Text<?I1.rdf:type.rdfs:label> Text

[0066] variable<?I1> may be bound to sap:PrintReceipt. Patterns such as the example pattern above may use front-dot notation (also called dot notation) to refer to fields, components, or sub-properties of a property. Because front-dot notation allows for more compact representation, this may provide an advantage for patterns over traditional SPARQL, which does not support front-dot notation.

[0067] Thus, the above example pattern may apply to the following three statements (e.g., a subset of a directed graph may contain the following three statements): sap:PrintReceipt rdf:type sap:Task. sap:Task rdfs:label "Task"@en. sap:Task rdfs:label "Process Task"@en.

[0068] In some cases, after a pattern is applied to each statement, the pattern is not applied to further statements that match the pattern (i.e., further statements in the subset of the directed graph that match the pattern may be skipped). Matching only one statement and skipping further statements may be an option that can be configured. For example, as discussed below, a postfix operator may cause a Cartesian product to be computed.

[0069] For example, after the pattern has been applied to each statement containing the rdfs:label of an object, further statements containing the rdfs:label of the object may be skipped (i.e., the pattern is not applied to further statements). Thus, computing a string from the above statements using the example pattern yields the following: "Text Task Text".

[0070] At least one of the patterns may include a filter condition and / or a postfix operator. The filter condition may specify a language. The postfix operator may cause a Cartesian product to be performed. As another example, a subset of a directed graph may include the following four statements: sap:PrintReceipt rdf:type sap:Task. sap:Task rdfs:label "Task"@en. sap:Task rdfs:label "Process Task"@en. sap:Task rdfs:label "Schritt"@de.

[0071] The following additional exemplary patterns may be included in multiple patterns and may apply to the four statements above: Text<?I1.rdf:type.rdfs:label(lang = 'en')*> Text

[0072] The above further exemplary pattern includes a filter condition to specify a language and an asterisk postfix operator "*" that results in a Cartesian product. A Cartesian product may result in all possible combinations of the preceding elements. Thus, since the further exemplary pattern is targeted at English labels, and there are two English labels in the above four sentences, the following string is computed by applying the further exemplary pattern that includes a Cartesian product: Text Task Text Text Process Task Text

[0073] Thus, the above statement containing "Schritt"@de would not be processed because the statement does not satisfy the filter condition of the further example pattern (i.e., the statement is not in English). Without the asterisk postfix operator in the above further example pattern, only the first string "Text Task Text" would be calculated.

[0074] The above exemplary patterns and further exemplary patterns may be specific patterns (ie, patterns applicable to one directed graph or a group of directed graphs).

[0075] A string may be computed from each statement using only the default pattern, but the use of a specific pattern may result in a computed string that more accurately and precisely describes the contents of a subset of the directed graph.

[0076] In addition to the asterisk, other postfix operators (i.e., operators placed at the end of a pattern, also called postfix operators) may also be used. For example, an additional postfix operator may limit the output of a Cartesian product to a specified number of combinations (e.g., about 10 combinations).

[0077] As another example, the following required pattern might apply to the four statements above: The<?I1.rdf:type.rdfs:label(lang = 'en')*><?I1.rdfs:label> requires a<?I2.rdfs:label> .

[0078] In this example,<?I1> may be bound to sap:PrintReceipt,<?I2> may be bound to a sap:Printer. Thus, by applying the required pattern to the four statements above, the following string may be computed: - The Process Task Print Receipt requires a Printer. - Task Print Receipt requires a Printer.

[0079] If the requirement pattern does not have an asterisk postfix operator, only the first string (i.e., "The Process Task Print Receipt requires a Printer.") is calculated.

[0080] In some cases, each of the computed strings may be a grammatically correct sentence, and conditions and / or patterns may ensure that the computed strings are grammatically correct sentences.

[0081] Additionally or alternatively, the at least one specific pattern may include a plurality of specific patterns. Each of the specific patterns may be applicable to a group of directed graphs defined according to the resource description framework or a group of knowledge graphs defined according to the resource description framework. Each of the default patterns may be applicable to any directed graph defined according to the resource description framework or any knowledge graph defined according to the resource description framework.

[0082] Thus, each specific pattern may be defined with respect to a single on-premises network and corresponding directed graph, or a group of on-premises networks and corresponding groups of directed graphs, while a default pattern may be applicable to any directed graph.

[0083] The pattern may include one or more of the following five patterns: - variable<I1, p, I2> Patterns applicable to instance-instance statements, including - variable<I1, p, C1> Patterns applicable to instance-class statements, including - variable<C1, p, C2> Patterns applicable to class-class statements, including - variable<I1, p, L1> Patterns applicable to instance-literal statements, including - variable<C1, p, L1> Patterns applicable to class-literal statements, including

[0084] The above five patterns are<I1, p, I2> ,<I1, p, C1> ,<C1, p, C2> ,<I1, p, L1> ,<C1, p, L1> The condition may be made applicable to instance-instance statements, instance-class statements, class-class statements, instance-literal statements, and class-literal statements via corresponding conditions including:

[0085] The five patterns mentioned above may be default patterns in the sense that they are applicable to any directed graph, or more specifically, any knowledge graph.

[0086] For the five patterns above, "I1" and "I2" are variables that point to instances (i.e., instances of a class), "C1" and "C2" are variables that point to classes, "L1" is a variable that points to a literal, and "p" is a variable that points to a predicate. The first of the five patterns, applicable to instance-instance statements, may be implemented as follows: The<?I1.rdf:type.rdfs:label><?I1.rdfs:label><?p.rdfs:label> the<?I2.rdf:type.rdfs:label><?I2.rdfs:label> .

[0087] The second of the five patterns, applicable to instance-class statements, may be implemented as follows: The<?I1.rdf:type.rdfs:label><?I1.rdfs:label><?p.rdfs:label> a<?C1.rdfs:label> .

[0088] The third of the five patterns, applicable to class-class statements, may be implemented as follows: A<?C1.rdfs:label><?p.rdfs:label> a<?C2.rdfs:label> .

[0089] The fourth of the five patterns, applicable to instance-literal statements, may be implemented as follows: The<?I1.rdfs:label><?p.rdfs:label><?L1> .

[0090] The fifth of the five patterns, applicable to class-literal statements, may be implemented as follows: A<?C1.rdfs:label><?p><?L1> .

[0091] A user or administrator may define additional default patterns or modify the exemplary default patterns given above.

[0092] Additionally or alternatively, the multiple patterns may include at least one text pattern and at least one question pattern. The text pattern and / or the question pattern may be specific patterns. The text pattern and / or the question pattern may be default patterns. Each condition may correspond to at least one text pattern, and at least one condition may correspond to at least one question pattern. For example, the conditions may be defined such that each condition must correspond to at least one text pattern, and each condition may correspond to at least one question pattern. A configuration option may be set to apply a question pattern in addition to or instead of a text pattern. When one of the conditions matches a respective statement, the configuration option is set to apply a question pattern, and the question pattern corresponding to the condition may be applied to the respective statement, the method may include computing a string from the respective statement using the question pattern and / or computing a further string from the respective statement using the question pattern in addition to the string computed from the respective statement using the text pattern.

[0093] For example, a subset of a directed graph might contain the following seven statements: sap:PrintReceipt sap:requires sap:Printer. sap:PrintReceipt rdf:type sap:Task. sap:PrintReceipt rdfs:label "Print Receipt". sap:Printer rdfs:label "Printer". sap:Task rdfs:label "Task"@en. sap:Task rdfs:label "Process Task"@en. sap:Task rdfs:label "Schritt"@de.

[0094] Continuing with the example, the multiple patterns may include the following patterns (eg, text patterns): The<I1.rdf:type.label(lang = 'en')*><I1.rdfs:label> requires a<I2.rdfs:label> .

[0095] Additionally, the multiple patterns may include the following question pattern that precedes the immediately preceding text pattern: What is required by<I1.rdf:type.label(lang = 'en')*><I1.rdfs:label> ?

[0096] The "*" (asterisk) operator (i.e., postfix operator) in a query pattern causes a Cartesian product to be computed. In the above text pattern and question pattern,<?I1> is bound to (i.e. holds a value for) sap:PrintReceipt,<?p> is bound to sap:requires,<?I2> is bound to a sap:Printer. Therefore, a configuration option may be set to apply both the question pattern and the text pattern. Thus, by applying both the question pattern and the text pattern to the above seven statements, the following question / answer string is computed: Q: What is required by Task Print Receipt? A: The Task Print Receipt requires a Printer. Q: What is required by Task Print Receipt? A: The Process Task Print Receipt requires a Printer. Q: What is required by Process Task Print Receipt? A: The Task Print Receipt requires a Printer. Q: What is required by Process Task Print Receipt? A: The Process Task Print Receipt requires a Printer.

[0097] The above strings are prefixed with "Q:" and "A:" for clarity. Thus, question patterns may be used to simulate question-answer interactions. As discussed in the above example regarding Cartesian products, without the asterisk operator in this example, the string would be computed only from the first statement to which the question pattern and text pattern can be applied (i.e., the first statement that matches the question pattern and text pattern).

[0098] In some cases, the subset of the directed graph may be the entire directed graph. Alternatively, the subset of the directed graph may be a proper subset of the entire directed graph and may be determined by a query of the directed graph. The query may be a SPARQL Protocol and RDF Query Language (SPARQL) query.

[0099] In some cases, a subset of a directed graph includes multiple nodes connected by edges. The nodes may represent real-world entities, and the edges may represent relationships between entities or between entities and types of entities (i.e., classes). Thus, predicates may be distinguished depending on whether they connect two entities or between an entity and a type of entity. An entity may also be called a resource. For each statement, the subject may correspond to a node, the object may correspond to a (different) node, and the edge corresponding to the predicate may connect the subject node to the object node.

[0100] Nodes may have corresponding classes, such that each node has a corresponding class. The (corresponding) classes may be part of (or configured in) a schema (i.e., a data schema or ontology). The schema may be defined in RDF or Web Ontology Language.

[0101] Below is an example class. :State a rdfs:Class. :EuropeanState a rdfs:Class. :City a rdfs:Class.

[0102] Thus, ":State" is a resource that is a class, more precisely an RDF class. The class ":EuropeanState" is another resource that is a class, more precisely a subclass of ":State". Thus, a hierarchy of classes is possible. Furthermore, multiple inheritance is possible.

[0103] Additionally or alternatively, a directed graph may be labeled and multi-relational. Thus, both nodes and edges may have labels, and edges may have directions. The object of a statement may be a label in the directed graph. A directed graph may be multi-relational in the sense that edges have different labels. Nodes in a directed graph may be subjects or objects, and edges may be predicates.

[0104] Additionally or alternatively, a schema may include properties. Each of the properties may apply to at least one of the classes in the schema. At least one of the properties may have a domain and / or a range. Each of the properties may be used by (or apply to) at least one statement. A domain (e.g., rdfs:domain) may specify a class to which a subject belongs, and a range (e.g., rdfs:range) may specify a class to which an object belongs. More specifically, a domain may specify a class to which a subject of a statement belongs, and a range may specify a class to which an object of a statement belongs. For information on RDF Schema, see the W3C RDF Schema specification, https: / / www.w3.org / TR / rdf-schema / .

[0105] Below are some example properties: rdf:type a rdf:Property dbo:foundationPlace a rdf:Property. :EuropeanState rdfs:subClassOf :State. :locatedIn a rdf:Property. :capitalOf a rdf:Property. :capitalOf rdfs:subPropertyOf :locatedIn.

[0106] Thus, ":locatedIn" and ":capitalOf" are properties. Furthermore, ":capitalOf" is a subproperty of ":locatedIn". Thus, properties can also form hierarchies. The property ":EuropeanState rdfs:subClassOf :State" indicates that ":EuropeanState" is a subclass in a class hierarchy that includes the class ":State" and the subclass ":EuropeanState".

[0107] Thus, a schema may provide a vocabulary for a directed graph (e.g., a knowledge graph). A directed graph may have predefined property prefixes that can indicate whether a node (i.e., subject or object) is an instance of a class or is a class (e.g., if a node has the prefix "dbo" to represent a DBpedia ontology, the node may be a class; if the node has the prefix "dbr" to represent a DBpedia resource, the node may be an instance). In certain cases, a directed graph may use a URI design to distinguish between instances and classes. A directed graph may include a statement that explicitly indicates that a particular node is a class. In certain cases, whether a particular node represents an instance or a class may depend on the underlying model. For example, whether a node is a class (and is included in the schema of the directed graph) or an instance (and therefore is not included in the schema of the directed graph) may be determined by checking the rdf:type property. If the type is owl:Class then the node is a class and is contained in the schema, otherwise the node is an instance (ie an instance of a class) and is not contained in the schema.

[0108] In some cases, the total number of patterns is equal to or greater than the total number of properties.

[0109] Furthermore, for an ontology (i.e., schema) O with a set of classes C and a set of properties P, only |P| conditions are needed to compute a string from the complete directed graph without syntax errors.

[0110] Additionally or alternatively, data from a directed graph covers multiple topical domains. Each statement may be identified by at least one Uniform Resource Identifier (URI). At least one of the nodes and edges may be identified by a URI or an internationalized resource identifier (IRI). More specifically, the nodes and edges may each be identified by a URI or an IRI. In some cases, one or more of the subject, object, and predicate may be a URI. Some nodes (e.g., nodes corresponding to objects) may be identified by literals rather than URIs. The directed graph may be represented using RDF. The directed graph may be a knowledge base and / or a knowledge graph. The statements may be referred to as facts or factual statements. Thus, the directed graph may have a structure similar to known knowledge graphs such as DBPedia, Wikidata, BabelNet, DBkWik, Freebase, and DBnaray.

[0111] Compared to a relational database, a knowledge graph has a more flexible data structure because the type of data provided by the knowledge graph can be varied. For example, the properties associated with different instances can be different even if these instances share the same class (e.g., "SAP_SE" and "BASF_SE" share the same class "Company" but can have different property data available). On the other hand, a relational database can be represented in the format of a knowledge graph (i.e., a knowledge graph can be a higher-level abstraction of a relational database).

[0112] In particular examples, the nodes of a directed graph (e.g., a knowledge graph) may be organized in a hierarchical structure in which lower-level nodes (representing more specific objects) may be connected to higher-level nodes (representing more generic objects) by one or more edges. A lower-level node (or the lower-level object it represents) may be referred to as a descendant of the higher-level node (or the higher-level object it represents), and a higher-level node (or the higher-level object it represents) may be referred to as an ancestor of the lower-level node (or the lower-level object it represents).

[0113] The method may further include receiving one or more rules corresponding to the subset of the directed graph. The rules may be inference, logic, conjecture, or RDF Schema rules. The method may further include materializing the subset of the directed graph by applying the rules to the plurality of statements to compute additional statements. Materializing the subset of the directed graph may be described as adding context data or references to context data to the subset of the directed graph.

[0114] Materializing a subset of the directed graph may be performed by applying inference or applying (inference) rules to the subset of the directed graph. A numbered example of a rule is: 1. Every object of the predicate "dbo:foundationPlace" is a country 2. Every subject of the predicate ":capitalOf" is a city 3. Every object of the predicate ":capitalOf" is a country

[0115] The first rule may be implemented by setting the range of the predicate "dbo:foundationPlace" such that the object of that predicate must be an instance of the Country class. The second rule may be implemented by setting the domain of the predicate ":capitalOf" such that the subject of that predicate must be an instance of the City class. Like the first rule, the third rule may be implemented by setting the range of the predicate ":capitalOf" such that the object of the predicate must be an instance of the Country class.

[0116] An example of materialization (i.e., inference) follows. Materialization is the inference that :Madrid :capitalOf :Spain. and the following properties: :capitalOf rdfs:domain :City. :capitalOf rdfs:range :Country :capitalOf rdfs:subPropertyOf :locatedIn. Based on.

[0117] Thus, materializing may include combining a statement with one or more properties. More specifically, materializing may include combining a statement with a property (e.g., a property restriction) that restricts the subject or object of the statement. The combination may be used to determine further statements (e.g., the class of which the subject of the statement is an instance and / or the class of which the object of the statement is an instance). Materializing may be understood as determining statements that can be implicitly derived from the directed graph and adding the determined statements to the directed graph. Three numbered examples of inference follow below. 1. :Madrid :capitalOf :Spain. :capitalOf rdfs:domain :City. → :Madrid a :City. 2. :Madrid :capitalOf :Spain. :capitalOf rdfs:range :Country → :Spain a :Country. 3. :Madrid :capitalOf :Spain. :capitalOf rdfs:subPropertyOf :locatedIn. → :Madrid :locatedIn :Spain.

[0118] Each of the above three examples combines the statement ':Madrid :capitalOf :Spain' with a different property to compute (i.e., derive) an additional statement. In the first example, the statement ':Madrid :capitalOf :Spain' is combined with the property ':capitalOf rdfs:domain :City' to compute ':Madrid a :City', which indicates that the subject of the statement, ':Madrid', belongs to (i.e., is an instance of) the class ':City'. In the second example, to compute ':Spain a :Country', ':Madrid :capitalOf :Spain' is combined with the property ':capitalOf rdfs:range :Country', which indicates that the object of the statement, ':Spain', is an instance of the class ':Country'. In the third example, :Madrid :capitalOf :Spain is combined with the property :capitalOf rdfs:subPropertyOf :locatedIn to compute :Madrid :locatedIn :Spain, which states that the subject :Madrid has the property :locatedIn with respect to the object :Spain.

[0119] Before the subset of the directed graph is received and before the string is computed, each additional computed statement (ie, materialized statement) may be added to the subset of the directed graph.

[0120] Continuing the example, given the pre-materialization statement and properties, the following SPARQL query returns FALSE: ASK { :Madrid a :City.}

[0121] After materialization, the same SPARQL query returns TRUE. Materializing a directed graph may increase the effectiveness of the computed strings in training a language model, given the additional inferences made and logical connections made between statements. Furthermore, the inferencing ability of a language model may improve with the level of detail of the input provided to the language model. Thus, because materializing a directed graph increases the level of detail of the directed graph, strings computed from a materialized directed graph may be more effective in training a language model than strings computed from an unmaterialized directed graph.

[0122] The directed graph may be materialized when statements are inserted into the directed graph (e.g., before defining the conditions and patterns). Thus, the step of receiving one or more rules corresponding to a subset of the directed graph, and the subsequent materialization step, may be performed before defining the conditions and patterns. This may lead to faster computation of strings from the subset of the directed graph because materialization has already been performed.

[0123] Alternatively, the step of materializing the directed graph may be performed on a subset of the received directed graph, which may have the advantage of providing better performance when materialization is not used, or may increase the efficiency of creating the directed graph.

[0124] Additionally or alternatively, the method may further include sorting the subset of the directed graph so that nodes are grouped together with their neighboring nodes. Sorting may be performed after materializing the subset of the directed graph. Sorting may include determining a list of nodes in the subset of the directed graph and adding a randomly selected node to the new list of nodes. For each node in the new list of nodes, determining connected nodes. For each connected node, adding the respective node to the list of nodes if the respective node is within the subset of the directed graph. The method may further include removing the node from the list of nodes.

[0125] Pseudocode for the sorting algorithm described in the previous paragraph is given below. graphNodeSet = graph.getAllNodes() backlog = [] WHILE graphNodeSet NOT EMPTY IF backlog IS EMPTY / / Perform a random selection / jump in the graph backlog.add(graphNodeSet.randomDraw()) END IF FOR r IN backlog DO / / Cover all n neighbors FOR n IN r.connectedTriple DO serialize(r, n.edge, n.node) IF n.node IN graphNodeSet backlog.add(n.node) END IF END FOR graphNodeSet.remove(r) END FOR

[0126] The serialize() function above may verbalize each triple ('r' -- subject, 'n.edge' -- object, 'n.node' -- predicate). In other words, the serialize() function may convert statements from a subset of a directed graph into a serialization format such as RDF / XML, RDFa, Notation3 (.n3), or Turtle (.ttl), N-Triples, or JSON-LD.

[0127] Sorting algorithms, sometimes called clustering algorithms, may ensure topicality (i.e., that computed strings are close to each other in the sense of relating to similar or the same topic). In other words, adjacent computed strings are semantically similar. This may increase the effectiveness of the strings in training language models.

[0128] The method may further include training a language model using the calculated strings.

[0129] For example, providing the computed string as input to a language model may include training (e.g., further training) the language model using the computed string. For example, the language model may be pre-trained or extensively trained, but the training may not include data from the subset of the knowledge graph. Thus, training the language model using the computed string may extend the capabilities of the language model and enable the language model to assist with tasks related to data from the subset of the knowledge graph.

[0130] Training a language model using a subset of the directed graph may be advantageous in leveraging or expanding upon the significant effort and expense that has gone into language models. For example, training a large language model such as ChatGPT, PaLM, Megatron, Titan, or Chinchilla may take months and cost tens of millions of dollars or euros. Enabling a large language model to apply information from a subset of the directed graph may be a way to further leverage the effort and expense already spent on training a large language model.

[0131] Additionally, training a language model using the subset of the directed graph may include fine-tuning the language model (e.g., by applying low-rank adaptation) to optimize the language model for a task or domain, e.g., the domain of the subset of the knowledge graph.

[0132] For more information on low-rank adaptation, see "LoRA: Low-Rank Adaptation of Large Language Models," Edward Hu et al., June 17, 2021.

[0133] In some cases, the language model is a probability distribution over sequences of words. The language model may be a large-scale language model (e.g., having at least 1 million parameters or at least 1 billion parameters).

[0134] In some cases, the language model includes a neural network. The neural network may be a deep neural network (e.g., a neural network with one or more hidden layers). The neural network may have at least 1 million parameters (e.g., weights and biases) or at least 1 billion parameters. The neural network may have been trained on unlabeled (i.e., unannotated) text using unsupervised (i.e., self-supervised) learning.

[0135] Additionally or alternatively, the neural network may include a transformer that uses self-attention, thereby differentially attentively attributing importance to portions of input data provided to the neural network. Input to the neural network may be parsed into tokens, which may be processed simultaneously by calculating weights for the tokens in successive layers of the neural network. The neural network may be designed to process sequential input data. The neural network may include weights that can be changed during runtime (e.g., soft weights).

[0136] According to another aspect, there is provided a computer program (e.g. a computer program product), the computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to any one of the preceding claims.

[0137] According to yet another aspect, a computer-readable medium stores a computer program. For example, the computer program may be tangibly embodied in the computer-readable medium. In other words, the computer-readable medium may be a non-transitory storage medium.

[0138] According to a further aspect, a computer system for providing data from a directed graph to a language model is provided. The system includes a database that stores the directed graph. The system further includes a software service configured to define a plurality of conditions and a plurality of patterns. Each of the conditions has at least one corresponding pattern. The software service is further configured to receive a subset of the directed graph from the database. The subset of the directed graph includes a plurality of statements. Each of the statements includes a subject, an object, and a predicate that associates the subject with the object. For each statement in the subset of the directed graph, the software service is configured to perform the following: when one of the conditions matches the respective statement and a pattern corresponding to the condition can be applied to the respective statement, compute a string from the respective statement using the pattern. The software service is further configured to provide the computed string as input to the language model.

[0139] The software service may be a web service, which may run on a server and listen for network requests on a port, for example, port 80.

[0140] The subject matter described in this disclosure may be implemented as a method or on a device, possibly in the form of one or more computer programs (e.g., computer program products). Such computer programs may cause a data processing apparatus to perform one or more of the operations described in this disclosure.

[0141] The subject matter described in this disclosure can be implemented in a data signal or on a machine-readable medium, the medium being embodied in one or more information carriers, such as a CD-ROM, a DVD-ROM, a semiconductor memory, or a hard disk. In particular, the disclosed subject matter may be tangibly embodied in a non-transitory machine (computer) readable medium.

[0142] Additionally, the subject matter described in this disclosure may be implemented as a system including a processor and a memory coupled to the processor. The memory may encode one or more programs for causing the processor to perform one or more of the methods described in this application. Further subject matter described in this disclosure may be implemented using a variety of machines.

[0143] The details of one or more implementations are set forth in the illustrative drawings and the description below. Other features will be apparent from the description and drawings.

[0144] In the following text, a detailed description of examples is given with reference to the drawings. Various modifications to the examples may be made. In particular, one or more elements of one example may be combined to form a new example and used in other examples.

[0145] Figure 1 shows a subset 100 of a directed graph. More specifically, Figure 1 shows the subset 100 of a knowledge graph. The nodes of the directed graph are shown as circles, and the edges of the directed graph are shown as arrows. The directed graph subset 100 includes labels 102, 104, 106, 108, and 110, which are URIs and defined in the Resource Description Framework (RDF). The node labels 102 and 104 are objects, and the edge labels 106, 108, and 110 are predicates. The string "1972-01-01" may also be a label (i.e., an object) of a node with type xsd:date.

[0146] The directed graph subset 100 includes a statement 112 (i.e., a triple statement) with a subject "dbr:SAP_SE", a predicate "dbo:foundationPlace", and an object "dbr:Germany", where the subject, predicate, and object are each URIs defined in RDF. An example serialization of statement 112 is dbr:SAP_SE dbo:foundationPlace dbr:Germany. The schema of the directed graph may be defined by the World Wide Web Consortium's (W3C) RDF Schema (RDFS) or Web Ontology Language (OWL).

[0147] 2 shows the domain 201 and range 203 of the property ":capitalOf." The domain 201 and range 203 may be defined as follows: :capitalOf rdfs:domain :City. :capitalOf rdfs:range :Country.

[0148] 3 shows an example SPARQL query 301 of a knowledge graph. Query 301 is configured to determine the answer to the following question: What is the population of Chinese cities with more than 100,000 residents? The results of running query 301 are also shown.

[0149] 4 shows the architecture of a system for providing data from a directed graph to a language model. The language model may be trained using data from the directed graph. User interface and software components are shown as rectangles, and storage components are shown with rounded corners.

[0150] The system may take as input a subset 100 of the directed graph (or a reference to the subset 100), custom conditions and patterns, and configuration options. The subset 100 may be provided as a set of triple statements. A pattern and configuration storage 409 may store default patterns, while custom conditions and patterns are provided by the client 401.

[0151] The configuration options may include indicating whether question patterns are used in addition to or exclusively with text patterns. The configuration options may specify how multiple patterns corresponding to a condition are handled.

[0152] RUN_ALL: If the condition is TRUE, all patterns assigned to the condition are applied.

[0153] RUN_RANDOM: The patterns are randomly ordered and the first pattern that applies to the statement is used.

[0154] Another configuration option may specify whether the directed graph should be materialized (default is TRUE). Other ways of handling multiple patterns and other configuration options may also be used.

[0155] The directed graph may be materialized (e.g., by generating agent 411) and the statements of the directed graph may be sorted. A string may then be computed from the statements of the subset 100 of the directed graph, as discussed in connection with FIG.

[0156] A client 401 may be used to interact with a software service 403. The client 401 may interact with the software service 403 through different user interfaces (UIs) 405 and 407 to maintain patterns and / or configurations in a pattern and configuration storage 409 or to compute strings from a subset 100 of a directed graph via a generation agent 411. The pattern and configuration storage may be accessible through a pattern maintenance and access application programming interface (API) 413. The subset 100 may be the entire directed graph or may be a proper subset of the directed graph identified by a query (e.g., a SPARQL query). The SPARQL query may be constructed through a user interface (e.g., a low-code or no-code platform) that abstracts the query language.

[0157] The directed graph may be stored in storage 415 and the strings computed from the statements of the directed graph subset 100 may be stored in storage 417 .

[0158] 5 shows an algorithm for computing a string from statements (i.e., triples) in a subset 100 of a directed graph. In the example shown, the directed graph is sorted before the string is computed. Before computing (i.e., generating) a string from a default pattern, it is first determined whether a condition corresponding to a specific (i.e., custom) pattern matches the statement. If a condition corresponding to a specific pattern matches the statement, the specific pattern is used to compute the string. Otherwise, the default pattern is used to compute the string.

[0159] When there are multiple specific patterns and multiple default patterns, the specific patterns may be checked first, and only if none of the specific patterns can be applied may one of the default patterns be used to calculate the string.

[0160] After the string is calculated, it may be reformed using a reformulation language model. The reformulation language model may be different from the trained language model. The reformulation language model may be a language model with high or very high accuracy with respect to f, where f is a reformulation function: f(sentence) = sentence'

[0161] The modified language model (also called a paraphrased language model or an encoder-decoder model) may be implemented using Google T5, FLAN-T5, or Quillbot. The modified language model may have high accuracy (e.g., at least 90% correct) or extremely high accuracy (e.g., at least 99% or at least 99.9% correct). The modified string may have a greater degree of language variation than the originally computed string. The modified string may be provided to a user and may attract more attention from the user than the originally computed string. Alternatively, the modified string may be provided to a language model to be trained. The language variation of the modified string may produce better results when training a language model than the originally computed string.

[0162] The modified strings may sometimes be incorrect. Therefore, a human may have the option to accept or reject the modified strings. The accepted and / or rejected modified strings may be used to retrain the modified language model. Furthermore, the accepted and / or rejected modified strings may be used to compare the quality of different modification functions f.

[0163] 6 shows an example sorting algorithm for sorting statements of a directed graph. Sorting may prevent outputting computed strings in a random order. Once statements are sorted, same or similar topics may be grouped together. The sorting algorithm may have the following form: List <triple>result = sort(Set <triple>)

[0164] Set <triple>corresponds to the statement on the directed graph before sorting, and <triple>result corresponds to the statement in the directed graph after sorting. "sort" invokes a sorting function such as a function implementing the exemplary sorting algorithm shown in Figure 6. Other sorting functions may also be used.

[0165] The graphNodeSet variable initially holds the unsorted statements of the directed graph, and the backlog variable contains the sorted directed graph upon completion of the sorting algorithm.

[0166] Figure 7 shows a directed graph that may be provided as input to the sorting algorithm of Figure 6. Assuming node 1 is drawn first, the output is the following statement: 1a2 1 b 3 1 c 4 2d6 7 e 8

[0167] As shown in Figure 7, each number above represents a node and each letter represents an edge. Therefore, rather than random output, the sorting algorithm ensures that nodes that are close to each other in the directed graph are grouped together in the computed string provided as output. In other words, after being computed, adjacent strings cover semantically similar content.

[0168] Figure 8 shows that a condition has at least one corresponding pattern (also called a text pattern), but may have multiple corresponding patterns. Given a statement (e.g., an RDF triple), the condition may evaluate to TRUE or FALSE. If the condition evaluates to TRUE (i.e., the condition matches the respective statement), it is determined whether at least one pattern corresponding to the condition can be applied to the respective statement. If a pattern can be applied, a string is computed from the respective statement using the pattern.

[0169] Figure 9 extends Figure 8 to cover question patterns in addition to text patterns, showing that a condition has at least one corresponding text pattern, but may have multiple corresponding text patterns. Furthermore, a condition may have zero or more question patterns. The syntax of the question pattern may be identical to the syntax of the text pattern.

[0170] The question pattern may allow a conversation-like string to be computed, which may be particularly useful for some language models (e.g., language models that require conversation).

[0171] Figure 10 extends the algorithm of Figure 5 to cover question patterns. In particular, if a configuration option that enables the use of question patterns is set, specific conditions corresponding to the question patterns are checked. If the question pattern corresponding to the specific condition can be applied, the question pattern is used to compute a string from each statement. Otherwise, a default pattern is used to compute a string from each statement.

[0172] The configuration options that specify how multiple patterns corresponding to a condition are handled may be extended to question patterns as follows: In particular, the following three configuration options may be used to handle multiple patterns assigned to at least one condition: RUN_ALL: If a condition matches a statement and multiple question patterns are assigned to the condition, all possible question patterns are executed. If there are multiple text patterns and multiple question patterns, the Cartesian product of the text patterns and question patterns is executed. RUN_RANDOM: If a condition matches a statement, a random order is determined for all available question patterns assigned to the condition. Then, the question patterns are tested for execution from top to bottom. The first question pattern that can be executed is executed and the process stops. RUN_ALL_QPATTERNS_RANDOM_TEXT_PATTERN: All question patterns are used, but if multiple text patterns exist, only random text patterns are used to generate answers.

[0173] Other methods of handling multiple patterns assigned to a condition may also be used.

[0174] Example 3 - Exemplary Relationships Between Knowledge Graphs, Ontologies, and Meta-Ontologies 11 illustrates the relationship between a knowledge graph 1108, a knowledge graph ontology 1110, and an ontology meta-ontology 1112. The ontology 1110 may be used to describe elements of the knowledge graph 1108. Similarly, the meta-ontology 1112 may be used to describe elements of the ontology 1110. The knowledge graph 1108 can be considered, in some sense, an instance of the ontology 1110, and the ontology can be considered an instance of the meta-ontology 1112. Generally, concepts are described from a more specific to a more conceptual perspective, progressing from the knowledge graph 1108 to the ontology 1110 to the meta-ontology 1112.

[0175] Aspects of the present disclosure are described with respect to the particular example of motor vehicles, which may be broadly referred to as vehicles or more specific vehicle types such as automobiles and motorcycles.

[0176] Looking at ontology 1110, it can be seen that the ontology describes several classes 1118, such as vehicle 1118a, car 1118b, or motorcycle 1118c. Ontology 1110 also defines several properties 1122, such as properties that may be possessed by class 1118, or more specifically, properties that may be possessed by an instance of a class. In this example, property 1122a may be an isA property, such as a property that may be used to define a particular instance as an instance of a particular class 1118. Properties 1122b and 1122c describe particular properties of class 1118, such as the property of having a license plate or the property of having an owner, respectively. As described, ontology 1110 can itself be considered an instance of meta-ontology 1112, and instance identifier 1126b indicates that the ontology is an instance of the meta-ontology, as being an instance of ontology concept class 1150a of meta-ontology class 1150.

[0177] The knowledge graph 1108 is shown as including an instance 1160 of a vehicle class 1118a and an instance 1162 of a bike class 1118c. Each instance 1160, 1162 includes values ​​1160a, 1160b, 1162a, 1162b for properties 1122b, 1122c. The knowledge graph 1108 also defines various relationships 1170. For example, a relationship 1170 may associate a particular node with a particular class 1118 of the ontology 1110, or may associate a node with a particular property 1122 and the property's corresponding value.

[0178] As explained, meta-ontology 1112 describes concepts used in ontology 1110 at a higher conceptual level. For example, ontology 1110 has specific classes 1118, while meta-ontology 1112 defines the concepts of classes. Similarly, meta-ontology 1112 defines concepts such as properties and relationships, including the properties by which one object can be an instance of another object.

[0179] For purposes of this disclosure, an object is used to refer to a particular instance of a class, and a class is represented by a collection of instances. A class and its instances can have properties, including data type properties and object properties. For example, object properties can refer to relationships between objects (which may also be called instances, entities, or resources), and data type properties associate objects with literal values.

[0180] Example 4 - An example property identification process FIG. 12 shows a flow diagram of a process 1200 for automatically extracting properties from a dataset. At 1210, a source is presented. The source corresponds to data representing one or more instances of a class. Examples of sources include electronic documents, such as text documents or tables. The source includes electronic data, but need not be in any particular format. The data can be structured, such as in CSV format, or unstructured, such as free text in a word processing document. In some cases, rather than providing the source directly, the location of the source can be specified, such as using a URL, and the URL can be used to retrieve the data from the source.

[0181] In at least some implementations, data from a source may be extracted and converted into one or more common formats. For example, a document (or more broadly, a "file") having text may have the text extracted as a string. The string may be used to generate a graph of the concepts in the string, such as a graph in RDF format. Some documents, such as web pages, may natively contain RDF descriptions (e.g., using JSON-LD, Microdata, or RDFa), and therefore the RDF descriptions may be extracted in RDF format. Some documents may have some text already represented in RDF format and other text represented only in non-RDF format. In such cases, text elements that are not in RDF format may be extracted as strings, converted to RDF format, and optionally concatenated with existing RDF descriptions in the source.

[0182] At 1220, an instance graph is created. The input can be text extracted from the document at 1210. The input can be processed using information extraction techniques, such as using OpenNRE (github.com / thunlp / OpenNRE, where NRE stands for neural relation extraction), which can extract relationships from the input text. In some cases, a knowledge graph can be generated using coreference resolution, named entity linking, and relationship extraction (as described at neo4j.com / blog / text-to-knowledge-graph-information-extraction-pipeline / ). Generally, information extraction can be performed using one or more of pattern matching, machine learning, named entity recognition, or dependency parsing, including those known in the fields of information extraction, natural language processing, and knowledge graph construction. Another suitable information extraction technique is disclosed in U.S. Patent Application No. 17 / 973,322, filed October 25, 2022, which is incorporated by reference herein to the extent not inconsistent with this disclosure.

[0183] At 1230, a matching process is performed using the instance graph. The matching process at 1230 generates a list of suggested properties and a list of instances that contain such properties. Note that the properties do not have to be associated with a class, and the list of properties need not all use the same names for properties that represent a common semantic concept. That is, the matching process can take into account differences in the terminology used to describe the same property. For example, different instances may use "fuel economy," "mileage," or "fuel efficiency" to refer to the same property.

[0184] Although any suitable matching technique may be used, suitable matching techniques include those described in US Pat. Nos. 11,487,721 and 11,263,187, which are incorporated herein by reference.

[0185] The results of the matching process 1230 are provided as input to a property verification and validation operation 1240. The operation of 1240 includes selecting properties to be used in the ontology, including resolving differences in terminology / labels for particular properties. If desired, the instance graph obtained in 1220 may be converted to use the aligned vocabulary. In some cases, the operation of 1240 may be at least significantly automated, while in other cases, at least some of the operation may be performed by a user.

[0186] Example 5 - An exemplary computing environment implementing a property identification process 13 provides an exemplary computing environment 1300 in which the disclosed technology may be implemented. For example, the computing environment 1300 may be used to perform the operations of the process 1200 of FIG.

[0187] The computing environment 1300 includes a computing system 1304 in communication with a client 1308. The computing system 1304 can include a user interface 1312 that can be used by the client 1308 to perform various operations, as described.

[0188] The computing system 1304 further includes a source manager 1320. The source manager 1320 can be responsible for acquiring documents, possibly including extracting data from the documents. The source manager 1320 can include one or more source adapters 1322, shown as 1322a and 1322b. The source adapter 1322 is configured to extract data from a particular source type, where extracting data can include formatting the data or removing formatting from the data. Consider a document created by a word processor. The document can include text and text formatting information, such as field codes, font styles, or spacing information. The source adapter 1322 can be configured for document types generated by word processors, which can parse the document to extract text while removing non-text information. Similarly, the source adapter 1322 can be configured to extract data from database tables, JSON documents, or web pages.

[0189] The source adapter 1322 may be configured to implement the interface INTERFACE extract (PARAMETER Source f) RETURNS ONE OF [RDF, String]

[0190] The source manager 1320 has access to the source store 1326. The source manager 1320 can retrieve documents from the source store 1326, such as for processing the documents with the source adapter 1322. The source manager 1320 can also cause the source store 1326 to store text extracted from the source.

[0191] A user, such as client 1308, can cause source manager 1320 to perform various actions, such as through user interface 1312. For example, a user can cause documents to be loaded into source store 1326, request that particular documents be processed by source adapter 1322, or request that a grouping of sources in the source store be created (such as a grouping of sources that are believed to represent related instances to be used in extracting properties of an ontology).

[0192] The computing system 1304 includes a graph generator 1332. The graph generator 1332 may perform at least a portion of operation 1220 of process 1200 of Figure 12. The graph generator 1332 may include functionality for performing named entity recognition and relationship extraction. The graph generator 1332 may have access to the source store 1326 and may store graph information, such as storing triples in an interim triple store 1336.

[0193] The matching executor 1344 may access the interim triple store 1336 to identify properties that are the same but referred to using different terminology, as in operation 1230 of process 1200. In some cases, mappings may already exist between different schemas, and the previous mappings can be retrieved from the mapping memory 1348, which may reduce the time and computing resources required to perform the matching process. The matching results generated by the matching executor 1344 may be stored in the triple store 1360. The triple store may also store triples associated with the list of properties identified as a result of process 1200.

[0194] Example 6 -- Example source for use with property identification 14A-14C illustrate exemplary sources, such as sources from which text may be extracted. The sources in Figures 14A-14C provide information about one or more car models, where a given car model may represent, for example, an instance of a car, and a particular car instance may be described by one or more sources.

[0195] FIG. 14A shows a descriptive source 1400 with the automobile model "SE" produced by the automobile company SAP. Note that source 1400 includes various keywords that a human might recognize as relating to automobile properties. However, the disclosed technology provides methods that can automate ontology creation / property identification, including methods that are not performed by a human. For example, a human might review instance data for one or two automobile models and then use intuition to extract properties such as the automobile class. A human, for example, would not create an instance graph from individual sources or perform a matching process as described.

[0196] In FIG. 14A , example words that may be associated with properties include “electric power,” “vehicle,” “plug-in,” “hybrid,” “kW,” “hp,” “fuel efficiency,” “electric power consumption,” “range,” and “emissions.” FIG. 14B shows another source 1420 describing a fictitious automobile from SAP SE. Note that source 1420 includes some words and properties that correspond to words and properties used in source 1400. For example, source 1420 also includes words such as “kw,” “hybrid,” and “hp.” Source 1420 also includes words that are not present in source 1410 but that may be properties of an automobile, such as “gasoline engine” or “engine,” “torque,” ​​“accelerates,” “adaptive,” “suspension,” and “differential.” Note that some of these words may indicate properties, while others may indicate property values. For example, “differential” may correspond to a “differential type” property, and “sport” may be an example literal value for “differential type.”

[0197] 14A and 14B, has text from source 1430, at least a portion of which may correspond to properties (or literal values) that are also present in source 1400 or source 1420, and the terms used for the properties may be the same or different. Source 1430 may include properties or example literal values ​​that are not in source 1400 or source 1420, and may exclude properties or literal values ​​that are present in source 1400 or source 1420. For example, source 1430 does not include the term "differential" that was included in source 1420.

[0198] Example 7 -- Exemplary User Interface for Executing a Property Identification Process and an Exemplary Property Counting and Alignment Process Figure 15 provides an exemplary user interface 1500 that allows a user to select source files to be analyzed, such as during operation 1210 of process 1200 of Figure 12. In panel 1510 of user interface 1500, a user can select specific files to be analyzed, such as by using control 1512 to add a file, control 1514 to add all files from a particular folder, or control 1516 to remove a particular file. Field 1520 displays a list of currently selected source or input files 1522. Generally, a user may select files that are expected to contain instances of one or more classes, etc., from which properties should be extracted, such as for use in building an ontology.

[0199] User interface 1500 provides access to functionality through instance calculation control 1528 that can be used to determine the instances that exist in file 1522 of field 1520. For example, selecting control 1528 may cause the instances that exist in file 1522 of field 1520 to be identified. As discussed, multiple files may represent / describe a single instance.

[0200] Recognizing instances may be performed in various ways. For example, the structure of files 1520 may be analyzed to determine if there are common elements, etc. In the example shown, files 1520 are identified by URLs, which have the form of an automobile manufacturer name followed by an automobile model name. In other cases, determining instances may be based on various heuristics, such as by tentatively assigning files to common instances based at least on term usage counts. For example, if "SE" is used multiple times within a file or set of files, it may be determined that "SE" likely represents a particular automobile instance.

[0201] Panel 1540 lists instances 1542 associated with input file 1522. Instances 1540 may be instances obtained by automated analysis of the contents of input file 1522, or may correspond to instances manually added by a user. For example, a user may choose to delete instance 1540 using delete instance control 1546 or add an instance using add instance control 1544.

[0202] Figure 16 provides a user interface 1600 through which a user can initiate a knowledge graph discovery or update process such as that corresponding to the actions performed during operation 1210 of process 1200 of Figure 12. User interface 1600 provides a panel 1610 that lists instances 1614 used in creating or updating the knowledge graph. Instance 1614 may correspond to instance 1540 of Figure 15.

[0203] Panel 1620 lists input (or source) files 1624 corresponding to a selected one of the instances 1614, 1614a. The input files 1624 correspond to some of the input files 1522 of FIG. 15. As mentioned above, multiple input files 1522 may describe the same instance 1614. User interface 1600 may provide controls 1628 that allow a user to add input files to the set of files for instance 1614a and controls 1632 that allow a user to remove such files. Panel 1636 provides a preview of a selected one of the input files 1624a. A user may initiate a graph creation / update process for one or more of the instances 1614 by selecting a user interface control 1640.

[0204] Figure 17 shows a user interface 1700 that allows a user to view and optionally modify a knowledge graph determined after selecting a user interface control 1640. Panel 1710 lists instances 1714 for which a knowledge graph was determined, which may correspond to instance 1614 in Figure 16. Panel 1720 displays the instance graph of the selected instance 1714, such as instance graph 1724 for instance 1714a.

[0205] The graph 1724 includes nodes and edges, where the nodes may correspond to instance nodes 1728, property nodes 1730, and literal value nodes 1732. The edges can represent relationships, such as edge 1740 indicating that an instance "hasProperty" or edge 1742 indicating that a property "hasValue."

[0206] Optionally, user interface 1700 includes controls that can enable a user to modify the graph, such as those shown with respect to instance graph 1724. The user interface controls may include a control 1750 for adding a node, a control 1752 for deleting a node, a control 1754 for adding an edge, or a control 1756 for deleting an edge.

[0207] A matching process may be used to determine the properties associated with different instances of a particular ontological concept, as described with respect to operation 1230 of Figure 12. A mapping software function may implement the interface: INTERFACE match (GraphSet G) RETURNS Alignment.

[0208] Some properties may be common to all or a significant number of concept instances, while others may be less common or even unique to a single instance. When determining the properties of an ontological concept, it may be useful to determine the frequency of occurrence of particular properties. Generally, what is desired is to determine which instances have properties that have the same semantic meaning, even if the terms / labels used for the properties differ between instances.

[0209] Figure 18 provides example pseudocode 1800 for performing instance alignment and counting the occurrences of a particular semantic property, regardless of the label applied. In loop 1810, pseudocode 1800 performs an alignment between a particular instance graph and the final graph being constructed for a set of instance graphs. In particular, line 1814 calls the "match" function, which performs the alignment. In line 1818, for a property in the instance graph being processed, if the property is already present in the alignment, the count of that property is incremented by one. If the property is not already present in the alignment, the count of that property is set to one.

[0210] 19A shows a user interface 1900 presenting results from the instance graph matching process, including property counts. User interface 1900 includes a table 1910 with a column 1914 for properties identified during the graph analysis and matching process, a column 1916 indicating the number of times the property was observed in the instance graph, and a column 1920 that accepts user input regarding whether a given property should be included in the “final” ontology generated from the instance data.

[0211] User interface 1900 includes controls 1930 that allow a user to add a property, controls 1932 that allow a user to modify a property, and controls 1934 that allow a user to delete a property. Modifying a property may include changing the name / label applied to the property. That is, the name of the property in column 1914 may be a property name automatically selected from the instance data. In certain implementations, a default name is selected that corresponds to the name that appears most frequently in the instance data or the name that appears most frequently in a set of separate instances (i.e., counting the number of instances in which the name is used at least once).

[0212] 19B shows user interface 1900 after selection of control 1932. In this case, window 1950 allows the user to select a name from a drop-down list 1954, whose choices may correspond to names specified in the input set. The user may be allowed to enter a different property name that does not appear in the instance data. Window 1950 is also shown as including fields indicating ranges (such as a data type and, optionally, a range or list of values ​​observed in / matched with the instance data).

[0213] In other implementations, instead of or in addition to having a user add properties to an ontology, properties may be added automatically, including if they meet defined criteria. For example, a rule may be defined that a property is added to a set (of ontology) if it appears in the set of instances more than a threshold number of times, or if it appears in a percentage of the instances more than a threshold amount.

[0214] Additionally, in some cases, a graph such as the finalGraph defined in pseudocode 1800 of Figure 18 may be modified based on the final set of selected properties. For example, nodes that do not correspond to instances or to properties in the set of selected properties may be removed from the graph.

[0215] Example 8 -- Example property evolution Figure 20 shows how the properties of an instance of a particular class (or subclass) defined in an ontology may evolve over time. Instance graph 2010 represents a particular car model 2012 (an instance of the Car Model class). At the time this instance was processed, the ontology included the following properties: unladen weight 2018, engine type 2020, number of doors 2022, and an indication of whether the car has seat belts 2024. Corresponding values ​​2028, 2030, 2032, and 2034 are associated with these properties 2018, 2020, 2022, and 2024 in the instance graph 2010.

[0216] Instance graph 2010 can correspond to a portion of a knowledge graph, and thus values ​​2028-2034 can be associated with properties of the knowledge graph, regardless of whether some properties may not currently be represented in the ontology associated with the knowledge graph. Alternatively, instance graph 2010 does not correspond to a knowledge graph, and the instance graph can have properties that do not correspond to properties in the knowledge graph or properties in the ontology. That is, the disclosed technology can be adapted for use with graphs similar to instance graph 2010, even if the instance graph is not formally a knowledge graph or associated with a knowledge graph, so long as techniques are provided for tracking changes to properties and property values ​​identified as being of interest to a particular user (or computing process).

[0217] In particular, when instance graph 2010 was first created, the ontology did not include properties related to electric vehicles, such as recuperation, an electric vehicle's ability to recharge its battery using regenerative braking or deceleration. However, later, another instance is processed to generate an instance graph 2050 for a particular vehicle model (instance) 2052. In addition to existing properties 2018, 2020, 2022, 2024 and their associated values ​​2058, 2060, 2062, 2064, instance graph 2050 introduces a "recuperation" property 2070 and its corresponding value 2072, indicating that vehicle instance 2052 has regeneration capabilities. Note that despite its presence in instance graph 2050, recuperation property 2070 is not present in the vehicle class ontology.

[0218] The disclosed technology can recognize the emergence of new instance properties and can track the occurrence of properties, including to add the property to a relevant ontology or to prompt a user to consider adding the property to an ontology.

[0219] Next, consider property 2024 indicating that a car has seat belts. Over time, instances such as those generated from a particular electronic document may no longer mention this property, even if the car reflected by the instance does include seat belts. For example, seat belts are now required to be included in all vehicles sold in the United States. Before the law required seat belts, including seat belts in a vehicle was optional. Thus, documents describing a particular vehicle may have mentioned whether the vehicle included seat belts. After the law mandated seat belts in all cars sold, at some point, use of the term "seat belts" in documents describing cars may decrease, perhaps because the general public takes for granted that new cars have seat belts. Maintaining the seat belt property may cause confusion for users, since it may be unclear whether the property is optional. Also, including obsolete properties in an ontology may make the ontology more complex and more difficult to maintain, update, and understand.

[0220] Maintaining obsolete properties can also lead to data integrity and consistency issues: for example, an aggregation operation performed on instances that have a value for an obsolete property and instances that do not have a value for the property may yield incorrect results—such as identifying a lower percentage of vehicles that have seat belts, even though newer vehicles include seat belts, because this information was not present in the documents processed to generate the vehicle instances.

[0221] The disclosed technology can be used to recognize when a property may be obsolete and either mark the property as obsolete or prompt the user to consider marking the property as obsolete. In some cases, the obsolete property is removed from the ontology, while in other cases, the obsolete property continues to be kept in the ontology but is marked as obsolete. In the latter case, keeping the property but marking it as obsolete can allow the property to be used in historical analyses, although the property may be "ignored" in scenarios where only active properties are of interest.

[0222] Example 9 - Exemplary Computing Environment for Property Management Generally, the disclosed techniques involve tracking the use of properties in an instance over time. This tracking includes new properties that appear in the instance. That is, in at least some implementations, the disclosed techniques assume that an upstream process is available to recognize new properties for the instance. In some cases, these properties may be manually added as instance properties, while in other cases, the properties may be automatically determined. For example, the techniques described in Examples 4-8 may be used to identify new properties.

[0223] 21 shows a computing environment 2100 that is a modified version of the computing environment 1300 of FIG. 13. Elements of the computing environment 2100 that are the same as or at least generally similar to elements of the computing environment 1300 are identified using the reference numbers of FIG. 13.

[0224] Compared to computing environment 1300, computing environment 2100 adds a property information store 2110. The property information store 2110 can maintain information about property usage by instances over time. As discussed further, the property information store 2110 can store information about properties, instances (e.g., from processed documents) that include the properties, and one or more dates associated with the usage of the properties in the instances. Optionally, the property information store 2110 can include aggregated information, such as a total count of property usage over a particular time interval. The information can also include specific property values ​​associated with instances that include the properties.

[0225] The matching and insertion executor 2114 may be similar to the matching executor 1344 of FIG. 13, but may also include functionality to write information about new properties or property usage to the property information store 2110. In other implementations, the property store 2110 may be used in a computing environment that does not include components of the computing environment 2100, in which case new properties and property usage information may be stored in the property store 2110 using a separate process. In some cases, aggregated values ​​are calculated and written to the property information store using components other than the property information store 2110. In other scenarios, the property information store 2110 includes functionality to perform aggregation. In yet other scenarios, the property information store 2110 does not store aggregated values, but does store information from which aggregated values ​​may be calculated.

[0226] The computing environment 2100 also includes a property manager 2130. The property manager 2130 can retrieve information from the property information store 2110 and, at least in some cases, can also write information to the property information store. For example, the property manager 2130 can write property status information, including changes in the status of a property, to the property information store 2110. In scenarios where the property manager 2130 performs aggregations or calculations, this information can optionally be written to the property information store 2110.

[0227] The property manager 2130 can perform operations such as tracking property usage and making or proposing property status changes based on such tracking. In at least some cases, the user is alerted when a property is added or a property status is changed, or when such an addition or property change is proposed, provided that implementing the addition or change requires the user's approval. The user interface 1312 can cause an appropriate user interface to be rendered to provide this functionality and can also enable the user to perform actions such as viewing information in the property information store 2110 (including executing queries against the property information store and displaying the query results).

[0228] The computing system 1304 can communicate with multiple clients 1308. When performing operations, the property manager 2130 can, in certain implementations, access data from multiple clients 1308. Evaluating data from multiple clients 1308 can provide a more comprehensive view of property usage. In some cases, evaluating data from multiple clients 1308 can result in changes being made to the client's ontology. In other cases, evaluating data from a single client 1308 that results in changes or proposed changes to the ontology can be used to modify or propose ontology changes to another client that either uses the same ontology or has an ontology that includes at least some overlapping properties.

[0229] Example 10 -- Example storage of property usage data 22 shows an example data object that may be used to store property information for a property. The data object may be used in the property information store 2110 of FIG.

[0230] Data object 2210 tracks the use of a particular property 2212a, including through the use of a business timestamp 2212b and a system timestamp 2212c associated with a particular document having a document identifier 2212d. Data object 2210 may be implemented in a variety of ways, including as a table in a relational database or in a key-value store. Data object 2210 may be implemented as a CSON / JSON object. A single data object 2210 may be used for a single property, or a single data object may track information for multiple properties.

[0231] One advantage of the structure of data object 2210 is that it allows documents to be associated with specific properties. Suppose a document originally has a property, but the property is not active in the ontology, and therefore the property's data is not recorded (e.g., in a data object such as a table or view). If the property is subsequently changed to an active state, data object 2210 can be used to identify documents that have the property, and the documents can be reprocessed so that the value of the now-active property can be identified and stored.

[0232] Data object 2220 tracks the status of a property over time. Data object 2220 may be implemented in a manner similar to data object 2210 and may also be used in property information store 2110. Data object 2220 may track information for a single property or multiple properties.

[0233] A data object 2220 is shown with three possible values ​​for the status attribute 2222b of a particular property 2222a. The statuses are active, appearing, and obsolete. An active property refers to a property that is active in the ontology. When an instance / document is processed, it can be determined whether the instance / document has the property and the appropriate value can be recorded.

[0234] A property is an occurrence when it has been recognized as a property but has not yet been promoted to active status. For example, a property may not have been used frequently enough to qualify for active status. The disclosed technology can track property usage over time. If the property is used more frequently, such as over a particular time interval, the property can be changed to active status, or the user can be prompted to consider changing the property to active status.

[0235] An obsolete property is a property that was once active but whose recent usage has become relatively infrequent. In some cases, the threshold for making a property obsolete can be the same as the threshold for making a property active. In other cases, the thresholds can be different. For example, the threshold for making a property obsolete can be higher than for changing a property from present to active. That is, once a property becomes active, its usage must be lower than the threshold for making the property active in the first place.

[0236] Other status types may be used in this disclosure. For example, a property may have a status of "active proposal" or "deprecated proposal." These status types may be useful when user input is required for a status change based on at least one set of criteria. The user may approve a change from "active proposal" or "deprecated proposal," resulting in the property having an "active" or "deprecated" status. In some cases, a combination of techniques may be used, such as proposing a change to active or deprecated at one threshold or automatically making the change if a higher threshold is met.

[0237] For each status, the data object 2220 further tracks a total count 2222c of instances having the property, a count 2222d of instances having the property over a particular interval, start and end dates 2222e, 2222f of the interval, and a value 2222g observed for the property during the interval. The value 2220g can be useful in determining why a property may have changed status during the interval.

[0238] In the case of automobiles, the "engine type" property may have indicated more frequent use. Analyzing the value of "engine type" compared to older intervals may reveal that a more recent interval has a new value for engine type of "electric," compared to older intervals that only had values ​​of "gas" or "diesel." For example, a user or process trying to determine whether to change "engine type" from obsolete back to active, or from emerging to active, may use the new value of "electric" to help determine why the increased use is occurring and how it may affect the decision to make the status change.

[0239] As the above discussion suggests, a given property may have multiple intervals with respect to a given status. A property may be new, have an emerging status, and be used frequently enough to have an active status for a period of time, after which the property becomes obsolete. Circumstances may change such that the property is used again or used more frequently, optionally again progressing to emerging status, and then again promoted to active status. The increased frequency of use may (or may not) coincide with a new value for the property. Thus, a property has two intervals associated with the active status. This type of cycle may similarly result in multiple intervals with respect to obsolete or emerging status, or other statuses such as proposed active or proposed retired.

[0240] The total count 2222c and the interval count 2222d may be implemented in different ways, as desired. In one implementation, the total count 2222c may be for all periods in which the property has a given status, and the interval associated with the interval count 2222d may be for a particular interval in which the property had a given status. That is, the data object 2220 may have a "record" of each period in which the property had a particular status. In other implementations, there may be multiple intervals within an entire continuous period in which the property had a particular status. In this case, the total count 2222c may be for the entire period, and the interval count 2222d may be for a particular interval within the entire period.

[0241] Example 11 - Example intervals and interval membership criteria Interval counts, how frequently a property is used over a period of time, can be determined using aggregations such as over a specific interval or "window." An interval can be defined by a date and a period from that date. Typically, a window is defined as a specific period (e.g., two weeks, one month) before a given date, but an interval can optionally be determined based on a period after a given date. A period can be expressed in any suitable way, such as a fixed number of years, months, weeks, or days. In other cases, an interval can be specified using a specific start and end date.

[0242] FIG. 23 illustrates an interval 2310 for a property. The interval 2310 is defined by a date D 2314 and an interval size. The start date 2318 of the interval 2310 is determined by subtracting the interval size from date D. The interval 2310 includes multiple documents 2322 related to the instance, each of which includes a business timestamp 2322a, a system timestamp 2322b, and the instance's document ID 2322c, as shown for document 2322a. Note that the interval 2310 does not necessarily include all documents processed in the interval. Rather, the interval 2310 typically includes only documents that have the relevant property.

[0243] In general, intervals can be calculated in a variety of ways. For example, intervals can be calculated using the date a document was processed, a system timestamp, or a business date (such as the actual date a document was created or the actual date an event occurred). When available, it is usually desirable to use the business date when determining the date of a document because it represents the "real word" use of the property. The system date can be much later than the business date. Consider a birth or marriage record. A birth certificate may not be issued, or an update indicating a change to an individual's marital status may not occur in the system until after the actual birth or marriage has occurred. However, the system date may be used if desired, especially in scenarios where there is little difference between the business date and the system date. In other scenarios, when available, the business date is used to determine whether a document should be included in an interval; otherwise, the system date is used.

[0244] Example 12 -- Example automatic instance counting and property classification This disclosure provides techniques for periodically calculating a count of a property over an interval and maintaining, proposing, or making a change to a status based on such count. For example, a threshold may be set for the number of instances having a property within an interval that will result in a change from appearing to active or proposed active, or from active to obsolete or proposed obsolete. The threshold can be set manually or can be determined automatically.

[0245] Automatically determining the threshold may include monitoring the number of occurrences within an interval for properties with an active status. The threshold for promoting from an occurring status to an active status may be set as the number of occurrences of the least frequently occurring active property. An average count or a median count may also be used to set the threshold. Additionally, statistical techniques including distributions may be used to determine the threshold. For example, the threshold may be set as a number of standard deviations from the average occurrence count of properties with an active status. Values ​​for other status types, such as obsolete, proposed to be obsolete, or proposed to be active, may be determined in a similar manner.

[0246] As described, in some scenarios, the user approves the status change. The threshold value may be determined with reference to the occurrences that resulted in the user making the status change. The threshold value may be calculated in a manner similar to that described above, including by using a median, an average, or may be calculated statistically using values ​​associated with user-initiated status changes.

[0247] 24 shows example pseudocode 2400 for calculating an aggregate for a property over an interval and then using the results to determine whether the property's status should be changed. The aggregate function 2410 has parameters for the location of the property characteristic information (or an object having the characteristic information) and an interval duration value. In example pseudocode 2400, the interval (e.g., start and end dates) is determined by subtracting the interval duration from the current date. In other implementations, the aggregate function 2410 can have parameters including the interval duration and a base date from which the interval is to be calculated, or can have parameters including the start and end dates used to define the interval.

[0248] In loop 2414, pseudocode 2400 loops through all properties in the characteristic information and calculates the sum of instances that have the property within the interval.

[0249] Function 2430 assigns a status to the property of the characteristic information over the duration of the interval. The status is determined based on a threshold, which may be set manually or determined as described above. Conditional statement 2434 applies if the property's count exceeds the threshold and the property does not currently have an active status. Conditional statement 2434 sets the property's status to "active suggestion."

[0250] Conditional statement 2436 is applied if the threshold is met and the property currently has an active status. In this case, the property remains set to the active status. Conditional statement 2438 is applied if the threshold is not met and the property currently has an active status. Conditional statement 2438 sets the property status to "proposed to deprecate." Conditional statement 2440 is applied if the threshold is not met and the property does not currently have a status. In this case, the property may be set to the "present" status. Although not shown, function 2430 may include conditional statements that retain the "present" status for properties that do not meet the threshold but were previously assigned the "present" status.

[0251] According to function 2430, the remaining properties are assumed to be obsolete, so conditional statement 2442 sets the "obsolete" status for all properties that did not satisfy the previous conditional statement.

[0252] Pseudocode 2400 may be modified in various ways. For example, function 2430 uses a single threshold. As described, multiple thresholds may be configured, including having different thresholds for when a property is promoted to active status and when a property is demoted to obsolete status. Pseudocode 2400 may also be modified to include other status changes, such as changing the status directly from "appearing" to "active," "active proposed" to "active," "active" to "appearing," "present" to "obsolete," "active proposed" to "obsolete," or "active" to "obsolete." These additional status changes may be associated with additional thresholds.

[0253] Example 13 -- An example user interface screen for viewing property information and modifying property status 25-31 provide exemplary user interfaces that may be used in implementing the disclosed techniques. Figure 25 shows a user interface 2500 that may be used to add emerging properties to a knowledge graph or other set of properties and ontologies associated with the knowledge graph.

[0254] User interface 2500 provides information about a particular instance, such as a document identified in field 2508. A user may choose to open the document by selecting user interface element 2512.

[0255] As described, there can be multiple documents that describe an instance. User interface 2500 provides user interface controls 2516, 2518 that allow a user to indicate that the document is a new instance and enter an identifier for the instance. User interface 2500 also provides user interface controls 2520, 2522 that allow a user to associate the document with an existing instance along with the identifier of the existing instance.

[0256] A table 2526 of properties 2528 identified in the instance is provided, along with the identified value 2530 of the property and an identifier 2532 indicating whether the property already exists in the ontology, knowledge graph, or other set of tracked properties.

[0257] A user can obtain additional information about a property by selecting property 2528 and selecting user interface element 2536. The additional details are presented in panel 2540. Panel 2540 includes a table 2542 listing the selected property's characteristic details 2544 and their values ​​2548. The characteristic details include the current status 2544a, a count 2544b of the property in the current interval, and a total count 2544c of the property over multiple intervals (including all intervals, at least all intervals since the property entered its current status).

[0258] Panel 2540 also provides information 2546 about the type, such as the data type, of the value, such as whether the value is numeric, Boolean, string, or associated with a category that has enumerated values.

[0259] Table 2550 provides links to documents / instances having the property, which may be selected and viewed by activating user interface element 2560. Table 2550 includes a document link or identifier 2554, a date 2556 the document was entered into the system, and a count number 2558 associated with the instance. That is, as explained, an instance may be described by multiple documents, and count number 2556 may indicate the number of documents for the instance that contain the selected property. Count number 2556 may reflect a total count or a count number over an interval. Table 2550 may optionally include additional information, such as a business date associated with the instance or document.

[0260] A user may choose to add the property to a knowledge graph or other set of tracked properties by selecting user interface element 2570. A user can select user interface element 2574 to add values ​​for other instances that have the property. That is, as discussed, once a property is marked active, previous instances or documents that have the property can be reprocessed so that their values ​​for the property are captured.

[0261] 26 shows a user interface 2600 that allows a user to view information about properties that may be considered for transition to an obsolete status. In panel 2610, table 2614 lists the property 2616, the property's current status with respect to the ontology 2618, the number of occurrences of the property in an interval 2620, and the start date of the most recent interval 2622. The user can obtain more information about the property by selecting property 2616 and selecting user interface element 2624.

[0262] Panel 2630 provides additional information for the selected property 2616. In panel 2630, table 2634 lists various characteristic details 2636 of the property and corresponding values ​​2638. Property details 2636 include a current status 2636a determined by a property manager, such as property manager 2130 of FIG. 21. For clarity, a property may have a current status with respect to an ontology, represented as current status 2618, and a status associated with analyzing the property's information over a particular interval, such as the status resulting from pseudocode 2400 of FIG. 24. Property details 2636 further include a count number in the current interval 2636b for the property and a total count number 2636c for the property, which may have the meanings discussed with respect to FIG. 25.

[0263] 25, user interface 2600 can provide data type information 2650 of the property's values ​​observed for instances within an interval, and information about the particular instance / document on which the property was observed 2660. A user can change the status of a selected property to an obsolete status by selecting user interface element 2664.

[0264] 27 illustrates a user interface 2700 that provides information for multiple properties, including details about values ​​associated with such properties. User interface 2700 includes a panel 2708 that provides a table 2710 that includes a column 2712a that identifies a particular property, a column 2712b that provides the current status of the given property, a column 2712c that provides the number of instances in the most recent interval that includes the property, and columns 2712d, 2712e that provide the start and end dates of the current status. Note that for properties that have an active status, a value may not be included in column 2712e, or a null value may be used, or another value may be used that indicates that the property status is not associated with an end date.

[0265] A user can obtain further details about a property by selecting a particular property and selecting user interface element 2720. The further details are displayed in panel 2730. Similar to user interface 2500, panel 2730 includes characteristic details 2734 and data type information 2738. In addition, panel 2730 provides table 2746 that includes column 2748a that lists values ​​observed for the property during the current interval and column 2748b that lists the number of occurrences of a particular property value over the interval. As described, this information can provide insight into the appropriate status of the property, including assessing why the property may be recommended for promotion to active status or demoted to obsolete status.

[0266] Additional details of a particular property can be obtained by selecting user interface element 2754, which causes user interface 2800 of Figure 28 to be rendered. User interface 2800 can provide comprehensive information about a particular property, including information displayed in other user interface screens. For example, user interface 2800 displays characteristic information 2734, data type information 2738, and table 2746.

[0267] User interface 2800 displays information about the various statuses that the property has been in in the form of a table 2824 that includes interval information shown in column 2826a, which provides the interval start date, column 2826b, which provides the interval end date (if the interval has ended), and column 2826c, which provides the status of the property over the interval.

[0268] Information about the sources describing a particular instance is also displayed in user interface 2800 in the form of table 2840. Table 2840 includes column 2842a that identifies the source, column 2842b that provides the system entry date of the source, and a count of the number of occurrences of the property for the source.

[0269] A user can choose to change the status of a property to "obsolete" by selecting user interface element 2850. Note that the nature of user interface element 2850 may change depending on the current status of the property being viewed on user interface 2800. For example, if a property is currently in an "obsolete" status, user interface element 2850 may allow a user to set the property to an "active" status. For properties with statuses such as "appearing," "active suggestion," or "obsolete suggestion," user interface 2800 may optionally provide multiple user interface elements that allow a user to make a status change. For example, for an "appearing" property, a user may be given the option to activate the property or delete the property. For an "active suggestion," a user may be given the option to activate the property, change the status to "appearing," or delete the property.

[0270] 29 provides a user interface 2900 that provides more detailed historical information about property status over time. In particular, user interface 2900 provides a table 2910 of usage intervals. Table 2910 includes columns 2914a, 2914b, which provide the start and end dates (if appropriate) of the interval. Column 2914c provides the status of the property over the interval, while column 2914d provides a total count of instances having the property over the interval. The instance values ​​over the interval are provided in column 2914e, while column 2914f may include notes about why a particular status change occurred.

[0271] From table 2910, it can be seen that intervals with an active status are associated with significantly higher counts than intervals with a retired status. As explained, these values ​​can be used in an automated process to set or recommend a status change. That is, if the count observed over an interval is closer to the count of an interval with a retired status, the property status can be changed from active to retired, or such a change can be recommended. If the count is closer to the count of an interval with an active status, the property can remain in active status, or if in retired status, can be changed from retired to active, or such a change can be recommended.

[0272] The counts in column 2914d and instance values ​​in column 2914f may be used to explain why a status change occurred. For example, in an earlier interval, alcohol-fueled vehicles may have been considered as an alternative to gasoline-powered vehicles. Over time, that fuel source became less likely to be in widespread commercial use, and therefore use of the term "fuel type" may have decreased as gasoline was the primary fuel type. At a later date, a similar scenario may have occurred with diesel-powered vehicles, where the property went from retired status to active, but then went from active to retired again because diesel-powered vehicles were found to have higher-than-expected emissions. New technology that uses electricity as a fuel type may cause the property to become active once again.

[0273] Automated techniques that assign statuses to properties can take into account the emergence or disappearance of particular property values. If a new property value emerges, it may be a stronger trigger to make the property active or recommend active status. If an existing property value begins to appear less frequently, it may be a trigger to make the property obsolete or recommend obsolete status.

[0274] The user interface 2900 can provide visual aids to help a user understand the usage of a property over time, such as a graph 2950 of the property's count over time. The graph 2950 can be provided for a single interval, multiple intervals, or the property's complete history. In FIG. 29, the graph 2950 shows a sudden increase in the property's usage, which may be associated with an active status. Over time, the property was used less, but still heavily used. At a later time period, the property was used less, which may correspond to a transition to an obsolete status.

[0275] 30 shows a user interface 3000 that can provide information about property usage over time, such as over a particular interval. The user interface 3000 includes a graph 3010 formed from nodes 3014 that represent properties. Edges 3018 between the nodes 3014 indicate relationships between the properties. Property relationships can be based on the properties being part of a common class or subclass, having a mutual relationship, a property / subproperty relationship, being part of a common property chain, or having other logic or restrictions that define the relationships between the properties.

[0276] Graph 3010 shows properties and their relationships, but does not provide information about how frequently the properties are used. The disclosed technology allows graph 3010 to be modified, such as graph 3020, to provide usage information, such as information about the count of properties over an interval. Nodes 3014 in graph 3020 are shown with different types of shading that correspond to typical levels of usage. For example, properties with high usage counts may be colored darker than properties with low usage counts. Thus, graph 3020 allows a user or computing process to understand how a property is "trending."

[0277] FIG. 31 provides a user interface 3100 that shows changes in property usage over multiple intervals and captures information about properties that may have been added or removed from the set of ontologies / properties being tracked.

[0278] User interface 3100 shows various states of graph 3110 over three intervals. Graph 3100 is constructed in a similar manner to graph 3010 of Figure 30, with nodes 3104 and edges 3106. Similar to graph 3020, graph 3110 uses shading to show information about property usage / counts over a given interval.

[0279] Going from the version of graph 3110 for interval 3130 to the version of the graph for interval 3132, it can be seen that node 3136 represents a property whose usage has increased, while node 3138 represents a property whose usage has decreased. Graph 3110 for interval 3132 also reflects the addition of a new property, represented by node 3142.

[0280] The graph 3110 for the later interval 3146 further shows nodes 3150 where usage has increased and nodes 3154 where usage has decreased.

[0281] Example 14 - Example source of change information 32 shows an example electronic newsletter 3200 that may have information that is relevant to at least some users. The newsletter 3200 includes text. The text may be processed, such as using named entity recognition techniques and entity linking, to associate entities in the text with a knowledge graph or ontology.

[0282] The text includes links 3210. In some cases, the disclosed technology can follow links 3210 to other data sources, which can be processed to determine whether they may provide additional instances of the knowledge graph element or additional information about the newly identified instance (such as an instance in the e-newsletter 3200).

[0283] 33 illustrates another type of electronic document that may contain information that is relevant to at least some users: email 3300. Similar to newsletter 3200, email 3300 contains text that may be processed and linked to elements in a knowledge graph or ontology. The text of email 3300 includes links 3310 that may be followed to other documents that may contain relevant information.

[0284] As discussed in Example 1, informational documents such as newsletters tend to contain information that may be relevant to various stakeholders. The relevant information may be only a particular passage of a much longer piece of text, or may be obtained through a link within the document, such as links 3210, 3310. In some cases, information relevant to a particular topic may be contained in the same area of ​​the document, such as subtopics 3230, 3330 in Figures 32 and 33. In the case of links 3210, 3310, this further illustrates how difficult it can be for a user to manually find relevant information.

[0285] Similar to how relevant different information may be to different users, the timeliness requirements for users to be presented with updated information may vary among different users. That is, some users may not need to know the information at all, some users may want to see the information but not with any particular urgency, and other users may have an urgent need for the information. As will be explained, the disclosed technology can consider not only whether information is relevant to a particular user, but also the degree of priority / relevance with which the information is relevant to the user.

[0286] The range of information that is relevant to a user may also vary. For example, some users may be satisfied with general information about a particular topic, while other users may want detailed information. From a knowledge graph perspective, this may be expressed as whether one or more identified nodes of the graph are relevant, or whether nodes related to such nodes should also be considered relevant. When related nodes are relevant, a degree of indirection may be used to limit the range of which nodes are considered relevant. As described, when defining what types of information are relevant to a user, the user may weight certain topics as being of higher or lower interest, which may be used in determining other information that the user may be interested in, even if they did not explicitly identify such information.

[0287] Example 15 -- Example data model for intent definition 34 shows an example data model 3400 that may be used in the disclosed technology. The data model 3400 includes a user element 3410. The user element 3410 may represent a particular user or a class of users. A given user or user class associated with the user element 3410 may be associated with zero or more intent elements 3414.

[0288] The intent element 3414 defines a specific topic / set of one or more knowledge graph or ontological elements in which the user is interested and wants to receive updates regarding changes in the knowledge graph. The intent element 3414 is associated with one or more intent nodes 3418. The intent nodes 3418 correspond to graph nodes 3424, nodes in the knowledge graph or ontology. When a graph node 3424 corresponds to a node in the ontology, the node is also linked to the corresponding node in the knowledge graph, so that when a value of a property in the knowledge graph is added, removed, or changed, it can be correlated with the intent element 3414.

[0289] An intent node 3418 differs from a graph node 3424 in that an intent node may be subject to restrictions or parameters for the particular intent associated with the intent element 3414. For example, an intent node 3418 may be associated with zero or more restriction properties 3432. Restriction properties 3432 may be defined as value restrictions on graph properties 3436. Graph properties 3436 are generally restricted to those properties defined for a particular class associated with the graph node 3424 (the graph node 3424 may correspond to an instance of a class).

[0290] A user can express an intent at a particular graph node 3424, either directly or by associating user-provided input with a corresponding knowledge graph node. However, in some cases, it may be desirable to broaden the intent to include other graph nodes 3424 that the user may be interested in. For example, the user may not be aware that such graph nodes 3424 exist and, therefore, may not be aware that they are relevant to the intent for which they are defined.

[0291] A user may choose to associate a priority element 3444 directly with an intent node 3418 or can assign a priority to the intent element 3414, which then applies to all intent nodes defined for the priority. Providing a user with the ability to weight particular intent nodes 3418 may provide a more refined representation of the inferred intent. In some cases, including when a user does not specifically define a priority element 3444 for one or more intent nodes 3418, a default priority may be assigned, including a default priority specified for an intent element 3414 and applied to all intent nodes defined for the intent element. In other cases, a particular intent node 3418 may be assigned a different priority even if the user does not manually assign a priority element 3444. For example, the system may track intent node 3418 usage with respect to intent elements 3414, and intent nodes that are more commonly used or have a higher priority associated with them may be assigned a higher priority in an automated process.

[0292] An update interval 3448 is also defined for the intent element 3414. In some cases, if the update interval 3448 is not specified, a default update interval may be used. The update interval 3448 may define how often the knowledge graph should be checked for updates relevant to the intent element 3414, or at least how often such updates should be collected and communicated to the user.

[0293] Although not shown in FIG. 34, the data model may include additional components, such as a component that associates the intent element 3414 with one or more communication types to be used to communicate update information to the user.

[0294] Although data model 3400 can be implemented in any suitable manner, in a particular implementation, the components of the data model may be implemented as one or more objects, such as tables or views in a relational database. For example, a table may be defined having columns that correspond to the elements of data model 3400. However, the data in data model 3400 may be held in other formats, including key-value stores such as JSON (JavaScript Object Notation) or CSN (Core Schema Notation).

[0295] In a further example, one or more of the elements of the data model 3400 may be defined as an abstract data type or a member variable of an abstract data type. For example, the intent element 3414 may be an abstract data type that may have user 3410, priority 3444, or update interval 3448 as member variables. The intent node 3418 may also be defined as an abstract data type, and the abstract data type of intent 3414 may include an instance of the intent node data type as a data member. The intent node data type may have member variables of graph node 3424 and limit property 3432, and optionally, the limit property may also be implemented as an abstract data type.

[0296] Example 16 -- Example intent registration and update process Figure 35 is a flow diagram of a process 3500 for registering a particular area of ​​the knowledge graph that a user is interested in, called an intent, monitoring the system for new information related to that intent, and notifying the user of such new information. The information used in process 3500 may be stored in data objects in data model 3400 of Figure 34.

[0297] At 3508, the user registers a definition of the user's intent, such as through a user interface. The intent may be specified as or correlated with a node in a knowledge graph. In some implementations, the user is presented with a list of available topics (corresponding to knowledge graph nodes) or a representation of a knowledge graph from which the user can select a topic / node. In another implementation, the user may be presented with a document having content linked to knowledge graph nodes, the user may select such content, and the corresponding knowledge graph node is added to the definition of the user's intent.

[0298] Also, at 3508, in some implementations, a user may provide input to restrict the intent with respect to one or more selected knowledge graph nodes of the intent definition. The nodes selected by the user may represent classes having properties for which restrictions may be specified. The values ​​of the properties set in the restrictions may be used to restrict the relevant results (instances). The restrictions can be positive restrictions (the property has a given value) or negative restrictions (the property does not have a particular value).

[0299] At 3512, the user may assign a priority and an update interval to the intent. One or both of the priority setting and the update interval setting may be optional. For example, the priority or update interval may not be user configurable, or if the user does not override the default priority or update interval with an input at 3512, a default priority or update interval may be used. As described in Example 15, the priority may be used to determine the inferred intent, as further described. Additionally, rather than having a priority assigned to the entire intent, priorities may be assigned to specific nodes within the intent, and different nodes may be assigned different priorities (and some nodes may not have an assigned priority).

[0300] At 3516, the defined intents are registered for the user. A given user may register multiple intents, including intents with overlapping knowledge graph nodes. Even the same set of nodes may be used in different intents, such as when one intent has a set of nodes subject to restrictions and another intent has a set of nodes with different restrictions or no restrictions. Different priorities or intervals may be defined for different intents.

[0301] At 3520, a listener is registered for the registered intent. As the term "listener" is used broadly in computer science, a listener refers to computer code that implements functionality to monitor a knowledge graph or other source of information about a knowledge graph. When an event that triggers a listener is detected, the event may be associated with one or more relevant intents. An exemplary listener and the message that triggers the listener are defined in the code below.

[0302]

number

[0303] In a particular implementation, the change that triggers the listener is a change that includes adding or removing a property value of an instance of a class in the knowledge graph, or changing the value of a property. Adding a value to a property can include adding a value for the instance that was not previously extracted from a document describing the instance, or the added value may correspond to a property that was recently added to the class. Similarly, a removed property value may correspond to a value that was removed because the property was determined to be obsolete.

[0304] At 3524, the detected changes are collected. Collecting the changes may occur at intervals defined in the intent. Alternatively, changes may be collected at other times, typically at more frequent intervals, and processed for a particular intent according to the intent's interval. When changes are collected, they may also be associated with a particular priority specified for the intent.

[0305] In a particular implementation, the changes are captured as a set of triples, such as those described in Example 2. The triples can then be verbalized at 3528, also as described in Example 2. At 3532, the triples are sent to a natural language generator, and a summary is received by the natural language generator. A notification, such as a text alert, email, or application message, is generated at 3536 and sent to relevant users at 3535.

[0306] Notifications may be created and sent according to the priority associated with the intent. For example, the selected priority may determine the communication channel used to send the notification. A text message may be used for a high priority, while an email may be used for a lower priority.

[0307] Example 17 – Exemplary Computing Environment for Intent Registration and Processing Figure 36 is a diagram of a computing environment 3600 used to implement process 3500 of Figure 35. Computing environment 3600 includes a client 3610 in communication with a subscription service 3614. The client 3610 can define one or more intents that are stored in a user configuration repository 3618. The subscription service 3614 processes the intents, including instantiating an appropriate listener 3622. The listener 3622 monitors a knowledge graph 3630 stored in a knowledge graph repository 3626.

[0308] The knowledge graph 3630 may be associated with an ontology 3634. As previously described, the ontology 3634 may be used to identify new properties, and the new properties may be added to the knowledge graph 3630. Information about these properties in the knowledge graph 3630 may be associated with knowledge graph nodes in the user's intent, thus allowing notifications of changes to be sent to the user. For example, if the user identifies a class instance node as being of interest, a message may be generated when a property value for a new property is added to the class instance.

[0309] The subscription service 3614 may include a verbalization component 3640. The verbalization component 3640 can perform actions both when processing user requests, such as a request to define an intent, and when processing messages to the user regarding such intents. In the case of an intent definition, the verbalization component 3640 can convert information provided by the user into a standardized format for expressing the intent, such as data model 3400 of FIG. 34. When sending information to the user about changes relevant to an intent, the verbalization component 3640 can convert the “raw” change information into a format more easily understood by the user, including providing contextual information that can remind the user of the context that caused the intent to be registered (which may include, for example, identifying the relevant knowledge graph or the knowledge graph elements selected by the user for the intent).

[0310] The following scenario illustrates how computing environment 3600 may be used in the disclosed technology. A user may express an intent through client 3610. As an example, user Sandra may express an intent as follows: "Please inform me once a week via email regarding changes about LLM use at SAP." This "free text" input may be processed, such as using named entity recognition and entity linking (which may be part of verbalization component 3640), to map the intent to a knowledge graph node. The knowledge graph node and intervals, as well as the preferred communication channel, may be registered in user configuration repository 3618.

[0311] In particular embodiments, a user intent may be specified as one or more triples (as described in Example 2). Given the example intent, the intent may be expressed as follows: Change update on [SAP; uses; LLM] Setting: once a week Setting: via email

[0312] The listener 3622 can check for relevant updates according to an interval. The changes can be expressed as triples. An example set of changes for an example update can be: Change Update on [SAP, uses; LLM]: {(insert node: ODM; uses; LLM); (change sub node: [P1; uses; LLM;] →[P1; ConceptVersion; ACD_P1v2,0])}

[0313] The above modifications can be sent to a natural language generator (which may be part of the verbalization component 3640), which creates an email message to be sent to the user. An example email message is: Dear Sandra, Heere is the information you requested regarding LLM use in SAP for KW 34: -ODM started using LLMs (25.08.022); click here for more information -P1 created new Concept (27.08.2022), click here for more information Regards, You KG Subscription Service (Sandra We will provide you with the information you requested regarding the use of the LLM in SAP in KW 34. - ODM has launched the LLM (25.08.022). For more information click here - P1 has created a new Concept (27.08.2022). For more information click here Thank you for your cooperation. You KG Subscription Service)

[0314] Example 18 -- Exemplary detailed computing environment for intent registration and processing 37 illustrates in more detail a computing environment 3700 in which the disclosed technology may be implemented. The computing environment 3700 includes a client 3708 in communication with a notification system 3712.

[0315] The notification system 3712 provides one or more user interfaces 3716 that a user can access to define, modify, or delete intents, including as described above. The intents may be processed by a request orchestrator 3720. The request orchestrator 3720 may store the intent information in configuration storage 3724.

[0316] The intent reasoner 3728 can perform various actions with respect to the intent. For example, as described in Example 17 with respect to the verbalization component 3640, the intent reasoner 3728 can parse the intent and express the intent in a standardized format, such as triple format. The more standardized version of the intent can then be stored in the configuration storage 3724.

[0317] The expressed intent may, at least in some scenarios, be supplemented. That is, for example, knowledge graph nodes that have relationships to explicitly identified knowledge graph nodes may be added to the intent. In some cases, whether or to what extent additional nodes are added may be correlated with user input. For example, a priority given by the user or another representation of how interested the user is in a particular topic may be used for this purpose. This “extended” intent definition may be stored in configuration storage 3724.

[0318] The listener service 3732 can instantiate a listener 3734 based on the intent stored in the configuration storage 3724. The listener 3734 created by the listener service 3732 can monitor the knowledge graph 3740 in the knowledge graph system 3738 in various ways. In the scenario shown in FIG. 37 , the listener 3734 can monitor the change log 3746 stored in the change log storage 3744.

[0319] Change log agent 3750 can write change logs to change log storage 3744. For example, when the knowledge graph is updated to add, remove, or change values ​​associated with a class instance, a log entry can be written to change log storage 3744. In some scenarios, intent reasoner 3728 can also access change log storage 3744. When an expressed intent is modified to include at least some additional nodes, the addition or removal of nodes in knowledge graph 3740 can cause changes to the expanded intent.

[0320] When listener 3734 detects changes, such as from change log storage 3744, the listener can notify request orchestrator 3720 of all changes. Request orchestrator 3720 can then send the changes to natural language generator verbalization module 3754. Request orchestrator 3720 can specify parameters for verbalization of the changes using natural language generator verbalization module 3754, such as by selecting a particular template instruction for a particular communication type to be sent.

[0321] The message 3756 generated by the natural language generator verbalization module 3754 is provided by the request orchestrator 3720 to the notification module 3760. The notification module 3760 can have communicators 3764 (shown as 3764a and 3764b) for various communication types, such as email communicator 3764a or collaboration software communicator 3764b. The communicators 3764 are configured to package and send the message 3756. For example, email communicator 3764a can paste the contents of the message 3756 into an email template, add a subject and recipients, and contact an email server to send the message. The communicator 3764 can, in some cases, access the configuration storage 3724 to retrieve contact information for the particular user to whom the message is to be sent. In other scenarios, the request orchestrator 3720 may retrieve the contact information from the configuration storage 3724 and provide the information along with the message 3756 to the notification module 3760 .

[0322] Example 19 -- Example determination of inferred intent As described, in some cases, instead of strictly limiting intents to specific topics reflected by knowledge graph nodes specifically identified by the user, intents may build on those topics and be expanded to identify other related topics that the user may also be interested in. For example, consider a scenario in which a user expresses interest in nodes belonging to the "movie" class. In a narrow interpretation, the user may only receive alerts when property values ​​of a given movie instance are changed, added, or removed. However, instances of the "movie" class may have connections with instances of the "actor" class. In certain situations, the user may receive alerts about changes to the movie's properties, but may also be presented with changes made to actors associated with the movie.

[0323] Optionally, when expressing an intent, a user can weight various concepts (e.g., assign a “priority” to the intent as a whole or to specific nodes in the intent) and, therefore, their corresponding knowledge graph nodes. The priorities can be used in determining which nodes to include in the inferred intent and, optionally, how frequently updates are provided. That is, for example, in some scenarios, updates can be provided at user-specified intervals for the specific intent expressed by the user. It is also possible to generate inferred intents that include more knowledge graph nodes than explicitly specified by the user. Inferred intents can be analyzed at the same frequency as they are expressed by the user, or at a different frequency. Generally, the fewer direct relationships a knowledge graph node has with explicitly specified knowledge graph nodes, the less frequently the knowledge graph will be analyzed for changes to them. If the relationships between nodes become too weak, the node will not be included in the inferred intent.

[0324] The concept of node priority and update frequency is illustrated in knowledge graph 3800 of Figure 38. Knowledge graph 3800 includes multiple nodes 3810 (shown as nodes 3810a-3810c) connected by edges 3814. Assume that a user provides an intent that directly specifies node 3810a. Updates related to the node are provided frequently, either due to the assigned priority or the priority that would be assumed if node 3810a were explicitly identified in the intent.

[0325] Node 3810b represents a node that was either not directly specified in the intent, or that was specified but has a lower priority than node 3810a. Because node 3810b is inferred or explicitly identified as a lower priority, updates related to such node are also provided to the user, but less frequently than node 3810a. Node 3810c represents a node that has a weaker relationship with the specified node 3810a or that has been assigned a lower priority than node 3810b. Node 3810c has a lower priority than the threshold set for "inferred intent," and therefore updates are not provided to the user.

[0326] In some cases, the relevance of a node 3810 may be affected not only by the degree of indirection but also by the number of connections to other nodes. This scenario is illustrated in graph 3900 of Figure 39. Graph 3900 includes multiple nodes 3910 (shown as nodes 3910a-3910n) connected by edges 3914.

[0327] In the scenario of graph 3900, assume that nodes 3910a and 3910b are directly identified by the user as relevant within the intent expression. The inferred intent can be determined by assigning values ​​to the relevant nodes, and half of the node's value is propagated to the relevant nodes. In this scenario, half of the value of node 3910b, 0.5, is propagated to nodes 3910c and 3910d that are directly connected to node 3910b. A similar situation occurs for node 3910a, where 0.5 is propagated to directly connected nodes 3910e and 3910f.

[0328] Now consider node 3910g, which is connected to nodes 3910c and 3910e. Half the value of nodes 3910c and 3910e, 0.25 for each node, is allocated to node 3910g. Therefore, node 3910g also has a value of 0.5. Values ​​can be allocated to nodes 3910h to 3910n in a similar manner.

[0329] Techniques for propagating node values ​​can be implemented in various ways. With respect to graph 3900, propagation is constrained to be unidirectional. For example, because node 3910c propagates its value to node 3910g, the value associated with node 3910e is propagated to node 3910g but not further propagated to node 3910c.

[0330] In some scenarios, constraints may be applied that nodes may have a maximum value, or that the value of an "inferred node" cannot exceed a maximum value. In one example, a constraint may be applied that an inferred node cannot have a higher value than the directly specified node (which is 1 in the case of graph 3900). This constraint is consistent with the presumption that the user has directly identified what they believe to be the most relevant node.

[0331] Similarly, it may be useful to normalize node values, such as to be between 0 and 1. Other constraints may include not propagating values ​​beyond a set limit of indirection. Additionally, graph 3900 assumes that all edges 3914 have equal weights. Because it may be possible that some nodes are more strongly associated than others, propagation between nodes can take into account edge weights and rules that determine how much of a node's value to propagate to related nodes.

[0332] Once the values ​​of the nodes in graph 3900 are determined, the values ​​can be compared to a threshold that determines whether the inferred intent is strong enough to include the inferred node in the intent or to create a new inferred intent that includes the stated intent and the specified node from the inferred intent. Assume that a threshold t≧0.70 is set. Using this criterion, node 3910i is added to the inferred intent.

[0333] Although a "distance factor" of 0.5 was used in the example using knowledge graph 3900, other distance factors may be used. In some cases, rather than using the same distance factor for all nodes, the factor may vary. For example, the distance factor itself may decrease as the number of detours increases. Different nodes in a knowledge graph may have different relationship strengths or trust levels, and this information may be used as or in addition to the distance factor. For example, the overall weighting used to propagate values ​​may be determined as the product of the strength of the relationship between two nodes and a defined distance factor.

[0334] Example 20 -- Example Intent Definition User Interface 40 and 41 provide exemplary user interfaces 4000, 4100 that allow a user to define an intent. In FIG. 40, user interface 4000 includes a panel 4010 that displays a knowledge graph 4014, which may be the entire knowledge graph or a subset of a larger knowledge graph. Graph 4014 includes nodes 4018 that correspond to instances of classes or subclasses and nodes 4020 that represent values ​​(literals) of properties of nodes 4018. Edges 4026 between nodes 4018 represent relationships between class instances (one class instance may correspond to a property value of another instance), while edges 4030 represent properties with literal values, linking the instances represented by nodes 4018 to the property values ​​represented by nodes 4020.

[0335] A user can define an intent by selecting user interface element 4032. A user can provide an identifier for the intent in user interface element 4038 of panel 4034. An update frequency for the intent can be specified using user interface element 4042.

[0336] The user can then add node 4018 to the intent by selecting user interface element 4046. For example, user interface element 4046 can be selected, and then the user can select node 4018. For a given node, the user can optionally specify a priority using user interface element 4050. The user can also select user interface element 4054 to add one or more constraints for the node. The constraints can be specified as a property, such as edge 4030, displayed in user interface element 4058, and one or more values, displayed in user interface element 4062. In other scenarios, the user can specify a particular combination of property and value by selecting node 4020. The techniques of this disclosure can provide updated information to the user regardless of whether the property value is a literal or a class instance.

[0337] For some nodes 4018, the user may want to receive updates more frequently than the overall intent defined. User interface 4000 provides a user interface element 4066 where the user can choose to be immediately notified of all changes affecting node 4018 (subject to any value constraints that may be defined).

[0338] Once the user has finished defining their intent, the user can register / activate the intent by selecting user interface element 4070.

[0339] User interface 4100 is generally similar to user interface 4000 of FIG. 40. However, rather than displaying a knowledge graph, panel 4110 displays content 4114. Content 4114 can be a document containing information relevant to one or more instances of one or more classes in the knowledge graph. In some cases, a user may select specific text of the content to add to an intent, such as by “right-clicking,” after the relevant text is selected. When the text is selected, the computing system can verify whether the text corresponds to an element in the knowledge graph. In other scenarios, the document can be preprocessed, as shown, and text that corresponds to a knowledge graph element and is selectable for the intent can be visually distinguished. With respect to content 4114, selectable text is indicated with a dashed underline. The written text can be selected, and a dialog window 4120 can be presented with an option 4122 to add the knowledge graph element to the intent or an option 4124 to cancel the selection.

[0340] Otherwise, intents can be defined as described with respect to user interface 4000, with user interface 4100 including controls labeled similarly to user interface 4000.

[0341] Example 21 -- Example log of Knowledge Graph changes As described with respect to Figure 37, in some implementations, changes may be determined by examining log entries. Figure 42 shows an example log 4200 with log entries 4210 (shown as 4210a-4210e). Each log entry 4210 is associated with a date 4220a and a time 4220b when the log entry 4210 was written. The log entry 4210 further includes a user identifier 4220c of the user associated with the logged activity and an identifier 4220d of the action performed.

[0342] Exemplary log 4200 includes entries 4210a and 4210e corresponding to updates to a knowledge graph. More specifically, the updates correspond to updating values ​​associated with particular properties of particular instances of a class. Update entries 4210a and 4210e include triple 4230, which represents the previous state of the knowledge graph, and triple 4234, which represents the updated state of the knowledge graph.

[0343] Log entry 4210b corresponds to the deprecation or removal of information from the knowledge graph. Triple 4238 reflects that information is deprecated in the knowledge graph.

[0344] Log entries 4210c and 4210d reflect the addition of information to the knowledge graph. Log entries 4210c, 4210d include triple 4242 that contains the information being added. In this case, triple 4242 reflects the addition of a value to a particular property of an instance of a particular class in the knowledge graph.

[0345] A log, such as exemplary log 4200, is scanned to determine changes relevant to a particular intent expressed by one or more users. For example, listener 3734 of FIG. 37 can scan the log periodically, such as according to an interval defined for an intent. In other cases, listener 3734 can register information about nodes relevant to an intent, and a component such as change log agent 3750 can notify listener 3734 of relevant activity. This configuration can be particularly useful when a user requests immediate, up-to-date information about a particular intent or one or more nodes of a particular intent.

[0346] The listener 3734, or another component such as the request orchestrator 3720, can extract the log information into a specified format, such as the example format 4300 shown in FIG. 43. Format 4300 includes a field 4304 that describes the topic corresponding to the intent or information related to the intent. Format 4300 further includes fields 4308, 4316 for triples that were added or deleted, respectively. Field 4312 for modified triples includes both the old and new triples.

[0347] FIG. 43 also provides an exemplary update message 4350 in format 4300.

[0348] Example 22 -- Example generation of NLG prompts and NLG responses 43 conveys information about changes to the knowledge graph, but in a format that is somewhat difficult to read. Thus, the present disclosure provides techniques for conveying information in a more user-friendly manner, which may include tailoring the information to the particular type of communication, such as email, used to deliver the message.

[0349] Generally, the technique involves providing information in a message format 4200 to a natural language generator, which is provided with the information in the message format and a standard set of instructions for generating a response. Figure 44 includes example components of a prompt template 4400 that may be populated with data and provided to the natural language generator.

[0350] The input includes commands 4404, which are typically static commands. That is, the commands are used regardless of the intent being processed and provide overall instructions to the natural language generator. The input also includes change log metadata 4408 and change log details 4412. These components are specific to a particular update (information that summarizes detected changes related to the intent). Optionally, additional custom constraints 4416 may be included in the prompt template 4400. A static closing instruction 4420 may also be included, which is used for all instructions to the natural language generator.

[0351] An example prompt 4500 generated using prompt template 4400 is shown in Figure 45. Text 4510 corresponds to command 4404. In this case, command 4404 provides overall context information, such as a description of the change log data to follow, a task to be performed (generating a "better" summary of the log details), and a specification of the output format.

[0352] Text 4514 corresponds to change log metadata 4408 of prompt template 4400. Text 4514 provides information about the intent, the user for whom the intent was defined, and information about the knowledge graph, such as the maintainer of the knowledge graph.

[0353] Text 4518 corresponds to change log details 4412. Text 4518 is in format 4400 of FIG.

[0354] Text 4522 corresponds to custom constraints 4416 of prompt template 4400. In this case, text 4522 specifies who should be contacted for questions about the message and also to generate a summary of the changes. Example prompt 4500 is not shown as including closing instructions 4420.

[0355] Although an implementation of prompt template 4400 may be standardized in the sense that it is used for multiple updates and even multiple intents, multiple implementations of prompt template 4400 can be used. For example, different knowledge graphs, or different uses of the same knowledge graph, may contain different instructions regarding how information should be presented or generated. As a very simple example, the content of example prompt 4500 could be modified to generate a response suitable for being sent by text message or through in-application messaging. In some cases, a knowledge graph can be assigned to one or more specific implementations of prompt template 4400, and a suitable template can be selected based on criteria such as the user or user's role, a selected knowledge graph element, a selected interval, or a selected priority for a particular knowledge graph node. In a further example, a user may be able to select a specific implementation of prompt template 4400 to be used with a particular intent.

[0356] Figure 46 shows an example response 4600 to the example prompt 4500 of Figure 45. It can be seen that the example response 4600 is formatted as an email and addressed to the user specified in text 4514. Text 4612 provides a summary of the changes specified in the custom constraints in text 4522. Text 4616 provides the content of text 4518, the change log details, in a more readable format. Consistent with text 4514, the change log metadata, and text 4522, the custom constraints, text 4620 describes who should be contacted if the user / recipient has questions regarding the changes.

[0357] In some implementations, communications such as exemplary response 4600 may be annotated to further assist a user in understanding a change or the cause of the change. For example, the text of the communication may be processed using techniques such as named entity recognition and entity linking to identify knowledge graph elements, such as class instances, within the text. These knowledge graph elements can then be highlighted or annotated, or associated with links to additional information.

[0358] For example, exemplary response 4600 includes links 4650. These links may be selectable and, in some instances, may direct the user to additional information describing the change. In some cases, one or more documents may be associated with the change, and link 4650 may be selected to display such documents. In other cases, link 4650 may be used to provide metadata about the change, such as the time / date the change was made or an identifier for the user who made the change.

[0359] Example 23 - Example behavior FIG. 47 is a flow diagram of a process 4700 for changing the property status of properties in a knowledge graph based on counts of properties in provided electronic documents over a period of time and processing the electronic documents using the updated property status.

[0360] At 4710, a plurality of properties are received. At 4720, a plurality of electronic documents are processed over a period of time. The processing includes incrementing a respective property count for each of the plurality of properties if the document being processed includes the respective property. At 4730, for at least one property of the plurality of properties, a total count of the at least one property over at least a portion of the period of time is determined. At 4740, the total count is compared to a threshold. At 4750, it is determined that the total count meets the threshold. At 4760, the status of the at least one property is changed based at least in part on the determination that the total count meets the threshold to provide an updated property status. At least another electronic document is processed using the updated property status at 4770.

[0361] Example 24--Computing System 48 illustrates a generalized example of a suitable computing system 4800 in which the described innovations may be implemented. The computing system 4800 is not intended to suggest any limitation as to the scope of use or functionality of the present disclosure, as the innovations may be implemented in a variety of general-purpose or special-purpose computing systems.

[0362] Referring to FIG. 48, a computing system 4800 includes one or more processing units 4810, 4815 and memories 4820, 4825. In FIG. 48, this basic configuration 4830 is included within the dashed line. The processing units 4810, 4815 execute computer-executable instructions, such as for implementing the technologies described in Examples 1-14. The processing units can be general-purpose central processing units (CPUs), processors in an application-specific integrated circuit (ASIC), or any other type of processor. In a multiprocessing system, multiple processing units execute computer-executable instructions to increase processing power. For example, FIG. 48 shows a central processing unit 4810 and a graphics processing unit or auxiliary processing unit 4815. The tangible memories 4820, 4825 may be volatile memory (e.g., registers, cache, RAM) accessible by the processing units 4810, 4815, non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. The memories 4820, 4825 store software 4880 in the form of computer-executable instructions suitable for execution by the processing units 4810, 4815 that implement one or more innovations described herein.

[0363] Computing system 4800 may have additional features. For example, computing system 4800 includes storage 4840, one or more input devices 4850, one or more output devices 4860, and one or more communication connections 4870. An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects the components of computing system 4800. Generally, operating system software (not shown) provides an operating environment for other software executing in computing system 4800 and coordinates the activities of the components of computing system 4800.

[0364] Tangible storage 4840 may be removable or non-removable and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory manner and that can be accessed within computing system 4800. Storage 4840 stores instructions for software 4880 that implement one or more innovations described herein.

[0365] The input device(s) 4850 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system 4800. The output device(s) 4860 may be a display, a printer, speakers, a CD writer, or another device that provides output from the computing system 4800.

[0366] The communications connection(s) 4870 enable communication over a communications medium with another computing entity. The communications medium carries information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, the communications medium may use an electrical, optical, RF, or other carrier.

[0367] The innovations may be described in the broad context of computer-executable instructions, such as those included in program modules, being executed on target real or virtual processors in a computing system. Generally, program modules or components include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided between program modules as desired in various embodiments. The computer-executable instructions of the program modules may be executed within a local or distributed computing system.

[0368] The terms "system" and "device" are used interchangeably herein. Neither term implies any limitation on the type of computing system or computing device unless the context clearly dictates otherwise. In general, a computing system or computing device can be local or distributed and can include any combination of dedicated and / or general-purpose hardware and software that implements the functionality described herein.

[0369] In various examples described herein, a module (e.g., a component or engine) may be "coded" to perform particular operations or provide particular functionality, indicating that computer-executable instructions for the module can be executed to perform such operations, cause such operations to be performed, or otherwise provide such functionality. Functionality described with respect to a software component, module, or engine may be performed as a separate software unit (e.g., a program, function, class method), but need not be implemented as a separate unit. That is, functionality may be incorporated into a larger or more general-purpose program, such as one or more lines of code within the larger or more general-purpose program.

[0370] For purposes of presentation, the detailed description uses terms such as "determine" or "use" to describe computer operations in a computing system. These terms are high-level abstractions of operations performed by a computer and should not be confused with acts performed by a human. The actual computer operations corresponding to these terms will vary depending on the implementation.

[0371] Example 25--Cloud Computing Environment 49 shows an example cloud computing environment 4900 in which the described technologies may be implemented. The cloud computing environment 4900 includes a cloud computing service 4910. The cloud computing service 4910 may include various types of cloud computing resources such as computer servers, data storage repositories, networking resources, etc. The cloud computing service 4910 can be centrally located (e.g., provided by a company or organization's data center) or distributed (e.g., provided by various computing resources located in different locations such as different data centers and / or located in different cities or countries).

[0372] Cloud computing service 4910 is utilized by various types of computing devices (e.g., client computing devices), such as computing devices 4920, 4922, and 4924. For example, the computing devices (e.g., 4920, 4922, and 4924) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smartphones), or other types of computing devices. For example, the computing devices (e.g., 4920, 4922, and 4924) can utilize cloud computing service 4910 to perform computing operations (e.g., data processing, data storage, etc.).

[0373] Example 26--Implementation Although some operations of the disclosed methods are described in a particular sequential order for convenience of presentation, it should be understood that this method of description encompasses reordering unless a particular order is required by specific language set forth below. For example, operations described in sequence may in some cases be reordered or performed simultaneously. Moreover, for simplicity, the accompanying figures may not show the various ways in which the disclosed methods can be used in conjunction with other methods.

[0374] Any of the disclosed methods may be implemented as computer-executable instructions or a computer program product stored on one or more computer-readable storage media, such as a tangible, non-transitory computer-readable storage medium, and executed on a computing device (e.g., any available computing device, including a smartphone or other mobile device that includes computing hardware). A tangible computer-readable storage medium is any available tangible medium that can be accessed within a computing environment (e.g., one or more optical media discs, such as DVDs or CDs, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as flash memory or a hard drive). By way of example, with reference to FIG. 48, computer-readable storage media include memory 4820 and 4825 and storage 4840. The term computer-readable storage medium does not include signals and carrier waves. Furthermore, the term computer-readable storage medium does not include communications connections (e.g., 4870).

[0375] Any of the computer-executable instructions for carrying out the disclosed techniques, and any data created or used during the practice of the disclosed embodiments, may be stored on one or more computer-readable storage media. The computer-executable instructions can be, for example, a dedicated software application, or part of a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a networked environment using one or more networked computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network).

[0376] For clarity, only selected aspects of software-based implementations are described. Other details well known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology may be implemented by software written in C, C++, C#, Java, Perl, JavaScript, Python, R, Ruby, ABAP, SQL, XCode, GO, Adobe Flash, or any other suitable programming language, or in some examples, a markup language such as html or XML, or a combination of a suitable programming language and a markup language. Similarly, the disclosed technology is not limited to any particular computer or hardware type. Specific details of suitable computers and hardware are well known and need not be detailed in this disclosure.

[0377] Additionally, any of the software-based embodiments (e.g., including computer-executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or remotely accessed through suitable communication means, including, for example, the Internet, the World Wide Web, an intranet, a software application, cable (including fiber optic cable), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communication means.

[0378] The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Rather, the present disclosure is directed to all novel and unobvious features and aspects of the various disclosed embodiments, alone and in various combinations and subcombinations with one another. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments claim that any one or more particular advantages exist or problems are solved.

[0379] The technology of any example may be combined with the technology described in any one or more of the other examples. In view of the many possible embodiments to which the principles of the disclosed technology may be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be taken as limitations on the scope of the disclosed technology. Rather, the scope of the disclosed technology includes what is encompassed by the scope and spirit of the following claims. [Explanation of symbols]

[0380] 1 Baggage drop station 1' Baggage drop station 2 Bluetooth (registered trademark) beacon 2' Bluetooth (registered trademark) beacon 3. Mobile phones and devices 4 Trigger Range 4' trigger range 5 Virtual Zones 5' Virtual Zone 6. Mini PC 7 Programmable Logic Controller 10 signals 10' signal 17 distance 20 Bluetooth (registered trademark) beacon 20' Bluetooth® Beacon 100 subsets 102 Label 104 Label 106 Label 108 Label 110 Label 112 Statements 201 Domain 203 Range 301 SPARQL Query 401 Client 403 Software Services 405 User Interface (UI) 407 User Interface (UI) 409 Pattern and Configuration Storage 411 Generating Agent 413 Pattern Maintenance and Access Application Programming Interface (API) 415 Storage 417 Storage 1108 Knowledge Graph 1110 Ontology 1112 Metaontology 1118 class 1118a vehicle 1118b Automobile 1118c Bike 1122 Properties 1122a Property 1122b Property 1122c Properties 1126b instance identifier 1150a Ontology Concept Class 1160 instances 1162 instances 1160a value 1160b value 1162a value 1162b value 1170 relations 1200 processes 1210 operation 1220 operation 1230 Matching Process 1240 Property Check and Validation Actions 1300 Computing Environment 1304 Computing Systems 1308 Client 1312 User Interface 1320 Source Manager 1322 Source Adapter 1322a Source Adapter 1322b Source Adapter 1326 Source Store 1332 Graph Generator 1336 Provisional Triple Store 1344 Matching Executor 1348 Mapping Memory 1360 Triple Store 1400 Source 1420 Source 1500 User Interface 1510 Panel 1512 Control 1514 Control 1516 Control 1520 Field 1522 Currently selected source or input file 1528 Instance Calculation Control 1540 Panel, Instance 1542 instances 1544 Instance Addition Control 1546 Instance Deletion Control 1600 User Interface 1610 Panel 1614 instances 1614a instance 1620 Panel 1624 input (or source) files 1624a Input file 1628 Control 1632 Control 1640 User Interface Controls 1700 User Interface 1710 Panel 1714 instances 1714a instance 1720 Panel 1724 instance graphs 1728 instance nodes 1730 Property Nodes 1732 Literal Value Node 1740 Edge 1742 Edge 1750 Control 1752 Control 1754 Control 1756 Control 1800 Pseudocode 1900 User Interface 1910 Table 1914 columns 1916 columns 1920 columns 1930 Control 1932 Control 1934 Control 1950 Window 1954 Drop-Down List 2010 instance graph 2012 car models 2018 Empty vehicle weight 2020 Engine Type 2022 Number of Doors 2024 Indication 2028 Value 2030 Value 2032 value 2034 value 2050 instance graph 2052 car models (instances) 2070 "recuperation" property 2072 value 2100 Computing Environment 2110 Attribute Information Store 2114 Matching and Insertion Executor 2130 Property Manager 2210 Data Objects 2212a Property 2212b Business Timestamp 2212c System Timestamp 2212d Document Identifier 2220 Data Objects 2222a Property 2222b Status Attributes 2222c Total Count 2222d Count 2222e Start Date 2222f End Date 2222g value 2310 Interval 2314 Date D 2318 Beginning date 2322 Documents 2322a Document, Business Timestamp 2322b System Timestamp 2322c Document ID 2400 Pseudocode 2410 Aggregate Functions 2414 Loop 2430 Functions 2434 Conditional Statements 2436 Conditional Statements 2438 Conditional Statements 2440 Conditional Statements 2442 conditional statements 2500 User Interface 2508 Field 2512 User Interface Elements 2516 User Interface Controls 2518 User Interface Controls 2520 User Interface Controls 2522 User Interface Controls 2526 Table 2528 Properties 2530 Value 2532 Identifier 2536 User Interface Elements 2540 Panel 2542 table 2544 Characteristics Details 2544a Current Status 2544b count 2544c Total Count 2546 Information 2548 Value 2550 Table 2554 Link or Identifier 2556 Date 2558 counts 2560 User Interface Elements 2570 User Interface Elements 2574 User Interface Elements 2600 User Interface 2610 Panel 2614 Table 2616 Properties 2618 Current Status 2620 occurrences 2622 Start Date 2624 User Interface Elements 2630 Panel 2634 Table 2636 Characteristics Details 2636a Current Status 2636b count number 2636c Total Count 2638 Value 2650 Data Type Information 2660 Information 2664 User Interface Elements 2700 User Interface 2708 Panel 2710 Table Column 2712a Column 2712b 2712c column 2712d column 2712e column 2720 ​​User Interface Elements 2730 Panel 2734 Details of characteristics, characteristic information 2738 Data Type Information 2746 Table Column 2748a Column 2748b 2800 User Interface 2824 Table 2826a column 2826b column 2826c column 2840 Table Column 2842a Column 2842b 2850 User Interface Elements 2900 User Interface 2910 Table Column 2914a Column 2914b 2914c column 2914d column 2914e column 2914f row 2950 graphs 3000 User Interface 3010 graph 3014 nodes 3018 Edge 3020 graph 3100 User Interface 3104 nodes 3106 Edge 3110 graph 3130 Interval 3132 Interval 3136 nodes 3138 nodes 3142 nodes 3150 node 3154 nodes 3200 e-newsletter 3210 Link 3300 Email 3310 Link 3400 Data Model 3410 User Elements 3414 Intent Element 3418 Intent Node 3424 Graph Nodes 3432 Restricted Properties 3436 Graph Properties 3444 Priority Element 3448 Update interval 3500 processes 3600 Computing Environment 3610 Client 3614 Subscription Services 3618 User Configuration Repository 3622 listeners 3626 Knowledge Graph Repository 3630 Knowledge Graph 3634 Ontology 3640 Verbalization Component 3700 Computing Environment 3708 Client 3712 Notification System 3716 User Interface 3720 Request Orchestrator 3724 Configuration Storage 3728 Intent Reasoner 3732 Listener Service 3734 listeners 3738 Knowledge Graph System 3740 Knowledge Graph 3744 Change Log Storage 3746 Change Log 3750 Changelog Agent 3754 Natural Language Generator Verbalization Module 3756 Messages 3760 Notification Module 3764 Communicator 3764a Email Communicator 3764b Collaboration Software Communicator 3800 Knowledge Graph 3814 Edge 3810 nodes 3810a~3810c nodes 3900 graphs 3914 Edge 3910 Node 3910a~3910n nodes 4000 User Interface 4010 Panel 4014 Knowledge Graph 4018 nodes 4020 nodes 4026 Edge 4030 Edge 4032 User Interface Elements 4034 Panel 4038 User Interface Elements 4042 User Interface Elements 4046 User Interface Elements 4050 User Interface Elements 4054 User Interface Elements 4058 User Interface Elements 4062 User Interface Elements 4066 User Interface Elements 4070 User Interface Elements 4100 User Interface 4114 Contents 4120 Dialogue Window 4122 Choices 4124 Choices 4200 Log 4210 log entries 4210a~4210e log entries 4220a Date 4220b hours 4220c User Identifier 4220d Action Identifier 4230 Triple 4234 Triple 4238 Triple 4242 Triple 4300 format 4304 Field 4308 Field 4312 Field 4316 Field 4350 Latest Update Message 4400 prompt templates 4404 Command 4408 Changelog Metadata 4412 Change Log Details 4416 Custom Constraints 4420 Closing Order 4500 prompt 4510 Text 4514 Text 4518 Text 4522 Text 4600 Response 4612 Text 4616 Text 4620 Text 4650 Links 4700 processes 4800 Computing System 4810 Processing unit, central processing unit 4815 Processing unit, graphics processing unit or auxiliary processing unit 4820 memory 4825 memory 4830 Basic configuration 4840 Storage 4850 input device 4860 output device 4870 communication connection 4880 software 4900 Cloud Computing Environment 4910 Cloud Computing Services 4920 Computing Devices 4922 Computing Devices 4924 Computing Devices< / triple> < / triple> < / triple> < / triple> < / sequenceid>

Claims

1. 1. A computing system comprising: At least one memory; one or more hardware processing units coupled to the at least one memory; When executed, the computing system: Receiving multiple properties, processing a plurality of electronic documents over a period of time, the processing including incrementing a respective count of each property of the plurality of properties if the document being processed includes the respective property; For at least one property of the plurality of properties, determining a total count of the at least one property over at least a portion of the time period; comparing said total count to a threshold; determining that the total count meets the threshold; changing the status of the at least one property based at least in part on a determination that the total count meets the threshold to provide an updated property status; and and one or more computer-readable storage media storing computer-executable instructions for performing operations including processing at least another electronic document using the updated property status.

2. The computing system of claim 1 , wherein changing the status of the at least one property comprises changing the status of the at least one property from an inactive status to an active status.

3. The operation is The computing system of claim 2 , further comprising adding the at least one property to an ontology.

4. The computing system of claim 3 , wherein the plurality of properties are defined in terms of a knowledge graph.

5. The operation is receiving a query for the knowledge graph; processing the query using the ontology; and 5. The computing system of claim 4, further comprising: returning a query response, the query response including a property value for the at least one property of the knowledge graph, the property value being determined using the ontology.

6. changing the status of the at least one property from the inactive status to the active status; providing a recommendation to a user on a user interface to change the at least one property to the active status; and receiving user input through the user interface to change the status of the at least one property to the active status.

7. 3. The computing system of claim 2, wherein the threshold is a first threshold, and when a second threshold requires a total count greater than the first threshold, in response to determining that the total count meets the second threshold, the status of the at least one property is automatically changed from the inactive status to the active status.

8. 3. The computing system of claim 2, wherein processing at least another electronic document using the updated property status includes storing a value of the at least one property, where the value of the at least one property of the plurality of electronic documents was not stored.

9. The computing system of claim 1 , wherein changing the status of the at least one property comprises changing the status of the at least one property from an active status to an inactive status.

10. 10. The computing system of claim 9, wherein processing at least another electronic document includes not storing a value of the at least one property.

11. The properties are defined in terms of a knowledge graph, and the actions include: receiving a query for the knowledge graph, the query specifying that a property having the inactive status is responsive to the query; processing the query using an ontology; 10. The computing system of claim 9, further comprising: returning a query response, the query response including a property value for the at least one property of the knowledge graph, the property value being determined using the ontology.

12. The properties are defined in terms of a knowledge graph, and the actions include: receiving a query for the knowledge graph, the query not specifying that a property having the inactive status is responsive to the query; processing the query using an ontology; 10. The computing system of claim 9, further comprising: returning a query response, wherein the query response does not include a property value for the at least one property of the knowledge graph.

13. The computing system of claim 1 , wherein a property of the plurality of properties having an active status is included in an ontology.

14. The operation is 14. The computing system of claim 13, further comprising: rendering a display of at least a portion of the ontology, the display providing an indication of a degree of use of each property of the plurality of properties of the at least a portion of the ontology.

15. changing the status of the at least one property includes changing the status from an inactive status to an active status, and the action During processing of the plurality of electronic documents, when each electronic document of the plurality of electronic documents includes the at least one property, storing an association between the each electronic document and the at least one property; 10. The computing system of claim 1, further comprising: in response to the change of the status of the at least one property to the active status, reprocessing at least a portion of the plurality of electronic documents that include the at least one property, the reprocessing including storing a value of each of the at least one property in association with a corresponding respective electronic document of the at least a portion of the plurality of electronic documents.

16. The operation is changing the status of the at least one property to a recommended status in response to determining that the total count meets the threshold; providing an indication of said recommendation status to a user; 10. The computing system of claim 1, further comprising: receiving a user input for changing the status of the at least one property to the recommended status, wherein changing the status of the at least one property is performed in response to the user input.

17. The total count is determined for a plurality of clients, and the operation comprises: The computing system of claim 1 , further comprising: proposing a status change to a client of the plurality of clients based at least in part on the total count.

18. 10. The computing system of claim 1, wherein changing the status of the at least one property is based at least in part on a change in the value of the at least one property during the period compared to a value of the at least one property during a previous period.

19. 1. A method implemented in a computing system including at least one hardware processor and at least one memory coupled to the at least one hardware processor, comprising: receiving a plurality of properties; processing a plurality of electronic documents over a period of time, the step including incrementing a respective count of each said respective property of said plurality of properties if the document being processed includes the respective property; For at least one property of the plurality of properties, determining a total count of the at least one property over at least a portion of the time period; comparing the total count to a threshold; determining that the total count meets the threshold; changing the status of the at least one property based at least in part on a determination that the total count meets the threshold to provide an updated property status; and processing at least another electronic document using the updated property status.

20. computer-executable instructions that, when executed by a computing system including at least one hardware processor and at least one memory coupled to the at least one hardware processor, cause the computing system to receive a plurality of properties; computer-executable instructions that, when executed by the computing system, cause the computing system to process a plurality of electronic documents over a period of time, including incrementing a respective count of each property of the plurality of properties if the document being processed includes the respective property; computer-executable instructions that, when executed by the computing system, cause the computing system to determine, for at least one property of the plurality of properties, a total count of the at least one property over at least a portion of the time period; computer-executable instructions that, when executed by the computing system, cause the computing system to compare the total count with a threshold value; computer-executable instructions that, when executed by the computing system, cause the computing system to determine that the total count meets the threshold; computer-executable instructions that, when executed by the computing system, cause the computing system to change a status of the at least one property based at least in part on a determination that the total count meets the threshold to provide an updated property status; and one or more non-transitory computer-readable storage media comprising computer-executable instructions that, when executed by the computing system, cause the computing system to process at least another electronic document using the updated property status.

Images