Translating natural language queries into formal data queries
The method converts natural language queries to formal data queries using a knowledge graph, allowing user interaction and feedback to generate ground truth, addressing the inefficiencies in existing systems and enhancing query accuracy.
Patent Information
- Application Number
- JP2021569840
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-06-20
- Filing Date
- 2020-06-16
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-06-16
AI Technical Summary
Existing natural language querying systems require detailed knowledge of formal data query languages and lack efficient methods for converting natural language queries into formal data queries, necessitating the need for ground truth training data.
A computer-implemented method and system that converts natural language queries into formal data queries using a knowledge graph, allowing users to visually inspect and edit the queries, generating ground truth pairs through user feedback.
Enables efficient generation of ground truth for natural language queries, improving the accuracy of converting natural language to formal data queries and enhancing the response to future queries through machine learning.
Smart Images

Figure 0007584449000001 
Figure 0007584449000002 
Figure 0007584449000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a computer-implemented method for generating ground truth.
[0002] The present disclosure further relates to corresponding systems, corresponding computer program products, and corresponding user interfaces. [Background technology]
[0003] Natural language querying (NLQ) is the scientific field that studies how machines can be queried using natural language. In general, performing a natural language query requires a data model, a formal data query language for that data model, and a way to translate natural language queries into the formal data query language. The formal data query language may also be referred to as a machine data query language.
[0004] The most common data structure used to represent knowledge is the knowledge graph. A knowledge graph contains nodes that represent entities and links between nodes, which may represent facts or relationships.
[0005] To query these knowledge graphs, several different types of formal data query languages have been designed over the past few decades. Examples of such query languages include SPARQL, Cypher (neo4j), and Gremlin (Apache Tinkerpop®).
[0006] These query languages have in common that queries usually must be constructed from a limited set of low-level operation primitives. Defining such queries usually requires detailed knowledge of the query language and its syntax, thus narrowing its scope.
[0007] Another approach is to use machine learning techniques to start translating natural language queries into a formal or machine data query language. Such approaches require ground truth as training data.
[0008] Therefore, there is a need in the art to address the aforementioned problems. Summary of the Invention
[0009] Viewed from a first aspect, the present invention provides a computer-implemented method for generating ground truth for natural language querying, the method comprising: providing a knowledge graph as a data model; receiving a natural language query from a user; converting the natural language query into a formal data query; visualizing the formal data query to the user; receiving a feedback response from the user including the verified and / or edited formal data query; and storing the natural language query and the corresponding feedback response as a ground truth pair.
[0010] Viewed from a further aspect, the present invention provides a computer system for performing a computer-implemented method for generating ground truth for natural language querying, the system comprising a processing unit, a storage system, and a user interface comprising an input unit and a display, the system being configured to: store a knowledge graph as a data model in the storage system, receive via the user interface a natural language query, convert the natural language query into a formal data query, visualize the formal data query on the display, receive via the input unit a feedback response from a user comprising the confirmed and / or edited formal data query, and store the natural language query and the corresponding feedback response as a ground truth pair in the storage system.
[0011] Viewed from a further aspect, the present invention provides a computer program product for executing a computer implemented method for generating ground truth for natural language querying, the computer program product comprising a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing the method for performing steps of the present invention.
[0012] Viewed from a further aspect, the present invention provides a computer program stored on a computer readable medium and loadable into an internal memory of a digital computer, the computer program comprising software code portions for performing the steps of the present invention when the program is run on the computer.
[0013] Viewed from a further aspect, the present invention provides a user interface comprising a display and an input unit, the user interface configured to: receive a natural language query from a user; provide the natural language query to a conversion module; receive a formal data query from the conversion module, the formal data query being a conversion of the natural language query into a formal language; visualize the formal data query on a display; receive a feedback response from the user including the confirmed and / or edited formal data query; and provide the natural language query and the corresponding feedback response as a ground truth pair.
[0014] According to a first aspect, the invention is embodied as a computer-implemented method for generating ground truth for natural language querying. The method comprises the steps of providing a knowledge graph as a data model, receiving a natural language query from a user, and converting the natural language query into a formal data query. The method comprises the further step of visualizing the formal data query to the user, and receiving a feedback response from the user. The feedback response comprises the confirmed and / or edited formal data query. The method comprises the further step of storing the natural language query and the corresponding feedback response as a ground truth pair.
[0015] Such a method according to an embodiment of the present invention may be used to generate a ground truth of a natural language query for a knowledge graph in an efficient manner. The method according to an embodiment of the present invention uses a visual programming approach to generate a ground truth of a natural language query. More specifically, a formal data query converted from a natural language query is provided to a user in a visual form. The ground truth is provided in a pair including a natural language query input and a corresponding feedback response. The feedback response may be, on the one hand, a confirmed formal data query. Such a confirmed formal data query indicates an acknowledgement by the respective user that the visualized formal data query is correct, in other words corresponds to his / her intention or understanding of the natural language query. On the other hand, the feedback response may be an edited formal data query, in particular a corrected formal data query. In such an edited formal data query, the user may make edits, modifications, or corrections, or a combination thereof, to the visualized formal data query, in case the visualized formal data query is incorrect or does not correspond to his / her intention or understanding of the natural language query.
[0016] According to the embodiment, a formal data query shall be understood as a data query having a format or form of a formal or machine data query language. For this reason, a formal data query may also be referred to as a machine data query. A query language or data query language (DQL) may be generally defined as a computer language that can be used to execute queries in databases and information systems. In contrast to natural language queries, they are designed to convey instructions to databases and information systems and are usually composed of a set of operators or operation primitives.
[0017] According to one embodiment of the present invention, converting the natural language query into a formal data query includes providing the natural language query to a workflow model and computing a workflow graph as the formal data query by the workflow model.
[0018] Such a workflow model receives a natural language query as input and provides a workflow graph as output. The workflow graph establishes a visual representation of the workflow used by the workflow model to respectively transform a natural language query into a formal data query. The workflow graph may be specifically embodied as a directed acyclic graph. The workflow graph may include a number of operators, i.e. operation primitives, including input operators, i.e. those that may specifically perform a search for input nodes of a certain type, edge traversal operators, node filtering operators, node ranking operators, logical operators such as "AND" and "OR", and output operators.
[0019] According to one embodiment of the present invention, ground truth pairs may be used in a machine learning algorithm to train a workflow model, which not only improves the workflow model but also the corresponding translation and response of natural language queries in future inference phases of the trained workflow model.
[0020] According to an embodiment, the formal data query may be visualized in a visual programming tool. The visual programming tool may be configured to provide a test function for testing the natural language query. In other words, a user may input a natural language query and the methods provide a corresponding output, i.e., a result of the natural language query, along with a visual representation of the formal data query.
[0021] Further, according to an embodiment, the visual programming tool may be configured to provide editing functionality for visually editing the formal data query, such editing functionality providing editing tools that can be used to make edits, corrections, modifications, and / or amendments to the visualized formal data query.
[0022] According to an embodiment, converting the natural language query to a formal data query may include identifying one or more input node types of a knowledge graph in the natural language query and identifying one or more output node types of the knowledge graph in the natural language query. According to an embodiment, the method may include computing one or more search paths through the knowledge graph between the one or more input node types and the one or more output node types. According to an embodiment, computing the one or more search paths may be performed by a breadth-first search algorithm.
[0023] According to an embodiment, computing the one or more search paths includes providing a natural language query as a string input to a natural language classifier and computing, by the natural language classifier, one or more input node types and one or more output node types. Using such a natural language classifier is an efficient and reliable approach.
[0024] According to another aspect, the invention is embodied as a computer system for executing a computer-implemented method for generating ground truth for natural language querying. The system comprises a processing unit, a storage system, and a user interface including an input unit and a display. The system is configured to store the knowledge graph as a data model in the storage system, receive a natural language query via the user interface, and convert the natural language query into a formal data query. The system is further configured to visualize the formal data query on the display and receive a feedback response from the user via the input unit. The feedback response includes the verified and / or edited formal data query. The system is further configured to store the natural language query and the corresponding feedback response as a ground truth pair in the storage system.
[0025] According to yet another aspect of the present invention, there is provided a computer program product for executing on a computing system a computer-implemented method for generating ground truth for natural language querying, the computer program product including a computer-readable storage medium embodying program instructions executable by the system to cause the system to perform a method including providing a knowledge graph as a data model, receiving a natural language query from a user, converting the natural language query into a formal data query, visualizing the formal data query to the user, receiving a feedback response from the user, and providing the natural language query and the corresponding feedback response as a ground truth pair.
[0026] According to yet another aspect of the present invention, there is provided a user interface comprising a display and an input unit. The user interface is configured to receive a natural language query from a user, provide the natural language query to a conversion module, and receive a formal data query from the conversion module. The formal data query is a conversion of the natural language query into a formal language. The user interface is further configured to visualize the formal data query on the display, receive a feedback response from the user, and provide the natural language query and the corresponding feedback response as a ground truth pair.
[0027] Embodiments of the invention will now be described in more detail, by way of illustrative and non-limiting example, with reference to the accompanying drawings, in which:
[0028] The invention will now be described, by way of example only, with reference to preferred embodiments illustrated in the following figures. [Brief description of the drawings]
[0029] [Figure 1] 1 is a schematic block diagram of a computer system according to one embodiment of the present invention. [Diagram 2] FIG. 1 is a functional block diagram illustrating functional blocks of a computer system for executing a computer-implemented method for generating ground truth for natural language querying. [Diagram 3] 1 is a flowchart of method steps of a computer-implemented method for generating ground truth for natural language querying according to an embodiment of the present invention. [Figure 4A] FIG. 1 illustrates an exemplary knowledge graph used as a data model for natural language queries. [Figure 4B] FIG. 2 illustrates an example of a natural language query. [Figure 4C] FIG. 4B illustrates the use of the knowledge graph of FIG. 4A to convert a natural language query into a formal data query. [Figure 4D]FIG. 1 shows a screen layout including a workflow graph. [Figure 4E] FIG. 13 shows the screen layout including the edited workflow graph. [Figure 5A] FIG. 13 illustrates another exemplary knowledge graph. [Figure 5B] FIG. 2 illustrates an example of a natural language query. [Figure 5C] FIG. 5B illustrates the use of the knowledge graph of FIG. 5A to convert a natural language query into a formal data query. [Figure 5D] FIG. 1 shows a screen layout including a workflow graph. [Figure 5E] FIG. 13 shows the screen layout including the edited workflow graph. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0030] With reference to Figures 1-5E, some general aspects and terminology of embodiments of the present invention will be described.
[0031] Embodiments of the present invention provide a computer-implemented method for generating ground truth for natural language querying in an efficient manner.
[0032] The term ground truth may generally refer to information provided by direct observation (i.e., empirical evidence) as opposed to information provided by inference.
[0033] A graph according to an embodiment of the invention is a knowledge representation system that includes nodes and edges between the nodes. Thus, the graph may be embodied as a knowledge graph. The nodes may have different node types. The nodes may hold information about information items. The edges indicate particular relationships between the nodes.
[0034] More specifically, an instantiation of a graph or knowledge graph KG is a set of triplets KG{V,E}, where set V contains some nodes, which have a type in a set of allowed types. Set E contains edges in an edge type list that link pairs of nodes in set V. The underlying mathematical structure of a KG is a directed or undirected graph {V,E}, where node and edge types can be represented with a numeric weighting scheme.
[0035] Knowledge graphs are by far the most popular data model because they do not require any type of data-scheme and are very flexible in terms of changes, updates, etc.
[0036] Embodiments of the present invention enable users to perform natural language queries with enhanced capabilities. More specifically, users may not only view responses to each natural language query, but also view, inspect, and manually edit the internal workflows on the knowledge graph that led to those responses. This generates ground truth, which can be used to improve responses to future queries.
[0037] Referring now to FIG. 1, a block diagram of a computer system 100 is shown. The computer system 100 may be configured to execute a computer-implemented method for generating ground truth for natural language querying. The computer system 100 may be operable with many other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with the computer system 100 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.
[0038] Computer system 100 may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system 100 is illustrated in the form of a general-purpose computing device. Components of computer system 100 may include, but are not limited to, one or more processors or processing units 116, a system memory 128, and a bus 118 that couples various system components, including the system memory 128, to the processor 116.
[0039] Bus 118 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, including, by way of example and not limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0040] Computer system 100 typically includes a variety of computer system readable media. Such media may be any available media that can be accessed by computer system 100 and includes both volatile and nonvolatile media, removable and non-removable media.
[0041] The system memory 128 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 130 and / or cache memory 132. The computer system 100 may further include other removable / non-removable volatile / non-volatile computer system storage media. By way of example only, a storage system 134 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such an example, each may be connected to the bus 118 by one or more data media interfaces. As further illustrated and described below, the memory 128 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of an embodiment of the present invention.
[0042] A program / utility 140 having a set (at least one) of program modules 142 may be stored in memory 128, as well as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. The program modules 142 may include a conversion module 142a and a training module 142b. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a network environment. The program modules 142 generally perform the functions and / or methodologies of the embodiments of the present invention described herein. The program modules 142 may specifically perform one or more steps of a computer-implemented method for generating ground truth for natural language querying, such as one or more steps of the method described below.
[0043] The computer system 100 may also communicate with one or more external devices 115, such as an input unit 125, a pointing device, a display 124, one or more devices that allow a user to interact with the computer system 100, or any device that allows the computer system 100 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), or a combination thereof. Such communication may occur via an input / output (I / O) interface 122. Still further, the computer system 100 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 120. As shown, the network adapter 120 communicates with other components of the computer system 100 via a bus 118. It is understood that other hardware and / or software components, not shown, may be used with the computer system 100. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems, etc. Input units 125, such as a keyboard and display 124, establish a user interface 126.
[0044] FIG. 2 shows a functional block diagram illustrating functional blocks of a computer system 200 for executing a computer-implemented method for generating ground truth for natural language querying.
[0045] The system 200 is configured to receive a natural language query 210 as input, such as from a user via an input unit, such as the input unit 125 shown in FIG.
[0046] The natural language query 210 is then provided as an input to a workflow model 220. The workflow model 220 converts the natural language query 210 into a formal data query. More specifically, it computes a workflow graph 230 as the formal data query. The workflow graph is then displayed to a user, who may provide an edited workflow graph 240 as a feedback response. The natural language query 210 and the corresponding edited workflow graph 240 together form a ground truth pair 250. The ground truth pair 250 may then be used by a machine learning algorithm 260 to update and improve the workflow model 220.
[0047] Figure 3 shows a flow chart of method steps of a computer-implemented method for generating ground truth for natural language querying according to an embodiment of the present invention. The method may be performed, for example, by the system 100 of Figure 1, and therefore will be described below with reference to components of the system of Figure 1. The method may be specifically performed under control of the transformation module 142a running on the system 100.
[0048] In step 310, a knowledge graph is provided as a data model. The knowledge graph may be stored, for example, in the storage system 134 of the computing system 100.
[0049] In step 320, a natural language query (NLQ) is received from a user of the computing system 100, such as via the input unit 125.
[0050] In step 330, one or more input node types and one or more output node types are identified by the computer in the received natural language query. This may be specifically performed by providing the natural language query as a string input to a natural language classifier. The natural language classifier may then calculate one or more input node types and one or more output node types. The natural language classifier may use, for example, a string classification method, or a sequence to sequence model, or both.
[0051] In step 340, the identified input and output nodes are provided to a workflow model.
[0052] In step 350, the workflow model computes a workflow graph. The workflow graph establishes a formalism for the natural language query or a machine data query.
[0053] Thus, steps 330-350 perform a translation from a received natural language query to a formal data query that is embodied as a workflow graph.
[0054] In step 360 , the workflow graph is made visible to a user, for example, by displaying it on the display 124 of the computing system 100 .
[0055] The user can then visually inspect and check the displayed workflow graph, specifically, whether the workflow graph and the results correspond to his or her desired meaning and intent of the natural language query.
[0056] Then, in step 370, the user may edit the workflow graph and thereby provide feedback in the form of a feedback response to the system 100, such as via the input unit 125. The feedback response may specifically include the edited workflow graph.
[0057] In step 380, the system 100 may store the feedback responses, specifically the compilation of the workflow graph and the corresponding natural language queries, as ground truth pairs. The ground truth pairs may be stored, for example, in the storage system 134 of the computing system 100.
[0058] The ground truth pairs may then be used in step 390 to train or retrain a workflow model to improve the response of the system 100 to future natural language queries.
[0059] 4A-4E illustrate an example of a computer-implemented method for generating ground truth for natural language querying according to an embodiment of the present invention.
[0060] FIG. 4A illustrates a knowledge graph 401 used as a data model for natural language queries. According to the illustrated example, the knowledge graph 401 includes four nodes 1, 2, 3, and 4, an edge 12 between nodes 1 and 2, an edge 14 between nodes 1 and 4, an edge 13 between nodes 1 and 3, and an edge 23 between nodes 2 and 3. The nodes and edges are divided into types, i.e., the knowledge graph 401 has a set of nodes and edges of a particular type. More specifically, FIG. 4A represents a "citation graph" and includes nodes of types paper, author, topic, and affiliation. Also, the edges of this graph have a particular type, i.e., "paper-paper", "paper-author", "paper-topic", and "author-affiliation". Thus, the knowledge graph 401 is embodied as a metagraph that includes node types of paper, author, topic, and affiliation, and edge types of “paper-paper”, “paper-author”, “paper-topic”, “author-topic”, and “author-affiliation”.
[0061] According to an embodiment, such a metagraph may be referred to as the topology of the original knowledge graph, where the original knowledge graph is the knowledge graph from which the metagraph is derived. The topology of the original knowledge graph, and therefore the metagraph, does not change unless new types of nodes are inserted into the knowledge graph. More documents can be inserted into the knowledge graph without changing the topology of the metagraph, i.e., the topology of the citation graph in this example.
[0062] If each of the nodes and edges of a knowledge graph represents a particular concept or type, then in general almost any knowledge graph can be described by such a topology or metagraph.
[0063] Figure 4B shows an example of a natural language query. In this example, the natural language query is written as "Who are the experts in the field of artificial intelligence (AI) at Company A?"
[0064] FIG. 4C illustrates the use of the metagraph of FIG. 4A to convert a natural language query 402 into a formal data query. The method identifies node 2 (topic) and node 4 (affiliation) as input nodes in the natural language query 402 and node 1 (author) as an output node type of the knowledge graph 401 in the natural language query 402. The identified input node types are shown in a wave pattern and the identified output node types are shown in a checkerboard pattern. The input node types and output node types may be identified by providing the natural language query 402 as a string input to a natural language classifier. The natural language classifier may then compute the input node types and output node types from the natural language query string 402.
[0065] Additionally, computing system 100 computes search paths between input node types and output node types. In this example, search path 410 is computed between node 2 of type "topic" and node 1 of type "author", and search path 411 is computed between node 4 of type "affiliation" and node 1 of type "author".
[0066] FIG. 4D illustrates a screen layout 403 that may be displayed, such as by the display 124 of FIG. 1. The screen layout 403 includes a workflow graph 420 embodied as a directed acyclic graph. The workflow graph 420 represents a formal data query that has been transformed, or in other words, derived, from the natural language query 402 by a transformation module, such as the transformation module 142a of FIG. 1. The transformation module uses an underlying workflow model to compute the workflow graph 420. The workflow graph 420 may include a number of operators or operation primitives. Such operators may include input operators, and the like, which may specifically perform a search for input nodes of a particular type. Further operators include edge traversal operators that may perform edge traversal between two different types of nodes, node filtering operators that filter nodes according to one or more criteria, and node ranking operators that rank nodes according to one or more predefined criteria. As further examples, logical operators such as "AND" and "OR" may be used, as well as output operators.
[0067] The workflow graph 420 includes an input operator 431, which is an “input belonging” IN A and performs a search for "Company A" in all input type nodes with type "Affiliation". The workflow graph 420 includes a further input operator 432, which provides a "topic input" IN T and performs a search for “artificial intelligence” in all input type nodes with type “topic”. The workflow graph 420 includes an edge traversal operator 433 that performs an edge traversal E AAThe workflow graph 420 includes a further edge traversal operator 434 that performs an edge traversal ET between the identified nodes of type “topic” that contain the phrase “artificial intelligence” and the corresponding nodes of type “author”. TA Execute "topic-author". The outputs of edge traversal operator 433 and edge traversal operator 434 are then merged by "AND" operator 435, i.e. "AND" operator 435 finds the intersection of the outputs of edge traversal operator 433 and edge traversal operator 434. Output operator 435 then generates the results 440 of the natural language query 402 and displays it on the screen layout 403. In this example, the results 440 include a list of authors "Author1, Author2, Author3". According to this example, it is assumed that the workflow model displays the results in the form of an alphabetical list of authors.
[0068] The visual programming tool provides an editing feature for visually editing the formal data query. In FIG. 4D, the tool is in edit mode, which is indicated by illustrating the word “edit” in bold, which represents the edit button 450. This means that the user may edit the workflow graph 420, specifically, by deleting, modifying, or adding one or more operators of the workflow graph 420.
[0069] FIG. 4E shows the screen layout 404 with an edited workflow graph 421. The workflow graph 421 represents an edited formal data query, which has been edited by the user after the user has inspected the original workflow graph 420. Compared to the original workflow graph 420, the user has inserted a ranking operator 437. The ranking operator 437 can be, for example, an operator that ranks the results (authors) according to their citation count. Thus, the user may not have been entirely satisfied with the results 440, which include alphabetical results, but would have preferred a ranking according to their citation count. The visual programming tool provides a test function for testing the natural language query. Accordingly, the user may then test the edited workflow graph 421 by activating the search button 451. Once activated, the screen layout displays the modified results 441, which include the authors "Author 3, Author 2, Author 1" sorted by citation count.
[0070] The computing system 100 may then store the natural language query 402 along with the edited workflow graph 421 (the formal data query) as a ground truth pair in the storage system 134. This ground truth pair, preferably in combination with multiple additional ground truth pairs, may then be used by the training module 142b to train the transformation module 142a. The computing system 100 may then provide a result list of authors, etc., in a ranked manner the next time it receives a similar query.
[0071] Figures 5A-5E show an example of a computer-implemented method for generating ground truth for natural language querying according to another embodiment of the present invention. The example of Figures 5A-5E corresponds in part to the example of Figures 4A-4E, and therefore identical or similar elements are designated with the same reference numerals.
[0072] FIG. 5A shows a knowledge graph 501 used as another data model for natural language queries. According to the illustrated example, the knowledge graph 501 includes four nodes 1, 2, 3, and 5, an edge 12 between nodes 1 and 2, an edge 13 between nodes 1 and 3, an edge 23 between nodes 2 and 3, and, unlike FIG. 4A, an edge 35 between nodes 3 and 5. The nodes and edges are divided into types, i.e., the knowledge graph 501 has a set of nodes and edges of a specific type. More specifically, FIG. 5A also represents a "citation graph" and includes nodes of types "paper", "author", "topic", and "affiliation" as in the example of FIG. 4A. Also, the edges of this graph include types "paper-paper", "paper-author", "paper-topic", and "author-topic", but instead of "author-affiliation", it includes edge 35 "paper-affiliation".
[0073] Therefore, the knowledge graph 501 is also embodied as a metagraph that represents the topology of the original graph.
[0074] 5B illustrates an example of a natural language query 502. This example uses the same natural language query as the previous example, namely, "Who are Company A's experts in the field of artificial intelligence (AI)?"
[0075] FIG. 5C illustrates the use of the metagraph of FIG. 5A to convert a natural language query 502 into a formal data query. The method identifies node 2 (topic) and node 5 (affiliation) as input nodes in the natural language query 502 and node 1 (author) as an output node type. The identified input node types are shown in a scalloped pattern and the identified output node types are shown in a checkered pattern. The computing system 100 computes search paths between the input node types and the output node types. In this example, a search path 510 is computed between node 2 of type “topic” and node 1 of type “author”, and a search path 511 is computed between node 5 of type “affiliation” and node 1 of type “author” via node 3 of type “article”.
[0076] FIG. 5D illustrates a screen layout 503 that may be displayed, such as by the display 124 of FIG. 1. The screen layout 503 includes a workflow graph 520 embodied as a directed acyclic graph. The workflow graph 520 represents a formal data query that has been transformed, or in other words derived, from the natural language query 502 by a transformation module, such as the transformation module 142a of FIG. 1. The transformation module uses an underlying workflow model to compute the workflow graph 520. The workflow graph 520 includes an input operator 531 that performs a search for "Company A" in all input type nodes with type "Affiliation". The workflow graph 520 includes a further input operator 532 that performs a search for "Artificial Intelligence" in all input type nodes with type "Topic". The workflow graph 520 includes an edge traversal operator 533, which performs an edge traversal "affiliation-paper" between the identified node of type "affiliation" containing the phrase "Company A" and a corresponding node of type "paper". The workflow graph 520 includes a further edge traversal operator 534, which performs an edge traversal "topic-paper" between the identified node of type "topic" containing the phrase "artificial intelligence" and a corresponding node of type "paper". The outputs of the edge traversal operator 533 and the edge traversal operator 534 are then merged by an "AND" operator 535, i.e., the "AND" operator 535 finds the intersection of the outputs of the edge traversal operator 533 and the edge traversal operator 534. In this example, an additional edge traversal operator 536 is provided, which performs an edge traversal ET AAExecute "articles-authors". The output operator 537 then generates the results 540 of the natural language query 502 and displays them on the screen layout 503. In this example, the results 540 include a list of authors: "Author 1, Author 2, Author 3". According to this example, it is assumed that the workflow model displays the results in the form of an alphabetical list of authors.
[0077] 5D, the visual programming tool is also in edit mode, which is indicated by illustrating the words "edit mode" in bold on the edit button / field 550. This means that the user may edit the workflow graph 520.
[0078] FIG. 5E shows the screen layout 504 with the edited workflow graph 521. The workflow graph 521 represents an edited formal data query, which is edited by the user after the user inspects the original workflow graph 520. Compared to the original workflow graph 520, the user has inserted a ranking operator 538. The ranking operator 538 is configured to rank the results (authors) according to their citation counts. The visual programming tool provides a test function for testing the natural language query. Accordingly, the user may then test the edited workflow graph 521 by activating the search button 551. Once activated, the screen layout 504 displays the modified results 541, which include the authors “Author 3, Author 2, Author 1” sorted by citation counts.
[0079] The computer system 100 may then store the natural language query 502 along with the workflow graph 521 (the formal data query) as a ground truth pair in the storage system 134. This ground truth pair, preferably in combination with multiple additional ground truth pairs, may then be used by the training module 142b to train the transformation module 142a.
[0080] The present invention may be a system, method, and / or computer program product, which may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor / processing unit of computer system 100 to carry out aspects of the present invention.
[0081] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or groove ridge structures having instructions recorded thereon, and any suitable combination thereof. A computer-readable storage medium, as used herein, should not be construed to be a transitory signal per se, such as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over a wire.
[0082] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device over a network, such as, for example, the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.
[0083] The computer readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object oriented programming languages such as Smalltalk®, C++, and traditional procedural programming languages such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or a connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.
[0084] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.
[0085] These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored constitutes an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.
[0086] Furthermore, the computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to create a computer-implemented process and cause the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.
[0087] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of instructions that includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order shown. For example, depending on the functionality involved, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may possibly be executed in the reverse order. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.
[0088] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many changes and modifications will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.
Claims
1. 1. A computer-implemented method for generating ground truth for natural language querying, comprising: storing the knowledge graph as a data model; storing the workflow model; receiving a natural language query from a user; converting the natural language query into a formal data query; visualizing the formal data query to the user on a display; receiving a feedback response from the user, the feedback response including the verified and / or edited formal data query; storing the natural language query and the corresponding formal data query included in the feedback response as a ground truth pair; Including, Transforming the natural language query into a formal data query includes: identifying one or more input node types and one or more output node types of the knowledge graph in the natural language query by providing the natural language query as a string input to a natural language classifier; the one or more identified input node types and the one or more providing an output node type of the workflow model; Computing one or more search paths between nodes of the one or more input node types and nodes of the one or more output node types of the knowledge graph using the workflow model to compute a workflow graph as the formal data query; Including, The method comprises: Using the ground truth pairs in a machine learning algorithm to train the workflow model. The method further comprising:
2. The method of claim 1 , wherein the workflow graph is a directed acyclic graph.
3. The method of claim 1 , wherein the workflow graph comprises a plurality of operators.
4. The plurality of operators are selected from a group, the group comprising: Input operators, Edge traversal operators, Node filtering operators, Node ranking operator, Logical operators, and Output operators, The method of claim 3, comprising:
5. The method of claim 1 , wherein computing the one or more search paths comprises performing a breadth-first search algorithm.
6. Visualizing the formal data query includes: visualizing said formal data query in a visual programming tool; The method of any one of claims 1 to 5, comprising:
7. The method of claim 6 , wherein the visual programming tool is configured to provide a test function for testing the natural language query.
8. The method of claim 6 , wherein the visual programming tool is configured to provide editing functionality for visually editing the formal data query.
9. A computer system for generating ground truth for natural language querying, the computer system comprising a processing unit, a storage system, and a user interface comprising an input unit and a display, the computer system comprising: storing a knowledge graph as a data model in the storage system; storing a workflow model in the storage system; receiving a natural language query via the user interface; converting the natural language query into a formal data query; visualizing the formal data query on the display; receiving a feedback response from a user via said input unit, said feedback response including the confirmed and / or edited formal data query; storing the natural language query and the corresponding formal data query included in the feedback response as a ground truth pair in the storage system; configured to: The computer system comprises: a transformation module configured to transform the natural language query into the formal data query, the transformation module identifying one or more input node types and one or more output node types of the knowledge graph in the natural language query by providing the natural language query as a string input to a natural language classifier, providing the identified one or more input node types and the one or more output node types to the workflow model, and computing, by the workflow model, one or more search paths between nodes of the one or more input node types and nodes of the one or more output node types of the knowledge graph to compute a workflow graph as the formal data query; A training module; The training module further comprises: receiving the ground truth pair; performing training of said workflow model using a machine learning algorithm; updating the workflow model; 16. A computer system configured to:
10. 10. The computer system of claim 9, wherein the workflow graph is a directed acyclic graph.
11. A computer program stored on a computer-readable recording medium and loadable into an internal memory of a computer, the computer program causing the computer to execute a method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Data expansion inquiry method, electronic device and computer-readable storage medium
CN109241103A
Access control for data resource
JP2019091474A
Natural language query interface, systems, and methods for a database
US20080235199A1
Neural network based translation of natural language queries to database queries
WO2018213530A2
