Resolving queries using structured and unstructured data

Abstract

Resolving queries using structured and unstructured data. [Solution] A computer-implemented method includes receiving a question in a natural language format; deriving a plurality of subqueries from the question; determining that a first subquery of the plurality of subqueries cannot be resolved using a curated knowledge base; resolving the first subquery using unstructured text; and adapting the curated knowledge base using the resolved first subquery.

Description

[Technical field]

[0001] The present invention relates generally to data analysis, and more specifically to resolving queries using a combination of indexed unstructured data and a curated knowledge base. [Background technology]

[0002] Knowledge Bases (KBs) are frequently used in applications such as data-centric question answering (QA) applications. Such knowledge bases are generated using curated unstructured text and provide accurate answers based on highly accurate knowledge. However, while knowledge bases provide accurate results for a specific subset of information, they cannot provide answers for queries outside of that specific subset. Summary of the Invention

[0003] In one aspect, a computer-implemented method includes receiving a question in a natural language format, deriving a plurality of sub-queries from the question, determining that a first sub-query of the plurality of sub-queries cannot be resolved using the curated knowledge base, resolving the first sub-query using unstructured text, and adapting the curated knowledge base using the resolved first sub-query.

[0004] According to another feature, a computer program product for resolving queries using structured and unstructured data includes a computer-readable medium having program instructions embodied therein, the computer-readable medium being not itself a transitional signal, the program instructions being executable by a processor to cause the processor to receive, by the processor, a query in a natural language format; deriving, by the processor, a plurality of sub-queries from the query; determining, by the processor, that a first sub-query of the plurality of sub-queries cannot be resolved using the curated knowledge base; resolving, by the processor, the first sub-query using unstructured text; and adapting, by the processor, the curated knowledge base using the resolved first sub-query.

[0005] According to another feature, the system includes a processor and logic logically integrated with the processor and executable by or integrated with the processor, the logic configured to receive a question in a natural language format, derive a plurality of sub-queries from the question, determine that a first sub-query of the plurality of sub-queries cannot be resolved using the curated knowledge base, resolve the first sub-query using the unstructured text, and adapt the curated knowledge base using the resolved first sub-query.

[0006] According to another feature, a computer-implemented method includes receiving a question in a natural language format, deriving a first subquery and a second subquery from the question, determining that a first subquery of the multiple subqueries cannot be resolved using the curated knowledge base, generating additional search terms for the first subquery using query relaxation, querying a search index for unstructured text using the first subquery and the additional search terms, receiving search results in response to querying the search index, generating new subject / predicate / object triples based on the search results, completing the first subquery using the new subject / predicate / object triples, updating the curated knowledge base to include the completed first subquery, determining that a second query can be resolved using the updated knowledge base, and completing the second subquery using the updated knowledge base.

[0007] According to yet another feature, a computer program product for implementing on-demand extraction and merging includes a computer readable medium having program instructions embodied therein, the computer readable medium being non-transient and the program instructions being executable by a processor, the computer program product including: receiving by the processor a query in a natural language format; deriving by the processor a plurality of sub-queries from the query; determining by the processor that a first sub-query of the plurality of sub-queries is not resolvable using the curated knowledge base; generating by the processor additional search terms for the first sub-query using query relaxation; The method includes querying a search index for the unstructured text using the query and additional search terms, the processor receiving search results in response to the querying of the search index, the processor generating new subject / predicate / object triples based on the search results, the processor completing a first subquery using the new subject / predicate / object triples, the processor updating a curated knowledge base to include the completed first subquery, the processor determining that a second query can be resolved using the updated knowledge base, and the processor completing the second subquery using the updated curated knowledge base.

[0008] Other features and embodiments of the present invention will become apparent from the following detailed description, which, when taken in conjunction with the drawings, illustrate by way of example the principles of the invention. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a network architecture according to one embodiment of the present invention. [Diagram 2]FIG. 2 illustrates a representative hardware environment associated with the server or client or both of FIG. 1 according to one embodiment of the present invention. [Diagram 3] FIG. 3 illustrates a method for query resolution using structured and unstructured data according to one embodiment of the present invention. [Figure 4] FIG. 4 illustrates a method for implementing on-demand extraction and consolidation according to one embodiment of the present invention. [Diagram 5] FIG. 5 illustrates an exemplary question-and-answer (QA) system that integrates unstructured and structured data according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following description is made for the purpose of illustrating the general principles of this invention and is not meant to limit the inventive concepts of the claims herein. Moreover, particular features described herein can be used in combination with other described features, in each of the various possible combinations and permutations.

[0011] Unless otherwise specifically defined herein, all terms are to be given their broadest possible interpretation, including the meanings implied from the specification, as well as those understood by a person of ordinary skill in the art or defined in dictionaries, treatises, etc., or any combination thereof.

[0012] It should also be noted that the terminology used in the specification and the appended claims is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, the terms "comprises" and "comprising", when used herein, specify the presence of stated features, integers, steps, operations, elements, and components or combinations thereof, but do not exclude one or more other features, integers, steps, operations, elements, components, and groups thereof or combinations thereof.

[0013] The following description discloses several preferred embodiments of systems, methods and computer program products for query resolution using structured and unstructured data.

[0014] In one general embodiment, a computer-implemented method includes receiving a question in a natural language format, deriving a plurality of sub-queries from the question, determining that a first sub-query of the plurality of sub-queries cannot be resolved using a curated knowledge base, resolving the first sub-query using unstructured text, and adapting the curated knowledge base using the resolved first sub-query.

[0015] In another general embodiment, a computer program product for resolving queries using structured and unstructured data includes a computer-readable medium having program instructions embodied therein, the computer-readable medium being not itself a transitional signal, the program instructions being executable by a processor to cause the processor to receive a question in a natural language format, derive a plurality of sub-queries from the question, determine that a first sub-query of the plurality of sub-queries cannot be resolved using the curated knowledge base, resolve the first sub-query using unstructured text, and adapt the curated knowledge base using the resolved first sub-query.

[0016] In another general feature, the system includes a processor and logic logically integrated with the processor and executable by the processor or integrated with the processor and executable by the processor, the logic configured to receive a question in a natural language format, derive a plurality of sub-queries from the question, determine that a first sub-query of the plurality of sub-queries cannot be resolved using the curated knowledge base, resolve the first sub-query using the unstructured text, and adapt the curated knowledge base using the resolved first sub-query.

[0017] In another general embodiment, a computer-implemented method includes receiving a question in a natural language format, deriving a first sub-query and a second sub-query from the question, determining that the first sub-query cannot be resolved using the curated knowledge base, generating additional search terms for the first sub-query using query relaxation, querying a search index for unstructured text using the first sub-query and the additional search terms, receiving search results in response to querying the search index, generating new subject / predicate / object triples based on the search results, completing the first sub-query using the new subject / predicate / object triples, updating the curated knowledge base to include the completed first sub-query, determining that a second query can be resolved using the updated curated knowledge base, and completing the second sub-query using the updated curated knowledge base.

[0018] In another general embodiment, a computer program product for implementing on-demand extraction and merging includes a computer-readable medium having program instructions embodied therein, the computer-readable medium being non-transient and the program instructions being executable by a processor, the computer-readable medium including: receiving, by the processor, a query in a natural language format; deriving, by the processor, a first sub-query and a second sub-query from the query; determining, by the processor, that the first sub-query is unresolvable using the curated knowledge base; generating, by the processor, additional search terms for the first sub-query using query relaxation; the processor receiving search results in response to the querying of the search index; the processor generating a new subject / predicate / object triple based on the search results; the processor completing the first subquery using the new subject / predicate / object triple; the processor updating the curated knowledge base to include the completed first subquery; the processor determining that a second query can be resolved using the updated curated knowledge base; and the processor completing the second subquery using the updated curated knowledge base.

[0019] Figure 1 illustrates an architecture 100 in one embodiment. As shown in Figure 1, multiple remote networks 102 are provided, including a first remote network 104 and a second remote network 106. A gateway 101 can couple between the remote networks 102 and a proximate network 108. In the context of this architecture 100, the networks 104, 106 can take any form, including, but not limited to, a LAN, a WAN such as the Internet, a Public Switched Telephone Network (PSTN), an internal telephone network, etc.

[0020] In use, the gateway 101 provides an entry point from the remote network 102 to the proximal network 108. As such, the gateway 101 may function as a router that manages a given data packet arriving at the gateway 101, and a switch that provides the actual path in and out of the gateway 101 for a given packet.

[0021] Also included is at least one data server 114 connected to the proximate network 108, which is accessible from multiple remote networks 102 via the gateway 101. It should be noted that the data server(s) 114 can include any type of computing device / groupware. Coupled to each data server 114 are multiple user devices 116. The user devices 116 may also be directly coupled through one of the networks 104, 106, 108. Such user devices 116 may include desktop computers, laptop computers, handheld computers, printers, or any other type of logic. It should be noted that the user devices 111 may also be directly coupled to any network in one embodiment.

[0022] A peripheral device 120 or a chain of multiple peripheral devices 120, such as a facsimile machine, a printer, a network or local or both storage unit or system, may be coupled to one or more of the networks 104, 106, 108. It should be noted that databases and / or additional components may be used and integrated with any type of network element coupled to the networks 104, 106, 108. In the context of this specification, a network element may refer to any component of a network.

[0023] Through several approaches, the methods and systems described herein may be implemented in conjunction with or on a virtualized system, or both, or on a system that emulates one or more systems, such as a UNIX system that emulates an IBM zOS environment, a UNIX system that virtually hosts a MICROSOFT WINDOWS environment, a MICROSOFT WINDOWS system that emulates an IBM z / OS environment, etc. This virtualization and / or emulation may be enhanced through VMWARE in some embodiments.

[0024] In a further approach, one or more networks 104, 106, 108 may represent a cluster of systems commonly referred to as a "cloud." In cloud computing, shared resources such as processing power, peripherals, software, data, servers, etc. are provided in an on-demand relationship to any system in the cloud, thereby enabling access to and distribution of services across multiple computing systems. Cloud computing typically involves Internet connections between systems running in the cloud, although other technologies for coupling the systems may also be used.

[0025] Figure 2 illustrates a typical hardware environment associated with the user device 116 or server 114, or both, of Figure 1. Figure 2 illustrates a typical hardware configuration of a workstation that includes a central processing unit 210, such as a microprocessor, and a number of other units interconnected via a system bus 212.

[0026] The workstation shown in FIG. 2 includes an I / O adapter 218 for coupling peripherals such as random access memory (RAM) 214, read only memory (ROM) 216, and disk storage unit 220 to bus 212, a user interface adapter 222 for coupling a keyboard 224, a mouse 226, speakers 228, a microphone 232 or other user interface devices such as a touch screen and a digital camera (not shown) to bus 212, a communications adapter 234 for coupling the workstation to a communications network 235 (e.g., a data processing network), and a display adapter 236 for coupling bus 212 to a display device 238.

[0027] The workstation has an operating system installed thereon, such as the MICROSOFT WINDOWS® operating system, MAC OS®, UNIX® OS, etc. The preferred embodiment may also be implemented on platforms and operating systems other than those mentioned above. The preferred embodiment may be written using XML, C, or C++ languages ​​or combinations thereof, or other programming languages, along with object oriented programming methodologies.

[0028] Referring now to Figure 3, there is shown a flow chart of a method 300 according to one embodiment. Method 300 may be performed in accordance with the present invention in any of the environments illustrated in Figures 1-2 and 5, among other embodiments. Of course, more or fewer operations may be included in method 300, not specifically depicted in Figure 3, as would be understood by one of ordinary skill in the art upon reading this specification.

[0029] Each step of the method 300 may be implemented by any suitable component of an operating environment. For example, in various embodiments, the method 300 may be performed in part or in whole by one or more servers, computers, or some other device having one or more processors therein. For example, a processor having at least one hardware component, preferably a processing circuit(s), chip(s), or module(s) implemented in hardware or software or both, may be used within any device to perform one or more steps of the method 300. Exemplary processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, combinations thereof, or any other suitable computing device known in the art.

[0030] 3, method 300 may begin at operation 302, where a question is received in a natural language format. In one embodiment, the question is received by a Question and Answer (QA) application. In another embodiment, the question may include a query to be answered by an OA application. In yet another embodiment, the natural language format may include a format used in everyday human conversation.

[0031] Additionally, in one embodiment, the natural language format can be distinguished from formal languages ​​such as those using program code. In another embodiment, an example question in the natural language format can include the question "What are the revenues of company X's competitors in 2018?"

[0032] Further, the method 300 may proceed to operation 304, where a number of sub-queries are derived from the question. In one embodiment, the sub-queries may be derived from the question by parsing a natural language format of the question into a logical form. For example, the question may be parsed to identify one or more components of the question. In another embodiment, the one or more components may be analyzed to generate one or more sub-queries of the question.

[0033] Further, in one embodiment, each subquery may include an n-tuple, such as a triple (e.g., a subject / predicate / object triple, etc.). For example, an incomplete triple may be missing data for one or more of the subject, predicate, and object fields. In another embodiment, a subject / predicate / object triple may include a data object including a subject, a predicate, and an object that is filled in based on the results of parsing the question. For example, for the question "What are the revenues of competitors of company X in 2018?", the derived subject / predicate / object triples may include the following incomplete triples: first subquery (?y“revenues”?x) and second subquery (?x,“competitors”,“company X”).

[0034] Method 300 may also proceed to operation 306, where it is determined that a first subquery of the plurality of subqueries cannot be resolved using the curated knowledge base. In one embodiment, the curated knowledge base may include data for identifying a predefined domain collected from unstructured text. For example, unstructured text (e.g., free text, etc.) may be parsed (manually or automatically or both) to determine text data (e.g., words, sentences, etc.) that match the predefined domain. In another embodiment, the curated knowledge base may include a plurality of complete triplets.

[0035] Additionally, in one embodiment, a given domain can include one or more predefined topics for the curated knowledge base (e.g., company financial information, etc.). In another embodiment, text data matching a given domain can be added to a curated knowledge base for the given domain. In yet another embodiment, the curated knowledge base can include a number of complete triples.

[0036] Further, in one embodiment, a first subquery of the plurality of subqueries may not be resolvable by a QA application using a curated knowledge base. The first subquery of the plurality of subqueries may include an incomplete triple that cannot be completed by comparing the incomplete triple to the curated knowledge base. In another example, a second subquery of the plurality of subqueries may be resolvable by a QA application using a curated knowledge base. For example, the second subquery of the plurality of subqueries may include an incomplete triple that can be completed by comparing the incomplete triple to the curated knowledge base. In another embodiment, the first subquery may be determined to be insolvable using the curated knowledge base based on an analysis of the semantic relevance of the first subquery to the curated knowledge base.

[0037] Further, the method 300 may proceed to operation 308, where the first sub-query is resolved using the unstructured text. In one embodiment, the unstructured text is From there, a curated knowledge base was generated. The unstructured text may include a text corpus. In another embodiment, the unstructured text may include historical text documents (e.g., text files, HTML, etc.) that do not have any predefined structure. In yet another embodiment, the first subquery may include an incomplete triple. In one embodiment, the first subquery may be resolved using the unstructured text in response to determining that the first subquery cannot be resolved using the curated knowledge base.

[0038] Also, in one embodiment, resolving the first subquery may include querying a search index previously generated using the unstructured text. For example, the unstructured text may be pre-processed (e.g., parsed, analyzed, etc.) to generate the search index. In another example, the search index may include a keyword-based search engine generated for the unstructured text. In yet another example, the search index may include a keyword-based search engine generated for the unstructured text. Illustratively, the search engine may search using terms found in the incomplete triples of the first subquery.

[0039] Additionally, in one embodiment, query relaxation can be used to generate additional terms to include in the search index query. For example, performing query relaxation can include determining similar terms (e.g., synonyms, etc.) for terms in the incomplete triples in the first subquery. In another example, the search index query can be modified to include these similar terms in addition to the terms in the incomplete triples.

[0040] Further, in one embodiment, resolving the first subquery can include receiving results of the search index query. For example, the results can include one or more fragments of interest. In another example, each of the one or more text fragments can include one or more portions of text (e.g., words, phrases, sentences, paragraphs, etc.) that match the search index query. In yet another example, the one or more text fragments can be ranked according to one or more criteria (e.g., relevance to the search index query, etc.).

[0041] Further, in one embodiment, resolving the first subquery includes generating new triples using the results of the query of the search index. For example, new subject / predicate / object triples can be generated using one or more text fragments received as a result of the query of the search index. In another embodiment, these new triples can be complete (e.g., they can have values ​​for the subject, predicate, and object fields).

[0042] Also, in one embodiment, resolving the first subquery can include completing the first subquery using the new triplet. For example, the first subquery can include an incomplete triple. In another example, the incomplete triples of the first subquery can be completed based on the new triplet. In yet another example, one or more alignment-based answer identification and similar operations are performed based on the new triplet for the incomplete triples of the first subquery to determine values ​​for missing fields or incomplete triples.

[0043] Also, in one embodiment, the unstructured data may be periodically updated to include information that was previously searched but not found in the unstructured data. For example, if the amount of text fragments returned in response to a search index query falls below a predetermined threshold, a search is conducted on additional unstructured data containing results that are determined to be more relevant to the search index query. This additional unstructured data is incorporated into the current unstructured data to augment the corpus of unstructured data.

[0044] Additionally, the method 300 proceeds to operation 310 where the curated knowledge base is adapted using the resolved first sub-query. In one embodiment, adapting the curated knowledge base includes adding the resolved first sub-query to the curated knowledge base. In another embodiment, the resolved first sub-query can include a completed triple to be added to an existing completed triple in the curated knowledge base based on a determination of a dependency between the resolved first sub-query and an existing triple in the curated knowledge base.

[0045] Further, in one embodiment, a second subquery of the plurality of subqueries can be resolved using the adapted curated knowledge base. For example, resolving the second subquery can include completing the second subquery using triples in the adapted curated knowledge base. In another embodiment, the second subquery of the plurality of subqueries can include an incomplete triple that is compared to a completed triple in the adapted curated knowledge base (e.g., including the resolved first completed triple).

[0046] Additionally, in one embodiment, one or more alignment-based answer identification and similarity operations may be performed on the incomplete triples of the second subquery based on the completed triples in the adapted curated knowledge base to determine values ​​for the missing or incomplete triples. In another embodiment, values ​​for the missing or incomplete triples of the second subquery may be added to the incomplete triples to generate complete triples (thereby resolving the second subquery).

[0047] Also, in one embodiment, the resolved first sub-query and the resolved second sub-query can be used to generate an answer to the received question. For example, information in the fields of the triples of the resolved first sub-query and the resolved second sub-query can be converted into a response having a natural language format. In another example, the response can be returned in a natural language format.

[0048] In this manner, both unstructured data (unstructured text) and structured data (curated knowledge base) can be used together to answer questions by the QA system. Additionally, the curated knowledge base can be dynamically augmented using the results of unstructured text search and analysis. This advances the relevance of the curated knowledge base, which can improve the accuracy and performance of future sub-query searches performed using the curated knowledge base. Due to the improvement in the relevance of the curated knowledge base, computing devices using the curated knowledge base can also benefit from improved performance.

[0049] Referring now to Figure 4, there is shown a flow chart of a method 400 for implementing on-demand extraction and consolidation, according to one embodiment. Method 400 may be performed in accordance with the present invention in any of the environments illustrated in Figures 1-2 and 5, among other embodiments. Of course, more or fewer operations may be included in method 400 than those specifically described in Figure 4, as would be understood by one of ordinary skill in the art upon reading this specification.

[0050] Each of the steps of method 400 may be performed by any suitable component of an operating environment. For example, in various embodiments, method 400 may be performed in part or in whole by one or more servers, computers, or some other device having one or more processors therein. For example, a processor having at least one hardware component, such as processing circuit(s), chip(s), or module(s) implemented in hardware or software or both, may be used within any device to perform one or more steps of method 400. Exemplary processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, combinations thereof, or any other suitable computing device known in the art.

[0051] As shown in FIG. 4, method 400 may begin at operation 402, where a question is received in a natural language format. Additionally, method 400 proceeds to operation 404, where a first sub-query and a second sub-query are derived from the question. In one embodiment, the first sub-query and the second sub-query both include incomplete subject / predicate / object triples. In another embodiment, the first sub-query is different from the second sub-query. For example, the first sub-query may have different triple field values ​​compared to the second sub-query.

[0052] Additionally, method 400 may proceed to operation 406, where it is determined that the first sub-query cannot be resolved using the curated knowledge base based on the unstructured text. Method 400 may further proceed to operation 408, where additional search terms are generated for the first sub-query using query relaxation. Method 400 may further proceed to operation 410, where a search index is queried for the unstructured text using the first sub-query and the additional search terms. In one embodiment, the search index may have been previously generated using the unstructured text. In another embodiment, a search index for the unstructured text may be queried in response to determining that the first sub-query cannot be resolved using the curated knowledge base.

[0053] The method 400 also proceeds to operation 412, where search results are received in response to the query of the search index. Additionally, the method 400 proceeds to operation 414, where new subject / predicate / object triples are generated based on the search results. For example, the search results may include multiple related fragments, and open information extraction may be used to extract new triples from the fragments. In another embodiment, the new triples may be ranked according to their semantic relevance to the first subquery and the second subquery, and a predetermined number of the top ranked triples are selected as the new triples. Additionally, the method 400 proceeds to operation 416, where the first subquery is completed using the new subject / predicate / object triples. Additionally, the method 400 proceeds to operation 418, where the curated knowledge base is updated to include the completed first subquery.

[0054] The method 400 also proceeds to operation 420, where it is determined that the second sub-query can be resolved using the updated curated knowledge base. In one embodiment, the second sub-query is deferred until the first sub-query is completed. For example, the second sub-query can be determined in response to the first sub-query based on an analysis of incomplete triples of the first sub-query and the second sub-query. By way of illustration, a field that is not in the second sub-query may be located in the first sub-query, and it can be determined that the field that is not in the second sub-query can be completed upon completion of the first sub-query. In another embodiment, the second sub-query can be deferred until the first sub-query is completed in response to this determination.

[0055] Additionally, method 400 may proceed to operation 422, where the second sub-query is completed using the updated curated knowledge base. In one embodiment, the second sub-query may be completed in response to determining that the first sub-query has been completed and added to the curated knowledge base. In another embodiment, the second sub-query is resolved using subject / predicate / object triplets in the updated curated knowledge base that include triples from the completed first sub-query.

[0056] In this manner, searches of both the curated knowledge base and unstructured text can be performed to augment the curated knowledge base and accurately answer questions resolved.

[0057] 5 illustrates a question-and-answer (QA) system 500 that integrates unstructured and structured data, according to an example embodiment. The QA system 500 answers questions by integrating information from both unstructured data (free text) and structured data (curated knowledge base 502) in real time.

[0058] As shown, the input to the system is a natural language question Q 504. Query Q 504 is parsed by a natural language understanding unit 506, which converts the question into a machine-understandable sub-query 508. For example, sub-query 508 may contain an incomplete subject / predicate / object triple.

[0059] Additionally, for each subquery q_x in subqueries 508, query plan generator 510 performs query expansion by expanding q_x with a set of similar queries. For example, if q_x is (company,acquisition,?) may include.

[0060] The query expansion then (Company, Acquisition, ?) and (?,Acquired by,Company) and so on, and is applied to the estimated queries.

[0061] In one embodiment, the above query expansion can be completed using one or more paraphrase models. Then, string-similarity matching, embedding (e.g., pre-trained word2vec embeddings) matching, and paraphrase-based matching can be used to determine which data source (e.g., unstructured text 512 or curated knowledge base 502) each subquery 508 should be sent to. Based on the determined data source, the subqueries 508 are split into two subsets: free text subqueries 514 and curated knowledge base subqueries 516.

[0062] Further, the free text sub-query 514 is sent to a free text query module 518, which searches an indexed text corpus 520 to identify a collection of relevant text fragments 522 that may contain an answer to the free text sub-query 514. The unstructured text 512 is indexed 524 to generate a text corpus 520, which is enriched using indexing techniques (such as TD-IF based methods) to quickly identify and retrieve the relevant text fragments 522. When searching the indexed corpus 520, the free text sub-query 514 can be expanded using paraphrase models to increase comprehensiveness.

[0063] Further, the relevant text fragments 522 are passed to an information extractor 526 (such as a relaxation extractor) to extract potential answers to each free text sub-query 514 (thereby completing each free text sub-query 514) and rank them based on their relevance scores.

[0064] Additionally, all or some of the completed free-text sub-queries 514 may then be added to the curated knowledge base 502, updating the curated knowledge base 502. In one embodiment, all or some of the completed free-text sub-queries 514 may be combined with the completed curated knowledge base sub-queries 516 (completed, for example, by a query to curated knowledge base module 528 using the curated knowledge base 502) to determine a final answer that may be provided in response to the natural language question Q504. In another embodiment, one or more curated knowledge base sub-queries 516 may be completed by a curated knowledge base query module 528 using the updated curated knowledge base 502, and the completed knowledge base sub-queries 516 may then be used to determine a final answer to be provided in response to the natural language question Q504.

[0065] In another embodiment, if there are aggregate sub-queries identified from the natural language question Q504, one or more aggregations may be performed.

[0066] On-demand extraction and assembly Knowledge Bases (KB) have become the core of many data-centric applications, such as question-answering (QA) applications. Large knowledge bases store vast amounts of knowledge about different entities, allowing systems to provide answers to entity-centric questions. Among these knowledge bases, curated KBs allow systems to reason over the knowledge with a high degree of precision and return accurate answers.

[0067] Knowledge bases are written and curated by domain experts who are required to propose domain-specific schemas and curation workflows to extract facts that are affirmed by the schema. Even with high accuracy, these KBs may be incomplete because they assume that the world knowledge can be encoded using selected linguistic predicates. Often, the absence of information from the KBs is present in the unstructured text on which they are based.

[0068] By way of example, a KB may contain information about public companies and some of their financial metrics, but may not include relationships such as acquirers, competitors, and subsidiaries that exist between the companies. This information, if available, can be curated from the text using information technology to complete and supplement information that is not in the KBs. This can help generalize the QA system that encounters incompleteness in the curated KBs. Domain experts may ask questions that take the automatically curated text and use insights from the answers provided by the system to further improve the curation workflow.

[0069] As an example, a domain expert might want to build a KB that contains exact financial metrics for public companies from tables in those companies' financial reports (e.g., 10-k reports). However, the queries that users are interested in would look the same. "Show me the revenues of company X's competitors in 2016"

[0070] This query cannot be supported from the curated KB alone and can be decomposed into two sub-queries: (1) (y, revenue, x), and (2) (?x,“competitor of”, “company X”), Here, query (1) can be answered by the curated knowledge base as soon as we know the answer to query (2), which can be found in the free text of Company X's financial report (assuming that the report actually contains the information of interest).

[0071] As a result, in one embodiment, heterogeneous data sources, structured KBs, and unstructured text can be combined and used to build a dynamic hybrid knowledge base that benefits the accuracy of the structured KBs and the comprehensiveness of facts from the unstructured text. For a given natural language question or interpretation of a natural language question, the system can selectively query the unstructured text or the structured KBs as a whole, extract and combine information as needed, and combine the results to answer the query.

[0072] Exemplary features include the following: 1. Given a local representation of a Natural Language Query (NLQ), referred to as a logical query, a corresponding query plan is generated. This plan consists of query operators that query existing knowledge bases and unstructured data to generate intermediate results, combine the intermediate answers, and aggregate the final results. 2. For query operators that access unstructured data, relevant fragments associated with the operator are retrieved from the unstructured data. a. Facts from relevant snippets are extracted and ranked. b. The results of the input query are determined from the extracted facts. will be done. 3. For those query operators that have access to the curated knowledge base, a query is made to the knowledge base to retrieve relevant facts. 4. The final result for the original query plan is generated.

[0073] One exemplary approach can answer queries by presenting free text in an on-demand manner to find missing information. This approach supports many more queries when compared to static knowledge base approaches. In addition, the missing information is extracted from the free text in an on-demand manner. Furthermore, this approach breaks down complex queries into multiple sub-queries that can be answered more reliably by the curated or extracted KBs when the relevant information is not present in the curated KB.

[0074] Before answering any question, the system requires a one-time preprocessing step: in an offline process, the system creates an inverted index that allows efficient keyword search at query time over multiple text corpora.

[0075] The system generates an inverted index, which allows efficient keyword searches over multiple text corpora in an offline process, in addition to which several metadata such as years, namespaces, and entities associated with the text are indexed.

[0076] The input to the system is a logical query (e.g., an interpretation of a question in natural language). A logical query is a combination of multiple clauses, where each clause is an n-tuple with one or more variable gist. The material plan generator translates the logical query to generate a material plan for the query. The material plan determines the execution order of the clauses and whether the clauses need to be executed against a curated KB or free text. In addition, the material plan determines how to combine and optionally aggregate intermediate results.

[0077] When a query clause needs to be executed against the free text, the query clause is first rewritten using metadata to generate a more flexible query plan. This helps to retrieve text fragments that may not express the information of interest. Each flexible query plan is then paraphrased to handle the many ways the same fact may be expressed in the free text. The query plans are ranked based on their semantic similarity to the original query. The top-k plans are optimized and queries with common keywords across the query plans are identified and executed at once.

[0078] Keyword queries are issued against the indexed text corpus to search for relevant text fragments. An open-domain information extractor extracts factual triples (subject-predicate-object) from the text fragments. As these triples are noisy and non-standardized, the system further processes and annotates them to help accurately identify answers. The triples are aggregated across different query plans and ranked based on their estimated confidence. The system then identifies answers from the triples using an alignment-based answer identification algorithm. These answers are further disambiguated and returned to the plan generator for further use.

[0079] When a query clause needs to be executed against the curated KB, it must first be translated into a query in a high-level declarative language. Ontology Query Language (OQL) can be used to access the information stored in the curated KB. Query decomposition breaks down the query into a number of simple clauses, so that a manually created template can be used to translate the clause into an OQL query. The overlap of information on the schema elements between the logical query and the OQL can be used to select a template to query the curated KB. The selected template is instantiated and the corresponding OQL query is issued against the KB, returning a set of tuples.

[0080] In one embodiment, the heterogeneous data sources can include a structured KB that has highly accurate real-world facts and free text, which includes a vast repository of sentences that capture the facts in the KB. The hybrid system benefits from the precision of the curated KB and the comprehensiveness of the textual facts, and helps domain experts identify predicates that are not in the curated KBs.

[0081] In addition, in one embodiment, keyword queries can be routed to a superset of relevant documents using metadata about query relaxation and rewriting. Additionally, paraphrasing models can be used for query expansion.

[0082] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail integration. The computer program product may include a recording medium (or media) having computer-readable program instructions thereon for causing a processor to perform features of the present invention.

[0083] A computer readable recording medium may be a tangible device capable of holding and storing a plurality of instructions for use by an instruction execution device. A computer readable medium may be, for example, but not limited to, an electrical recording device, a magnetic recording device, an optical recording device, an electro-magnetic recording device, a semiconductor recording device, or any suitable combination thereof. More specific examples of computer readable recording media include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a punch card, or a mechanically encoded device having protruding structures within a groove having instructions recorded therein, and any suitable combination thereof. As used herein, a computer-readable recording medium is not to be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves such as wave guides or other communications media (e.g., light pulses passing through fiber optic cable), or electrical signals communicated through wires.

[0084] The computer program instructions described herein can be downloaded from a computer readable recording medium into each computing / processing device, or can be downloaded to an external computer or external recording device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, and combinations thereof. The network can include copper communication cables, fiber optic communication, wireless communication routers, firewalls, switches, gateway computers, and edge servers, and combinations thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions into a computer readable recording medium in each computing / processing device for storage.

[0085] The computer readable program instructions for carrying out the operations of the present invention can be either source code or object code written in any combination of programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or one or more procedural programming languages, such as Smalltalk®, object oriented programming languages ​​such as C++, the “C” programming language, or similar programming languages. The computer readable program instructions can be executed entirely on the user computer, partially on the user computer as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user computer through any type of network, including a local area network (LAN), a wide area network (WAN), or the connection can be to an external computer (e.g., through an Internet service provider). In some embodiments, computer readable program instructions may be executed by electrical circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), and state information of the computer readable program instructions may be used to personalize the electrical circuitry to perform features of the present invention.

[0086] The embodiments of the invention described herein have been described with reference to flowchart instructions and / or block diagrams of methods, apparatus (systems), and computer readable recording media and computer programs according to embodiments of the invention. It is understood that any combination of flowchart illustrations and / or block diagrams and / or blocks in flowchart illustrations and / or block diagrams can be implemented by computer readable program instructions.

[0087] These computer readable program instructions can be provided to a general purpose computer, a special purpose computer, or other processor or other programmable data processing device to produce a machine, which when executed by the processor of the computer or other programmable data processing device produces means for implementing the functions / operations specified in the block or blocks of the flowcharts and block diagrams, or combinations thereof. These computer readable program instructions that direct a computer, programmable data processing device, and other device, or combinations thereof, to function in a particular manner can also be stored on a computer readable recording medium, and the computer readable recording medium having instructions stored therein constitutes an article of manufacture containing instructions that implement the functional / operational features specified in the block or blocks of the flowcharts and block diagrams, or combinations thereof.

[0088] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device and cause a computer implemented process to perform a series of operational steps on the computer, other programmable apparatus, or other device to implement the functions / acts specified in the blocks of the flowcharts and block diagrams or in a combination thereof.

[0089] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and possible implementation operations of systems, methods, and computer programs according to various embodiments of the present invention. In this respect, the flowcharts or block diagrams may represent modules, segments, or portions of instructions, which include one or more executable instructions for implementing a particular logical function(s). In some alternative implementations, the functions described in the blocks may be performed other than as illustrated. For example, two blocks shown in succession may be performed in a manner that is actually performed as one step, may be performed simultaneously, substantially simultaneously, partially or completely overlapped in time, or the blocks may be performed in reverse order, depending on the functionality involved. It is also noted that the illustration of the blocks in the block diagrams and / or the illustration of the flowcharts or combinations thereof may be implemented by a system based on special purpose hardware that performs a particular function or operation or performs specific purpose hardware and computer instructions.

[0090] Further, systems according to various embodiments may include a processor and logic integrated and / or executable on the processor, the logic configured to perform one or more process steps listed herein. The processor may be of any configuration, including a number of components, such as a discrete processor or processing hardware, memory, I / O interfaces, as described herein. By integrated, it is meant that the processor has logic embedded therein as hardware logic, such as an application specific integrated circuit (ASIC), FPGA, etc. By executable by the processor, it is meant that the logic is hardware logic, such as firmware, software logic, part of an operating system, part of an application program, etc., or software logic that is accessible by the hardware and the processor and configured to cause the processor to perform some function upon execution by the processor. The software logic may be stored in any memory type, local or remote, or both. Any processor known in the art may be used, such as a software processor module or a hardware processor, such as an ASIC, FPGA, central processing unit (CPU), integrated circuit (IC), graphics processing unit (GPU), etc.

[0091] It will be apparent that the above described systems and / or methodologies can be combined in any manner and multiple combinations can be created from the description provided above.

[0092] Additionally, embodiments of the present invention may be provided in a manner that provides services on demand, deployed on behalf of the customer.

[0093] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exclusive or limited to the disclosed embodiments. Many modifications or variations will be obvious to those skilled in the art without departing from the scope and spirit of the present disclosure. The terms used in this specification are selected to best explain the principles of the present embodiments, practical applications, or technical improvements beyond the technology found in the market, or to allow others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method, comprising: receiving queries in natural language format; deriving a plurality of sub-queries from said query; determining that a first subquery of the plurality of subqueries cannot be resolved using a curated knowledge base; Based on the first sub-query derived from the received question, resolving the first sub-query having incomplete tuples using a search index generated by unstructured text; and adapting the curated knowledge base by appending the resolved first subquery having a complete tuple.

4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the question is received by a question and answer (QA) application.

3. 3. The computer-implemented method of claim 1, wherein each of the plurality of sub-queries is derived by parsing a natural language format of the question into a logical form.

4. 4. The computer-implemented method of claim 1, wherein the incomplete tuples include incomplete subject / predicate / object triples and the complete tuples include complete subject / predicate / object triples.

5. The computer-implemented method of any one of claims 1 to 4, wherein the unstructured text comprises a text corpus, and the curated knowledge base is generated from the text corpus.

6. The computer-implemented method of any one of claims 1 to 5, wherein query relaxation is used to generate additional terms for inclusion in queries of the search index.

7. The computer-implemented method of any one of claims 1 to 6, wherein resolving the first sub-query comprises querying the search index and receiving results of the query.

8. 8. The computer-implemented method of claim 7, wherein resolving the first subquery comprises generating new triples using results of querying the search index.

9. 9. The computer-implemented method of claim 8, wherein resolving the first subquery comprises completing the first subquery using the new triple.

10. The computer-implemented method of any one of claims 1 to 9, further comprising resolving a second sub-query of the plurality of sub-queries by using the adapted curated knowledge base.

11. 11. The computer-implemented method of claim 10, wherein the resolved first sub-query and the resolved second sub-query are used to generate an answer to the received question.

12. The computer-implemented method of any one of claims 1 to 11, wherein the incomplete tuples are incomplete tuples that cannot be completed by comparison with the curated knowledge base.

13. A computer program product capable of executing the computer-implemented method according to any one of claims 1 to 12.

14. 1. A system comprising: A processor; logic integrated into, executable by or integrated into said processor, said logic configured to perform a computer-implemented method according to any one of claims 1 to 12; A system including:

15. 1. A computer-implemented method, comprising: receiving queries in natural language format; deriving a first sub-query and a second sub-query from the query; determining that the first sub-query cannot be resolved using a curated knowledge base generated based on unstructured text; generating additional search terms for the first sub-query using query relaxation; querying a search index generated from the unstructured text using the first sub-query derived from the received question and additional of the search terms; receiving search results in response to querying the search index; generating new subject / predicate / object triples based on the search results; completing the first subquery having an incomplete subject / predicate / object triple using the new subject / predicate / object triple; updating the curated knowledge base by adding the completed first subquery having a complete subject / predicate / object triple; determining that the second sub-query can be resolved using the updated curated knowledge base; and completing the second sub-query using the updated curated knowledge base.

23. A computer-implemented method comprising:

16. 16. The computer-implemented method of claim 15, wherein the unstructured text comprises a text corpus, and the curated knowledge base is generated from the text corpus.

17. 17. The computer-implemented method of claim 15 or claim 16, wherein the incomplete triples are incomplete triples that cannot be completed by comparison to the curated knowledge base.

18. 1. A computer-implemented method, comprising: receiving a natural language input corresponding to a query with a natural language parser that reduces the natural language input to subject / predicate / object (SPO) triples; directing a first subset of SPO triples having incomplete SPO triples to a query module for free text and a second subset of SPO triples different from the first subset to a query module for a curated knowledge base, taking into account semantic relevance to a curated knowledge base; accessing, by said free text query module, the indexed text data to identify fragments of interest in said first subset of said SPO triples; extracting SPO triples from the fragment of interest that correspond to the first subset of SPO triples; ranking the extracted SPO triples according to their semantic relevance to the first subset of SPO triples; selecting a predetermined number of SPO triples from the top ranked and extracted SPO triples; and combining the selected number of SPO triples into a second subset of SPO triples to form an answer to the question; the computer-implemented method further comprising: The computer-implemented method further comprising completing the first subset having the incomplete SPO triples based on the fragment of interest and adding the completed first subset having complete SPO triples to the curated knowledge base.

19. 20. The computer-implemented method of claim 18, wherein the combining is performed in terms of dependencies generated by the natural language parser between a selected predetermined number of the SPO triples and the second subset of SPO triples.

Images