Resolving queries using structured and unstructured data

Abstract

Resolving queries using structured and unstructured data. [Solution] A computer-implemented method includes receiving a question in a natural language format; deriving a plurality of subqueries from the question; determining that a first subquery of the plurality of subqueries cannot be resolved using a curated knowledge base; resolving the first subquery using unstructured text; and adapting the curated knowledge base using the resolved first subquery.

Description

Technical Field

[0001] The present invention generally relates to data analysis, and more specifically to the resolution of queries using a combination of indexed unstructured data and a curated knowledge base.

Background Art

[0002] Multiple knowledge bases (KBs) are frequently used within applications such as question answering (QA) applications that are centered around data. Such knowledge bases are generated using curated unstructured text and provide accurate answers based on high-precision knowledge. However, knowledge bases provide accurate results for a specific subset of information and cannot provide answers to queries outside of those specific subsets.

Summary of the Invention

[0003] A computer-implemented method according to one feature includes receiving a question in natural language format, deriving a plurality of subqueries from the question, determining that a first subquery of the plurality of subqueries cannot be resolved using a curated knowledge base, resolving the first subquery using unstructured text, and adapting the curated knowledge base using the resolved first subquery.

[0004] According to another feature, a computer program product for resolving queries using structured and unstructured data includes a computer-readable recording medium having program instructions materialized therein, the computer-readable recording medium is not a transient signal in itself, the program instructions are executable by a processor, and the processor performs the following actions: the processor receives a query in natural language format; the processor derives multiple subqueries from the query; the processor determines that the first subquery of the multiple subqueries cannot be resolved using a curated knowledge base; the processor resolves the first subquery using unstructured text; and the processor adapts the curated knowledge base using the resolved first subquery.

[0005] According to another feature, the system includes a processor and logic that is logically integrated with the processor, executable by the processor or integrated with the processor, or executable logic that receives a question in natural language format, derives multiple subqueries from the question, determines that the first subquery of the multiple subqueries cannot be resolved using a curated knowledge base, resolves the first subquery using unstructured text, and adapts the curated knowledge base using the resolved first subquery.

[0006] According to another feature, the computer implementation method includes receiving a question in natural language format, deriving a first subquery and a second subquery from the question, determining that the first subquery of multiple subqueries cannot be resolved using the curated knowledge base, generating additional search terms for the first subquery using query relaxation, querying a search index for unstructured text using the first subquery and additional search terms, receiving search results in response to the query of the search index, generating new subject / predicate / object triples based on the search results, completing the first subquery using the new subject / predicate / object triples, updating the curated knowledge base to include the completed first subquery, determining that the second query can be resolved using the updated knowledge base, and completing the second subquery using the updated knowledge base.

[0007] According to another feature, a computer program product for implementing on-demand extraction and integration includes a computer-readable recording medium having program instructions materialized therein, this computer-readable recording medium is not transient itself, and the program instructions are executable by a processor, causing the processor to: receive a query in natural language format by the processor; derive multiple subqueries from the query by the processor; determine that the first subquery of the multiple subqueries cannot be resolved using a curated knowledge base by the processor; generate additional search terms for the first subquery using query relaxation by the processor; and the first sub The process involves querying a search index for unstructured text using a query and additional search terms; the processor receiving search results in response to the query; the processor generating new subject / predicate / object triples based on the search results; the processor completing the first subquery using the new subject / predicate / object triples; the processor updating the curated knowledge base to include the completed first subquery; the processor determining that the second query can be resolved using the updated knowledge base; and the processor executing a method to complete the second subquery using the updated curated knowledge base.

[0008] Other features and embodiments of the present invention will become apparent from the detailed description to be provided below, where, in combination with the drawings, the drawings are provided for the purpose of illustrating embodiments of the principles of the present invention. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows a network architecture according to one embodiment of the present invention. [Figure 2]Figure 2 shows a typical hardware environment associated with the server, client, or both of the servers and clients shown in Figure 1 according to one embodiment of the present invention. [Figure 3] Figure 3 shows a method for query resolution using structured and unstructured data according to one embodiment of the present invention. [Figure 4] Figure 4 shows a method for implementing on-demand extraction and integration according to one embodiment of the present invention. [Figure 5] Figure 5 shows an exemplary question-and-answer (QA) system that integrates unstructured and structured data according to one embodiment of the present invention. [Modes for carrying out the invention]

[0010] The following description is provided to illustrate the general principles of the present invention and is not intended to limit the inventive concepts of the claims herein. Furthermore, certain features described herein can be used in combination with other described features in each of the various possible combinations and substitutions.

[0011] Unless otherwise specified herein, all terms are given the broadest and most possible interpretation, including the meanings understood by those skilled in the art, or defined in dictionaries, papers, etc., or combinations thereof, as well as the meanings implied by the specification.

[0012] It should also be noted that the terms used herein and in the appended claims are for the purpose of describing only specific embodiments and are not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are also intended to include plural forms unless the context clearly indicates otherwise. Furthermore, the terms “contains” and “contains,” as used herein, identify the presence of the mentioned features, integers, steps, operations, elements, and components or combinations thereof, but do not exclude one or more other features, integers, steps, operations, elements, components, and groups thereof or combinations thereof.

[0013] The following description discloses several preferred embodiments of systems, methods, and computer program products for resolving queries using structured and unstructured data.

[0014] In one general embodiment, a computer implementation method includes receiving a question in natural language format, deriving several subqueries from the question, determining that the first subquery of the several subqueries cannot be resolved using a curated knowledge base, resolving the first subquery using unstructured text, and adapting the curated knowledge base using the resolved first subquery.

[0015] In another common embodiment, a computer program product for resolving queries using structured and unstructured data includes a computer-readable recording medium having program instructions materialized therein, wherein the computer-readable recording medium is not a transient signal in itself, and the program instructions are executable by a processor, causing the processor to: receive a query in natural language format; derive multiple subqueries from the query; determine that the first subquery of the multiple subqueries cannot be resolved using a curated knowledge base; resolve the first subquery using unstructured text; and adapt the curated knowledge base using the resolved first subquery.

[0016] In another general characteristic, the system includes a processor and logic that is logically integrated with the processor and executable by the processor, which is configured to receive a question in natural language format, derive multiple subqueries from the question, determine that the first subquery of the multiple subqueries cannot be resolved using a curated knowledge base, resolve the first subquery using unstructured text, and adapt the curated knowledge base using the resolved first subquery.

[0017] In another common embodiment, a computer implementation method includes receiving a question in natural language format, deriving a first subquery and a second subquery from the question, determining that the first subquery cannot be resolved using a curated knowledge base, generating additional search terms for the first subquery using query relaxation, querying a search index for unstructured text using the first subquery and the additional search terms, receiving search results in response to the query of the search index, generating new subject / predicate / object triples based on the search results, completing the first subquery using the new subject / predicate / object triples, updating the curated knowledge base to include the completed first subquery, determining that the second query can be resolved using the updated curated knowledge base, and completing the second subquery using the updated curated knowledge base.

[0018] In another common embodiment, a computer program product for implementing on-demand extraction and integration includes a computer-readable recording medium having program instructions materialized therein, wherein the computer-readable recording medium is not transient itself, and the program instructions are executable by a processor, causing the processor to: receive a query in natural language format by the processor; derive a first subquery and a second subquery from the query by the processor; determine that the first subquery cannot be resolved using a curated knowledge base by the processor; generate additional search terms for the first subquery using query relaxation by the processor; and The process involves querying the search index for unstructured text using the search terms and additional search terms, having the processor receive search results in response to the query, having the processor generate three new subjects / predicates / objects based on the search results, having the processor complete the first subquery using the three new subjects / predicates / objects, having the processor update the curated knowledge base to include the completed first subquery, having the processor determine that the second query can be resolved using the updated curated knowledge base, and having the processor complete the second subquery using the updated curated knowledge base.

[0019] Figure 1 shows an architecture 100 in one embodiment. As shown in Figure 1, a plurality of remote networks 102 are provided, including a first remote network 104 and a second remote network 106. A gateway 101 can connect the remote networks 102 with the neighboring network 108. In the context of this architecture 100, networks 104 and 106 can take any form, including but not limited to LANs, WANs such as the Internet, public switched telephone networks (PSTNs), and internal telephone networks.

[0020] In use, gateway 101 provides an entry point from remote network 102 to neighboring network 108. In this way, gateway 101 can function as a router, and the router can manage a given data packet arriving at gateway 101, and the switch provides the actual path for a given packet in and out of gateway 101.

[0021] Furthermore, the system includes at least one data server 114 connected to the proximity network 108, which is accessible from multiple remote networks 102 via the gateway 101. It should be noted that the data server(s) 114 may include any type of computing device / groupware. Each data server 114 is coupled to multiple user devices 116. The user devices 116 may also be coupled directly through one of the networks 104, 106, or 108. Such user devices 116 may include desktop computers, laptop computers, handheld computers, printers, or any other type of logic. It should be noted that in one embodiment, user devices 111 may also be coupled directly to any network.

[0022] Peripheral device 120 or a series of multiple peripheral devices 120, such as a facsimile machine, a printer, a network or local or both storage units or systems, can be coupled to one or more networks 104, 106, 108. It should be noted that a database or additional components or both can be used with and integrated with any type of network element coupled to networks 104, 106, 108. In the context of this specification, a network element can also refer to a component whether or not it is part of a network.

[0023] By some approaches, the methods and systems described herein can be implemented with or on or both a virtualization system or a system that emulates one or more systems such as a UNIX (registered trademark) system that emulates an IBM zOS environment, a UNIX (registered trademark) system that virtually hosts a MICROSOFT WINDOWS (registered trademark) environment, a MICROSOFT WINDOWS (registered trademark) system that emulates an IBM z / OS environment. This virtualization or emulation or both can be enhanced through VMWARE (registered trademark) in some embodiments.

[0024] In a further approach, one or more networks 104, 106, 108 can represent a cluster of systems commonly referred to as a “cloud”. In cloud computing, shared resources such as processing power, peripheral devices, software, data, servers, etc. are provided on an on-demand basis to any system within the cloud, thereby enabling access to and distribution of services across a number of computing systems. Cloud computing typically includes an Internet connection between systems operating within the cloud, but other technologies for coupling systems can also be used.

[0025] FIG. 2 shows a representative hardware environment associated with the user device 116 or the server 114 or both in FIG. 1. FIG. 2 shows a typical hardware configuration of a workstation including a central processing unit 210 such as a microprocessor and a number of other units interconnected via a system bus 212.

[0026] The workstation shown in FIG. 2 includes an I / O adapter 218 that couples peripheral devices such as a random access memory (RAM) 214, a read only memory (ROM) 216, and a disk storage unit 220 to the bus 212, a user interface adapter 222 for coupling other user interface devices such as a keyboard 224, a mouse 226, a speaker 228, a microphone 232 or a touch screen and a digital camera (not shown) to the bus 212, a communication adapter 234 for coupling the workstation to a communication network 235 (e.g., a data processing network), and a display adapter 236 for coupling the bus 212 to a display device 238.

[0027] The workstation has thereon a storage of an operating system such as the MICROSOFT WINDOWS (registered trademark) operating system, MAC OS (registered trademark), UNIX (registered trademark) OS, etc. Preferred embodiments can also be implemented on platforms and operating systems not described above. Preferred embodiments can be described using an object-oriented programming methodology with XML, C, or C++ language or combinations thereof, or other programming languages.

[0028] Referring now to Figure 3, a flowchart of Method 300 according to one embodiment is shown. Method 300 can be carried out according to the present invention in any of the environments illustrated in Figures 1-2 and 5, among other various embodiments. Naturally, Method 300 may include more or fewer operations than those specifically described in Figure 3, as will be understood by those skilled in the art who have read this specification.

[0029] Each step of Method 300 can be implemented by any suitable component of an operating environment. For example, in various embodiments, Method 300 can be performed in part or in whole by one or more servers, computers, or by several other devices having one or more processors internally. For example, a processor having processing circuits (may be multiple), chips (may be multiple), or modules (may be multiple) and preferably at least one hardware component implemented in hardware, software, or both, can be used internally in any device to perform one or more steps of Method 300. Exemplary processors include, but are not limited to, central processing units (CPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), combinations thereof, or any other suitable computing devices known in the Art.

[0030] As shown in Figure 3, Method 300 can be initiated with Operation 302, where the question is received in natural language format. In one embodiment, the question is received by a question-and-answer (QA) application. In another embodiment, the question may include a query to be answered by an office automation (OA) application. In yet another embodiment, the natural language format may include a format used in everyday human conversation.

[0031] In addition, in one embodiment, the natural language format can be distinguished from formal languages ​​that use program code, etc. In another embodiment, an exemplary question in the natural language format may include the question "What were the earnings of Company X's competitors in 2018?"

[0032] Furthermore, method 300 can proceed to operation 304, in which multiple subqueries are derived from the question. In one embodiment, subqueries can be derived from the question by parsing the natural language format of the question into a logical form. For example, the question can be parsed to identify one or more components of the question. In another embodiment, one or more components can be analyzed to generate one or more subqueries of the question.

[0033] Furthermore, in one embodiment, each subquery may contain n-fold tuples such as triples (e.g., subject / predicate / object triples). For example, an incomplete triple may lack data for one or more of the subject, predicate, and object fields. In another embodiment, the subject / predicate / object triple may contain a data object containing the subject, predicate, and object, which is populated based on the result of parsing the query. For example, for the query "What were the earnings of Company X's competitors in 2018?", the derived subject / predicate / object triple may include the following incomplete triples: a first subquery (?y "Earnings" ?x) and a second subquery (?x, "Competitors", "Company X").

[0034] Method 300 can then proceed to operation 306, where it is determined that the first subquery of a set of subqueries cannot be resolved using the curated knowledge base. In one embodiment, the curated knowledge base may include data for identifying a given domain collected from unstructured text. For example, unstructured text (e.g., free text) may be parsed (manually, automatically, or both) to determine text data (e.g., words, sentences, etc.) that match a given domain. In another embodiment, the curated knowledge base may include multiple complete triplets.

[0035] In addition, in one embodiment, a given domain may include one or more predetermined topics (e.g., company financial information) for the curated knowledge base. In another embodiment, text data matching a given domain may be added to the curated knowledge base for that domain. In yet another embodiment, the curated knowledge base may include multiple complete triples.

[0036] Furthermore, in one embodiment, the first subquery of a set of subqueries may not be solvable by a QA application using a curated knowledge base. The first subquery of a set of subqueries may contain an incomplete triple that cannot be completed by comparing the incomplete triple with the curated knowledge base. In another embodiment, the second subquery of a set of subqueries may be solvable by a QA application using a curated knowledge base. For example, the second subquery of a set of subqueries may contain an incomplete triple that can be completed by comparing the incomplete triple with the curated knowledge base. In yet another embodiment, the first subquery may be determined to be unsolvable using the curated knowledge base based on an analysis of the semantic relevance of the first subquery to the curated knowledge base.

[0037] Furthermore, method 300 can proceed to operation 308, in which the first subquery is resolved using unstructured text. In one embodiment, the unstructured text may include a text corpus since the curated knowledge base was generated. In another embodiment, the unstructured text may include historical text documents (e.g., text files, HTML, etc.) that have no given structure at all. In yet another embodiment, the first subquery may include an incomplete triple. In one embodiment, the first subquery may be resolved using unstructured text in response to a determination that the first subquery cannot be resolved using the curated knowledge base.

[0038] In one embodiment, the resolution of the first subquery may include querying a previously generated search index using unstructured text. For example, the unstructured text may be preprocessed (e.g., parsed and analyzed) to generate the search index. In another embodiment, the search index may include a keyword-based search engine generated for unstructured text. Exemplarily, the search engine may search using words found within the incomplete triple of the first subquery.

[0039] In addition, in one embodiment, query relaxation can be used to generate additional words to include in the query for the search index. For example, performing query relaxation could include determining synonyms (e.g., similar words) for words in an incomplete triple within a first subquery. In another embodiment, the query for the search index could be modified to include these synonyms in addition to the words in the incomplete triple.

[0040] Furthermore, in one embodiment, resolving the first subquery may include receiving the results of a search index query. For example, the results may contain one or more fragments of interest. In another embodiment, each of the one or more text fragments may contain one or more parts of text (e.g., words, phrases, sentences, paragraphs, etc.) that match the search index query. In yet another embodiment, one or more text fragments may be ranked according to one or more criteria (e.g., relevance to the search index query).

[0041] Furthermore, in one embodiment, resolving the first subquery involves generating new triples using the results of the search index query. For example, new subject / predicate / object triples can be generated using one or more text fragments received as a result of the search index query. In another embodiment, these new triples may be complete (for example, they may have values ​​for the subject, predicate, and object fields).

[0042] In one embodiment, resolving the first subquery may include completing the first subquery using a new triplet. For example, the first subquery may contain an incomplete triple. In another embodiment, the incomplete triple in the first subquery may be completed based on a new triple. In yet another embodiment, one or more sort-based answer identification and similar operations are performed on the incomplete triple in the first subquery based on the new triplet to determine values ​​for missing fields or incomplete triples.

[0043] In one embodiment, the unstructured data can be periodically updated to include information that was previously searched but not found within the unstructured data. For example, if the amount of text fragments returned in response to a search index query falls below a predetermined threshold, the search is performed on additional unstructured data containing results deemed more relevant to the search index query. This additional unstructured data is incorporated into the current unstructured data to augment the corpus of unstructured data.

[0044] In addition, method 300 proceeds to operation 310, where the curated knowledge base is adapted using the resolved first subquery. In one embodiment, adapting the curated knowledge base involves adding the resolved first subquery to the curated knowledge base. In another embodiment, the resolved first subquery may include a completed triple that will be added to an existing completed triple in the curated knowledge base based on a determination of the dependency between the resolved first subquery and an existing triple in the curated knowledge base.

[0045] Furthermore, in one embodiment, a second subquery of multiple subqueries can be resolved using an adapted curated knowledge base. For example, resolving a second subquery may include completing the second subquery using a triple in the adapted curated knowledge base. In another embodiment, a second subquery of multiple subqueries may include an incomplete triple that is compared to a completed triple in the adapted curated knowledge base (e.g., including the resolved first completed triple).

[0046] Furthermore, in one embodiment, one or more alignment-based answer identification and similarity operations can be performed on the incomplete triple of a second subquery based on the completed triple in the adapted curated knowledge base to determine the values ​​for the fields of the missing or incomplete triple. In another embodiment, the values ​​for the fields of the missing or incomplete triple of the second subquery can be added to the incomplete triple to generate a complete triple (thus resolving the second subquery).

[0047] In one embodiment, the resolved first subquery and the resolved second subquery can be used to generate an answer to a received question. For example, the information in the triple fields of the resolved first subquery and the resolved second subquery can be converted into a response in natural language format. In another embodiment, the response can be returned in natural language format.

[0048] In this method, both unstructured data (unstructured text) and structured data (curated knowledge base) can be used together by the QA system to answer questions. In addition, the curated knowledge base can be dynamically augmented using the results of unstructured text searches and analyses. This improves the validity of the curated knowledge base, which can improve the accuracy and performance of future subquery searches performed using the curated knowledge base. Due to the improved validity of the curated knowledge base, computing devices using the curated knowledge base can also benefit from improved performance.

[0049] Referring here to Figure 4, a flowchart of Method 400 for implementing on-demand extraction and integration according to one embodiment is shown. Method 400 can be carried out according to the present invention in any environment illustrated in Figures 1-2 and 5, among other various embodiments. Naturally, more or less operations than those specifically described in Figure 4 may be included in Method 400, as will be understood by those skilled in the art who have read this specification.

[0050] Each step of Method 400 can be performed by any suitable component of an operating environment. For example, in various embodiments, Method 400 can be performed in part or in whole by one or more servers, computers, or several other devices having one or more processors inside them. For example, a processor having processing circuits (may be multiple), chips (may be multiple), or modules (may be multiple) and preferably at least one hardware component implemented in hardware, software, or both, can be used inside any device to perform one or more steps of Method 400. Exemplary processors include, but are not limited to, central processing units (CPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), combinations thereof, or any other suitable computing devices known in the Art.

[0051] As shown in Figure 4, Method 400 can begin with Operation 402, where the question is received in natural language format. Additionally, Method 400 proceeds to Operation 404, where a first subquery and a second subquery are derived from the question. In one embodiment, both the first and second subqueries contain incomplete subject / predicate / object triples. In another embodiment, the first subquery differs from the second subquery. For example, the first subquery may have different triple field values ​​compared to the second subquery.

[0052] Additionally, method 400 may proceed to operation 406, where it is determined that the first subquery cannot be resolved using the curated knowledge base based on unstructured text. Further, method 400 may proceed to operation 408, where query relaxation is used to generate additional search terms for the first subquery. Further, method 400 may proceed to operation 410, where the search index for the unstructured text is queried using the first subquery and additional search terms. In one embodiment, the search index may be one previously generated using the unstructured text. In another embodiment, the search index for the unstructured text may be queried in response to the determination that the first subquery cannot be resolved using the curated knowledge base.

[0053] Method 400 then proceeds to operation 412, where the search results are received in response to the query of the search index. In addition, Method 400 proceeds to operation 414, where new subject / predicate / object triples are generated based on the search results. For example, the search results may contain multiple related fragments, and open information extraction can be used to extract new triples from the fragments. In another embodiment, the new triples may be ranked according to their semantic relevance to the first and second subqueries, and a predetermined number of top-ranked triples are selected as new triples. Furthermore, Method 400 proceeds to operation 416, where the first subquery is completed using the new subject / predicate / object triples. Furthermore, Method 400 proceeds to operation 418, where the curated knowledge base is updated to include the completed first subquery.

[0054] Method 400 then proceeds to operation 420, where it is determined that the second subquery can be resolved using the updated curated knowledge base. In one embodiment, the second subquery is deferred until the first subquery is completed. For example, the second subquery may be determined in accordance with the first subquery based on an analysis of the incomplete triples of the first and second subqueries. For example, fields that are not in the second subquery may be located in the first subquery, and it may be determined that the fields that are not in the second subquery can be completed when the first subquery is completed. In another embodiment, the second subquery may be deferred in response to this determination until the first subquery is completed.

[0055] Additionally, method 400 can proceed to operation 422, where a second subquery is completed using the updated curated knowledge base. In one embodiment, the second subquery may be completed in response to the determination that the first subquery has been completed and added to the curated knowledge base. In another embodiment, the second subquery is resolved using a subject / predicate / object triplet in the updated curated knowledge base that contains a triple of the completed first subquery.

[0056] In this method, searching both curated knowledge bases and unstructured texts can be performed to enhance the curated knowledge base and accurately respond to the questions that have been resolved.

[0057] Figure 5 shows a question-and-answer (QA) system 500 integrating unstructured and structured data according to an exemplary embodiment. The QA system 500 answers questions by integrating information in real time from both unstructured data (free text) and structured data (curated knowledge base 502).

[0058] As illustrated, the input to the system is a natural language question Q504. Question Q504 is parsed by a natural language understanding unit 506, which converts the question into a machine-understandable subquery 508. For example, the subquery 508 may contain an incomplete subject / predicate / object triple.

[0059] Additionally, for each subquery q_x of subquery 508, the query plan generator 510 performs query expansion by expanding q_x using a set of similar queries. For example, if q_x is: (Company, acquisition,?) It can include...

[0060] Query extension then, (Company, acquisition,?) and (?, acquired by, company) This also applies to estimated queries.

[0061] In one embodiment, the query extension described above can be completed using one or more paraphrasing models. String-similarity matching, embedding (e.g., pre-trained word2vec embeddings) matching, and paraphrase-based matching can then be used to determine which data source (e.g., unstructured text 512 or curated knowledge base 502) each subquery 508 should be sent to. Based on the determined data source, the subquery 508 is split into two subsets—a free-text subquery 514 and a curated knowledge base subquery 516.

[0062] Furthermore, the free-text subquery 514 is sent to the free-text query module 518, which searches the indexed text corpus 520 and identifies the collection of relevant text fragments 522 that may contain answers to the free-text subquery 514. The unstructured text 512 is indexed 524 to generate the text corpus 520, which is completed using indexing techniques (e.g., methods based on TD-IF) to quickly identify and retrieve relevant text fragments 522. When searching the indexed corpus 520, the free-text subquery 514 can be expanded using a paraphrase model to increase its comprehensiveness.

[0063] Furthermore, the relevant text fragments 522 are passed to an information extractor 526 (such as a relaxation extractor) to extract candidate answers to each free-text subquery 514 (thus completing each free-text subquery 514) and rank them based on their relevance scores.

[0064] Furthermore, all or some of the completed free-text subqueries 514 are subsequently added to the curated knowledge base 502 to update the curated knowledge base 502. In one embodiment, all or some of the completed free-text subqueries 514 are integrated with a completed curated knowledge base subquery 516 (completed, by definition, for example, by a query module 528 against the curated knowledge base using the curated knowledge base 502) to determine the final answer provided in response to the natural language question Q504. In another embodiment, one or more curated knowledge base subqueries 516 can be completed by a query module 528 against a curated knowledge base using an updated curated knowledge base 502, and the completed knowledge base subqueries 516 can then be used to determine the final answer to be provided in response to a natural language question Q504.

[0065] In another embodiment, if there are aggregate subqueries identified from the natural language question Q504, one or more aggregations can be performed.

[0066] On-demand extraction and integration Knowledge bases (KBs) have become the core of applications such as question-and-answer (QA) applications, which are centered around large amounts of data. Large knowledge bases store vast amounts of knowledge about different entities, enabling systems to provide answers to entity-centric questions. Among these knowledge bases, curated KBs enable systems to provide reasoned and accurate answers based on highly accurate knowledge.

[0067] Knowledge bases are written and curated by domain experts who are required to propose domain-specific schemas and curation workflows, enabling the extraction of facts affirmed by the schema. Even with high accuracy, these KBs can be incomplete because they assume that the world's knowledge can be encoded using selected linguistic predicates. Often, the lack of information from KBs exists in the unstructured text on which they are based.

[0068] For example, a KB might include public information about a company and some of its financial criteria, but may not include relationships such as acquirers, competitors, and subsidiaries existing between the companies. This information can be curated and completed from text using information technology, where available, and supplemented with information not present in the KBs. This can help generalize QA systems that encounter incompleteness in curated KBs. Domain experts may ask questions to automatically retrieve curated text and use insights from system-provided answers to further improve the curation workflow.

[0069] As an example, a domain expert might try to build a Knowledge Base (KB) containing the exact financial standards of public companies from tables within those companies' financial reports (e.g., 10-k reports). However, the queries that users are interested in would look similar. "Show me the earnings of Company X's competitors in 2016."

[0070] This query cannot be supported solely from the curated KB and can be broken down into two subqueries: (1) (Revenue from ?y, ?x), and (2) (?x, "Competitor", "Company X"), Here, query (1) can be answered by a curated knowledge base as soon as we know the answer to query (2), and this answer can be found in the free text of company X's financial report (assuming the report actually contains the information of interest).

[0071] As a result, in one embodiment, heterogeneous data sources, structured knowledge bases (KBs), and unstructured text can be combined to build a dynamic hybrid knowledge base that contributes to the accuracy of structured KBs and the comprehensiveness of facts from unstructured text. Given a natural language question or interpretation of a natural language question, the system as a whole can selectively query unstructured text or structured KBs, extract and integrate information as needed, and integrate the results to answer the query.

[0072] Exemplary features include: 1. When a local representative of a given natural language query (NLQ) is referenced as a logical query, a corresponding query plan is generated. This plan consists of query operators that query existing knowledge bases and unstructured data to generate intermediate results, combine the intermediate answers, and aggregate the final result. 2. For query operators that access unstructured data, relevant fragments related to the operator are retrieved from the unstructured data. a. Facts are extracted and ranked from relevant fragments. b. The results for the inquiry operation entered are determined from the extracted facts. It will be done. 3. For query operators that have accessed the curated knowledge base, queries are made against the knowledge base to retrieve relevant facts. 4. The final result for the original query plan is generated.

[0073] One exemplary approach is to answer queries by presenting free text in an on-demand format to find missing information. This approach supports more queries compared to a static knowledge base approach. In addition, missing information is extracted from the free text in an on-demand format. Furthermore, this approach breaks down complex queries into numerous subqueries, allowing for more reliable answers from curated or extracted KBs if the relevant information is not found in the curated KB.

[0074] Before answering any question, the system requires a one-time preprocessing step. In the offline process, the system generates an inverted index across multiple text corpora, enabling more efficient keyword searching during queries.

[0075] The system generates an inverted index through offline processing, enabling more efficient keyword searches across multiple text corpora. In addition, several metadata elements, such as year, namespace, and entities associated with the text, are indexed.

[0076] The system input is a logical query (e.g., one interpretation of a question in natural language). The logical query is a combination of many closes, where each close is an n-fold tuple with one or more variable summaries. The substantial plan generator translates the logical query to produce a substantial plan for the query. The substantial plan determines the execution order of the closes and whether it is necessary to execute the closes against a curated KB or free text. In addition, the substantial plan determines how to combine and optionally aggregate intermediate results.

[0077] When a query close needs to be executed against free text, the query close is first rewritten using metadata to generate a more flexible query plan. This helps in searching for text fragments that might not represent the information of interest. Each of the flexible query plans is then paraphrased to handle the numerous ways the same fact might be expressed within the free text. The query plans are ranked based on their semantic similarity to the original query. The top k plans are optimized, and queries for common keywords across the query plans are identified and executed in one go.

[0078] Keyword queries are issued against an indexed text corpus to search for relevant text fragments. An open-domain information extractor extracts factual triples (subject-predicate-object) from the text fragments. Because these triples are noisy and unnormalized, the system further processes and annotates them to help accurately identify answers. The triples are aggregated across different query plans and ranked based on their estimated reliability. The system then identifies answers from the triples using a sort-based answer identification algorithm. These answers are further defummed and returned to the plan generator for further use.

[0079] When a query close needs to be executed against a curated KB, it must first be translated into a query in a high-level declarative language. The Ontology Query Language (OQL) can be used to access information stored in a curated KB. Query decomposition breaks down a query into a number of simple closes, so a manually created template can be used to translate the closes into OQL queries. The overlap of schema element information between the logical query and OQL can be used to select a template for querying the curated KB. The selected template is instantiated, the corresponding OQL query is issued against the KB, and a set of tuples is returned.

[0080] In one embodiment, heterogeneous data sources may include structured KBs, which have high-accuracy real-world facts and free text, with the free text comprising a vast repository of sentences capturing the facts within the KBs. The hybrid system benefits from the accuracy of curated KBs and the comprehensiveness of facts in text format, and helps domain experts identify predicates not present in curated KBs.

[0081] In addition, in one embodiment, keyword queries can be routed to a superset of relevant documents using metadata about query relaxation and rewriting. Furthermore, paraphrasing models can be used to extend queries.

[0082] The present invention may be a system, method, or computer program product or a combination thereof, at any possible level of technical detail integration. The computer program product may include a recording medium (or a set of media) having computer-readable program instructions thereon for causing a processor to perform the features of the present invention.

[0083] A computer-readable recording medium can be a tangible device capable of holding and storing multiple instructions for use by an instruction execution device. A computer-readable medium can be, for example, but is not limited to, an electrical recording device, a magnetic recording device, an optical recording device, an electromagnetoelectric recording device, a semiconductor recording device, or any preferred combination thereof. More specific embodiments of a computer-readable recording medium include portable computer disks, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory®), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disk (DVD), memory stick, floppy disk®, punch cards, or mechanically encoded devices having structures protruding into grooves on which instructions are recorded, and any preferred combination thereof. As used herein, computer-readable recording media are not interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, waveguides or other communication media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0084] The computer program instructions described herein can be downloaded from a computer-readable recording medium to each computing / processing device, or downloaded to an external computer or external recording device via a network such as the Internet, a local area network, a wide area network, or a wireless network or a combination thereof. The network may include copper communication cables, optical communication fibers, wireless communication routers, firewalls, switches, gateway computers and edge servers or a combination thereof. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and transfers these computer-readable program instructions to a computer-readable recording medium within the respective computing / processing device for storage.

[0085] Computer-readable program instructions for performing the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of programming languages, including one or more object-oriented programming languages ​​such as Smalltalk®, C++, the “C” programming language, or similar programming languages. Computer-readable program instructions may be executed entirely on a user computer, partially as a standalone software package on a user computer, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., through an Internet service provider). In some embodiments, an electrical circuit including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions, and the electrical circuit can be personalized to perform features of the present invention using computer-readable program instruction state information.

[0086] Embodiments of the present invention described herein have been described in accordance with embodiments of the present invention, with reference to flowchart instructions and block diagrams of methods, or both, apparatus (systems), and computer-readable recording media and computer programs. It should be understood that any combination of flowchart illustrations and block diagrams, or both, and blocks and block diagrams in flowchart illustrations, or both, can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions can be provided to a general-purpose computer, a specific-purpose computer, or other programmable data processing device for generating other processors or machines, which, when executed by the computer's processor or other programmable data processing device, generate means for implementing functions / operations specified by blocks or combinations thereof in flowcharts and block diagrams. These computer-readable program instructions, which direct computers, programmable data processing devices, and other devices or combinations thereof to function in a specific manner, can also be stored in a computer-readable recording medium, and a computer-readable recording medium containing the instructions constitutes a product containing instructions that implement the features of functions / operations specified by blocks or combinations thereof in flowcharts and block diagrams.

[0088] Computer-readable program instructions are also loaded onto a computer, other programmable data processing device, or other device, causing a computer implementation process for a series of operation steps on the computer, other programmable device, or other device, thereby implementing functions / operations specified by blocks or multiple blocks or combinations thereof in flowcharts and block diagrams on the computer, other programmable device, or other device.

[0089] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and possible implementation operations of systems, methods, and computer programs according to various embodiments of the present invention. In this view, a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a particular logical function (or multiple functions). In some alternative implementations, the functions described in a block may be performed in ways other than those illustrated. For example, two consecutively shown blocks may be performed practically as a single step, simultaneously, substantially simultaneously, partially or completely superimposed, depending on the functions they contain, or multiple blocks may sometimes be executed in reverse order. It should also be noted that the illustration of block diagrams and flowcharts, or both thereof, and the illustration of blocks in block diagrams and flowcharts, or a combination thereof, may be implemented by systems based on hardware for a particular purpose and hardware for a particular purpose that performs a particular function or operation or executes a particular computer instruction.

[0090] Furthermore, systems according to various embodiments may include a processor and logic integrated with or executable by the processor, or both, which is configured to perform one or more process steps described herein. The processor can be any configuration, including a discrete processor or many components such as processing hardware, memory, and I / O interfaces, as described herein. "Integrated" means that the processor has logic embedded within it as hardware logic, such as an application-specific integrated circuit (ASIC) or FPGA. "Executable by the processor" means that the logic is hardware logic such as firmware, software logic, parts of an operating system, parts of an application program, or software logic that is accessible by both hardware and the processor and configured to cause the processor to perform certain functions during execution by the processor. The software logic can be stored in any form of memory, local, remote, or both. Any processor known in this art, such as a software processor module or hardware processors such as an ASIC, FPGA, central processing unit (CPU), integrated circuit (IC), or graphics processing unit (GPU), can be used.

[0091] The systems, methodologies, or both described above can be combined in any way, and it will be clear that multiple combinations can be generated from the explanations presented above.

[0092] Furthermore, embodiments of the present invention can be provided in a form that is deployed on behalf of the customer and provides services on demand.

[0093] The descriptions of the various embodiments of this disclosure are presented for illustrative purposes only and are not intended to be exclusive or limiting to the disclosed embodiments. Many modifications or variations will be obvious to those skilled in the art without departing from the scope and spirit of this disclosure. The terms used herein have been chosen to best describe the principles, practical applications, or technological improvements beyond the technology available on the market of these embodiments, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method comprising: receiving queries in natural language format; deriving a plurality of sub-queries from said query; determining that a first subquery of the plurality of subqueries cannot be resolved using a curated knowledge base; Based on the first sub-query derived from the received question, resolving the first sub-query having incomplete tuples using a search index generated by unstructured text; and adapting the curated knowledge base by appending the resolved first subquery having a complete tuple.

4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the question is received by a question and answer (QA) application.

3. 3. The computer-implemented method of claim 1, wherein each of the plurality of sub-queries is derived by parsing a natural language format of the question into a logical form.

4. 4. The computer-implemented method of claim 1, wherein the incomplete tuples include incomplete subject / predicate / object triples and the complete tuples include complete subject / predicate / object triples.

5. The computer-implemented method of any one of claims 1 to 4, wherein the unstructured text comprises a text corpus, and the curated knowledge base is generated from the text corpus.

6. The computer-implemented method of any one of claims 1 to 5, wherein query relaxation is used to generate additional terms for inclusion in queries of the search index.

7. The computer-implemented method of any one of claims 1 to 6, wherein resolving the first sub-query comprises querying the search index and receiving results of the query.

8. 8. The computer-implemented method of claim 7, wherein resolving the first subquery comprises generating new triples using results of querying the search index.

9. 9. The computer-implemented method of claim 8, wherein resolving the first subquery comprises completing the first subquery using the new triple.

10. The computer-implemented method of any one of claims 1 to 9, further comprising resolving a second sub-query of the plurality of sub-queries by using the adapted curated knowledge base.

11. 11. The computer-implemented method of claim 10, wherein the resolved first sub-query and the resolved second sub-query are used to generate an answer to the received question.

12. The computer-implemented method of any one of claims 1 to 11, wherein the incomplete tuples are incomplete tuples that cannot be completed by comparison with the curated knowledge base.

13. A computer program product capable of executing the computer-implemented method according to any one of claims 1 to 12.

14. 1. A system comprising: A processor; logic integrated into, executable by or integrated into said processor, said logic being configured to perform a computer-implemented method according to any one of claims 1 to 12; A system including:

15. 1. A computer-implemented method comprising: receiving queries in natural language format; deriving a first sub-query and a second sub-query from the query; determining that the first sub-query cannot be resolved using a curated knowledge base generated based on unstructured text; generating additional search terms for the first sub-query using query relaxation; querying a search index generated from the unstructured text using the first sub-query derived from the received question and additional of the search terms; receiving search results in response to querying the search index; generating new subject / predicate / object triples based on the search results; completing the first subquery having an incomplete subject / predicate / object triple using the new subject / predicate / object triple; updating the curated knowledge base by adding the completed first subquery having a complete subject / predicate / object triple; determining that the second sub-query can be resolved using the updated curated knowledge base; and completing the second sub-query using the updated curated knowledge base.

23. A computer-implemented method comprising:

16. 16. The computer-implemented method of claim 15, wherein the unstructured text comprises a text corpus, and the curated knowledge base is generated from the text corpus.

17. 17. The computer-implemented method of claim 15 or claim 16, wherein the incomplete triples are incomplete triples that cannot be completed by comparison to the curated knowledge base.

18. 1. A computer-implemented method comprising: receiving a natural language input corresponding to a query with a natural language parser that reduces the natural language input to subject / predicate / object (SPO) triples; directing a first subset of SPO triples having incomplete SPO triples to a query module for free text and a second subset of SPO triples different from the first subset to a query module for a curated knowledge base, taking into account semantic relevance to a curated knowledge base; accessing, via the free text query module, the indexed text data to identify fragments of interest in the first subset of the SPO triples; extracting SPO triples from the fragment of interest that correspond to the first subset of SPO triples; ranking the extracted SPO triples according to their semantic relevance to the first subset of SPO triples; selecting a predetermined number of SPO triples from the top ranked and extracted SPO triples; and combining the selected number of SPO triples into a second subset of SPO triples to form an answer to the question; the computer-implemented method further comprising: The computer-implemented method further comprising completing the first subset having the incomplete SPO triples based on the fragment of interest and adding the completed first subset having complete SPO triples to the curated knowledge base.

19. 20. The computer-implemented method of claim 18, wherein the combining is performed in terms of dependencies generated by the natural language parser between a selected predetermined number of the SPO triples and the second subset of SPO triples.

Images