Extending the Knowledge Graph with External Data Sources
By selecting entities from the original knowledge graph and accessing external data sources to generate an extended knowledge graph, the limitations of traditional question-and-answer systems when utilizing external knowledge graphs are solved, and the accuracy of finding answers to natural language questions is improved.
Patent Information
- Application Number
- CN202010657144.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-07-10
- Filing Date
- 2020-07-09
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-07-09
AI Technical Summary
When dealing with natural language problems, traditional question-and-answer systems are limited by text data in paragraphs and cannot effectively utilize entities and relationships in external knowledge graphs, resulting in the potential usefulness of knowledge graphs being limited.
By selecting entities from the original knowledge graph, accessing external data sources, such as online encyclopedias, identifying new entities and relationships, generating extended knowledge graphs, and using vector spatial similarity calculations and knowledge graph inference, improve the score and ranking of candidate answers.
It improves the ability of the Q&A system to find correct answers, increases the accuracy and trustworthiness of the answers to submitted questions, and enhances the performance of traditional QA pipelines.
Smart Images

Figure CN112214583B_ABST
Abstract
Description
Background Art
[0001] In computer science, a knowledge graph represents a collection of interlinked descriptions of entities connected to each other by relationships (associations). An entity can be a real-world object, an event, a situation, or an abstract concept. A knowledge graph includes descriptions with a formal structure that allows computer processes to access them in an efficient and unambiguous way. Entity descriptions build on each other, forming a network in which each entity represents part of the description of entities related to it.
[0002] Knowledge graphs are used in conjunction with ontologies. Ontologies contain representations, formal naming, and definitions of categories, attributes, and relationships between concepts, data, and entities that substantiate one or more or all discourse domains. Each domain creates an ontology to limit complexity and organize information into data and knowledge. As new ontologies are created, their use is expected to improve problem solving within that domain.
[0003] As a broad term, knowledge graph is sometimes used as a synonym for ontology. A common explanation is that a knowledge graph represents a collection of interlinked descriptions of entities (real-world objects, events, situations, or abstract concepts). Unlike an ontology, a knowledge graph typically contains a large amount of factual information with less formal semantics. In some contexts, the term "knowledge graph" is used to refer to any knowledge base represented as a graph.
[0004] Question answering (QA) is a computer science discipline within the fields of information retrieval and natural language processing (NLP) that is concerned with building systems that answer questions posed by humans in natural language. A QA implementation (usually a computer program) can construct its answers by querying a structured database of knowledge or information (usually a knowledge base or "corpus"). A QA system can ingest data from an unstructured collection of natural language documents (e.g., documents found on the Internet). The data is ingested into the corpus of the QA system in a format that makes it more easily available to the QA system than having to search unstructured documents. Examples of natural language document collections that a QA system may ingest and use may include reference texts, organizational documents and web pages, newswire reports, online encyclopedia pages, and other data pages found on the Internet.
[0005] QA systems ingest large amounts of documents. These documents typically contain many paragraphs. When using a traditional QA pipeline to discover possible candidate answers to a submitted question, the pipeline identifies paragraphs that are found to be helpful in providing possible answers to the question. Paragraphs in traditional systems are limited to the text or data contained in the paragraph, and any knowledge graph resulting from a knowledge graph engine processing such paragraphs is limited to the entities and relations found in the corresponding paragraph, thus limiting the potential usefulness of the resulting knowledge graph. Summary of the invention
[0006] A method is provided for selecting an original entity from an original knowledge graph. The method then accesses a data source external to the original knowledge graph, such as an online encyclopedia. Entities in the data source are identified based on entities that match the original entities. Then, new relationships between the identified entities and new entities are identified in the data source, where the new entities are missing in the original knowledge graph. Then, an extended knowledge graph is generated, where the extended knowledge graph is formed by adding the new entities to the original knowledge graph.
[0007] The foregoing is an overview and therefore necessarily contains simplifications, generalizations and omissions of details; therefore, those skilled in the art will appreciate that this overview is illustrative only and is not intended to be limiting in any way. Other aspects, creative features and advantages of the present invention will be apparent in the non-limiting detailed description set forth below. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The present invention may be better understood, and its numerous objects, features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which:
[0009] Figure 1 Describe a network environment including a knowledge manager utilizing a knowledge base;
[0010] Figure 2 Such as Figure 1 A block diagram of the processor and components of the information processing system shown in;
[0011] Figure 3 is a component diagram illustrating various components included in a system for exploiting entity relationships to discover answers using a knowledge graph;
[0012] Figure 4 is an illustration of a flow chart showing logic for exploiting entity relationships to discover answers using a knowledge graph;
[0013] Figure 5 is an illustration of a flow chart showing logic for extending a knowledge graph using data from an external source;
[0014] Figure 6is an illustration of a flow chart showing logic for calculating similarity between knowledge graphs; and
[0015] Figure 7 is an illustration of a flow chart showing logic for scoring candidate answers (CAs), including CAs generated by utilizing entity relationships found in a knowledge graph. DETAILED DESCRIPTION
[0016] Figure 1-Figure 7 A method is described that utilizes entity relationship data from a knowledge graph (KG) and computes similarity scores to find missing information of entities and also improve the scores of candidate answers to better rank correct answers (reasonable / trustworthy answers). The method employs knowledge graph reasoning, which focuses on the analysis of the knowledge graph and finding the occurrence of entities in the graph. The method matches KG entities by using thresholds and computes candidate answer (CA) scores to improve CA in question answering systems.
[0017] In one embodiment, the method includes two phases: (1) a candidate answer generator phase, and (2) a candidate answer scorer phase. During the candidate answer generator phase, the method processes questions and passages from an existing QA pipeline through a knowledge graph database. The process extends the graph by adding neighbors to existing entities using common relationships, where neighbors are added from external data (e.g., an online encyclopedia) used to extend the graph. The method then calculates a vector space similarity score using a predefined threshold to decide whether the external data references the same active entity, and then generates a list of candidate answers.
[0018] During the candidate answer scorer stage, for each candidate answer generated from the previous stage, a knowledge graph (KG) score (KG score) is stored along with a KG Boolean value indicating whether the candidate answer already exists in the existing list of candidate answers generated by the traditional QA pipeline. In one embodiment, the final result is generated by combining the KG score and the KG Boolean value. This process results in the inclusion of new candidate answers that were not generated by the traditional QA pipeline, as well as improving the scores of candidate answers generated by the traditional QA pipeline that were also found by the KG analysis described herein. By using both candidate answers from the traditional QA pipeline and other data derived from the KG graph analysis, the method produces an improved QA system that is more likely to find the correct answer to the question submitted to the QA system.
[0019] In more detail, the candidate answer generator stage first creates a knowledge graph database from a corpus such as an online encyclopedia or other external knowledge base. In the created knowledge graph database, each node represents an entity, and the edge between nodes represents the relationship between two nodes / entities. When a question is submitted, the method extracts entities and relationships from the question text and creates a KG-like data structure that includes the missing entities or relationships from the question. For example, if the question submitted is: "Which president visited England to sign an environmental treaty?", the missing entity will be the name of the president.
[0020] The method runs the KG data through a knowledge graph database previously created using the method, and the method has the ability to expand the knowledge graph by adding adjacent entities using common relationships. The question also goes through a traditional QA, in which it generates a list of paragraphs (and later generates candidate answers from the list of paragraphs). Each of these paragraphs follows the same steps above and generates an extended graph. The method then compares the extended graph obtained from each paragraph with the extended graph of the question to calculate a similarity score based on the properties of the graph using a vector space model. Entities from paragraphs that match entities missing from the question are extracted as candidate answers. In one embodiment, when the similarity score of an entity is above a predefined threshold, the entity is added to the candidate answer list, indicating that the graphs are very similar.
[0021] In more detail, the candidate answer scorer process stores the similarity score for each candidate answer as a new feature / scorer: (KG score). The generated list of candidate answers is then compared to the list of candidate answers generated by the traditional QA pipeline. The process populates the value of another feature called "KG Boolean" which indicates whether a given candidate answer was found by both the traditional QA pipeline and the KG graph analysis process disclosed herein. In case of a match, the method sets the KG Boolean to true, otherwise it sets the KG Boolean to false.
[0022] The addition of these two features results in the addition of additional new candidate answers to the candidate answer list based on the knowledge graph database analysis, as well as improving the scores of candidate answers found by both the traditional QA pipeline method and the knowledge graph database analysis method. The inclusion of new candidate answers and the improvement of scores result in an improved set of scores for ranking the candidate answer list. The QA pipeline then continues with its remaining steps to select one or more candidate answers as the most likely answer to the question submitted to the QA system.
[0023] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0024] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0025] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0026] The computer program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, etc., and traditional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.
[0027] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.
[0028] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0029] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0030] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0031] Figure 1 A schematic diagram depicting one illustrative embodiment of a question / answer creation (QA) system 100 in a computer network 102. The QA system 100 may include a knowledge manager computing device 104 (including one or more processors and one or more memories, as well as any other computing device elements that may be well known in the art (including buses, storage devices, communication interfaces, etc.)) that connects the QA system 100 to the computer network 102. The network 102 may include multiple computing devices 104 that communicate with each other and with other devices or components via one or more wired and / or wireless data communication links, where each communication link may include one or more of wires, routers, switches, transmitters, receivers, etc. The QA system 100 and network 102 may enable question / answer (QA) generation functionality for one or more content users. Other embodiments of the QA system 100 may be used with components, systems, subsystems, and / or devices other than those depicted herein.
[0032] The QA system 100 can be configured to receive input from various sources. For example, the QA system 100 can receive input from a corpus of network 102, electronic document 107 or other data, content creators, content users and other possible input sources. In one embodiment, some or all inputs to the QA system 100 can be routed through the network 102. Various computing devices on the network 102 may include access points for content creators and content users. Some computing devices may include devices for storing databases of corpora of data. In various embodiments, the network 102 may include local network connections and remote connections so that the knowledge manager 100 can operate in an environment of any size including local and global (e.g., the Internet). In addition, the knowledge manager 100 is used as a front-end system that can make various knowledge extracted or represented from documents, network accessible sources and / or structured data sources available. In this way, some processing uses a knowledge manager that also includes an input interface to receive knowledge requests and responds accordingly to fill the knowledge manager.
[0033] In one embodiment, a content creator creates content in an electronic document 107 to be used as part of the data corpus of the QA system 100. The electronic document 107 may include any file, text, article, or data source for use in QA. Content users may access the QA system 100 via a network connection or an Internet connection to the network 102, and may input questions to the QA system 100 that can be answered by the content in the data corpus. As further described below, when a given portion of a document is evaluated for semantic content, the process may query it from a knowledge manager using various conventions. One convention is to send a well-formed question. Semantic content is content based on the relationship between indicators (e.g., words, phrases, signs, and symbols) and the meanings, denotations, or connotations they represent. In other words, semantic content is content that is expressed such as by using natural language (NL) processing to interpret expressions. Semantic data 108 is stored as part of the knowledge base 106. In one embodiment, the process sends well-formed questions (e.g., natural language questions, etc.) to the knowledge manager. The QA system 100 can interpret the question and provide a response containing one or more answers to the question to the content user. In some embodiments, the QA system 100 can provide the response to the user in a ranked list of answers.
[0034] In some illustrative embodiments, QA system 100 may be an IBM Watson® system available from International Business Machines Corporation in Armonk, New York. TM QA system enhanced with the mechanisms of the illustrative embodiments described below. IBM WatsonTM The knowledge manager system can receive an input question, parse the question to extract key features of the question, and then in turn use the key features to formulate a query that is applied to the data corpus. Based on applying the query to the data corpus, a set of candidate answers or hypotheses to the input question is generated by searching the data corpus for portions of the data corpus that have some potential to contain valuable responses to the input question.
[0035] Then, IBM Watson TM QA systems use a variety of reasoning algorithms to perform in-depth analysis of the language of the question entered and the language used in each portion of the data corpus found during the application of the query. Hundreds or even thousands of reasoning algorithms may be applied, each performing a different analysis (e.g., comparison) and generating a score. For example, some reasoning algorithms may look at the matching of terms and synonyms within the language of the question entered and the portion of the data corpus found. Other reasoning algorithms may look at temporal or spatial features in the language, while others may evaluate the provenance of a portion of the data corpus and assess its veracity.
[0036] The scores obtained from various inference algorithms indicate how well potential responses are inferred from the input question based on the specific area of interest of that inference algorithm. Each resulting score is then weighted according to a statistical model. The statistical model is captured in IBM Watson TM How well the reasoning algorithm performs during the training period of the QA system in establishing inferences between two similar passages in a specific domain. Then, statistical models can be used to summarize the IBM Watson TM The confidence that the QA system has in the evidence for the potential answer (i.e., the candidate answer) inferred from the question. This process can be repeated for each candidate answer until IBM Watson TM The QA system generates a final answer or a ranked set of answers for the input question until it identifies a candidate answer that appears to be significantly stronger than other candidate answers.
[0037] The types of information handling systems that can utilize the QA system 100 range from small handheld devices (e.g., handheld computers / mobile phones 110) to large mainframe systems (e.g., mainframe computers 170). Examples of handheld computers 110 include personal digital assistants (PDAs), personal entertainment devices (e.g., MP3 players, portable televisions, and compact disk players). Other examples of information handling systems include pen-type or tablet computers 120, laptop or notebook computers 130, personal computer systems 150, and servers 160. As shown, a computer network 102 can be used to network various information handling systems together. The types of computer networks 102 that can be used to interconnect various information handling systems include local area networks (LANs), wireless local area networks (WLANs), the Internet, public switched telephone networks (PSTNs), other wireless networks, and any other network topologies that can be used to interconnect information handling systems. Many information handling systems include non-volatile data storage devices, such as hard disk drives and / or non-volatile memories. Figure 1 Some of the information handling systems shown in depict separate nonvolatile data storage devices (server 160 uses nonvolatile data storage device 165, while mainframe computer 170 uses nonvolatile data storage device 175). The nonvolatile data storage device may be a component external to the various information handling systems, or may be internal to one of the information handling systems. Figure 2 An illustrative example of an information handling system is shown in , which shows an exemplary processor and various components that are typically accessed by the processor.
[0038] Figure 2 An information handling system 200 (particularly a processor and general components) is shown, which is a simplified example of a computer system capable of performing the computing operations described herein. Information handling system 200 includes one or more processors 210 coupled to a processor interface bus 212. Processor interface bus 212 connects processor 210 to Northbridge 215, which is also referred to as a memory controller hub (MCH). Northbridge 215 is connected to system memory 220 and provides a means for processor 210 to access system memory. Graphics controller 225 is also connected to Northbridge 215. In one embodiment, PCI Express bus 218 connects Northbridge 215 to graphics controller 225. Graphics controller 225 is connected to display device 230, such as a computer monitor.
[0039] The north bridge 215 and the south bridge 235 are connected to each other using a bus 219. In one embodiment, the bus is a direct media interface (DMI) bus that transmits data at high speed in each direction between the north bridge 215 and the south bridge 235. In another embodiment, a peripheral component interconnect (PCI) bus connects the north bridge and the south bridge. The south bridge 235 (also known as the I / O controller hub (ICH)) is a chip that typically implements functions that operate at a slower speed than the functions provided by the north bridge. The south bridge 235 typically provides various buses for connecting various components. For example, these buses include PCI and PCI Express buses, ISA buses, system management buses (SM buses or SMBs), and / or low pin count (LPC) buses. The LPC bus typically connects low-bandwidth devices, such as boot ROM 296 and "legacy" I / O devices (using a "super I / O" chip). "Legacy" I / O devices (298) can include, for example, serial and parallel ports, keyboards, mice, and / or floppy disk controllers. The LPC bus also connects Southbridge 235 to a Trusted Platform Module (TPM) 295. Other components often included in Southbridge 235 include a direct memory access (DMA) controller, a programmable interrupt controller (PIC), and a storage device controller that uses bus 284 to connect Southbridge 235 to a non-volatile storage device 285, such as a hard drive.
[0040] ExpressCard 255 is a slot that connects hot-pluggable devices to the information handling system. Because ExpressCard 255 connects to Southbridge 235 using both Universal Serial Bus (USB) and PCI Express bus, ExpressCard 255 supports both PCI Express and USB connections. Southbridge 235 includes USB controller 240, which provides USB connections to devices connected to the USB. These devices include webcam (camera) 250, infrared (IR) receiver 248, keyboard and trackpad 244, and Bluetooth device 246 that provides wireless personal area network (PAN). USB controller 240 also provides USB connections to various other USB-connected devices 242, such as mice, removable non-volatile storage devices 245, modems, network cards, ISDN connectors, faxes, printers, USB hubs, and many other types of USB-connected devices. Although removable nonvolatile storage device 245 is shown as a USB connected device, removable nonvolatile storage device 245 may be connected using a different interface, such as a Firewire interface.
[0041] A wireless local area network (LAN) device 275 is connected to the Southbridge 235 via a PCI or PCI Express bus 272. The LAN device 275 typically implements one of the IEEE 802.11 standards for wireless modulation techniques, all of which use the same protocol for wireless communication between the information handling system 200 and another computer system or device. An optical storage device 290 is connected to the Southbridge 235 using a Serial ATA (SATA) bus 288. Serial ATA adapters and devices communicate over a high-speed serial link. The Serial ATA bus also connects the Southbridge 235 to other forms of storage devices, such as hard drives. An audio circuit 260 (e.g., a sound card) is connected to the Southbridge 235 via bus 258. The audio circuit 260 also provides functions such as an audio line input and optical digital audio input port 262, an optical digital output and headphone jack 264, an internal speaker 266, and an internal microphone 268. An Ethernet controller 270 is connected to the Southbridge 235 using a bus such as a PCI or PCI Express bus. Ethernet controller 270 connects information handling system 200 to a computer network, such as a local area network (LAN), the Internet, and other public and private computer networks.
[0042] although Figure 2 An information processing system is shown, but an information processing system may take many forms, some of which are Figure 1 For example, an information handling system may take the form of a desktop, server, portable, laptop, notebook, or other form factor computer or data processing system. In addition, an information handling system may take other form factors such as a personal digital assistant (PDA), a gaming device, an ATM machine, a portable telephone device, a communication device, or other device that includes a processor and memory.
[0043] Figure 3 3 is a component diagram illustrating various components included in a system that utilizes entity relationships to discover answers using a knowledge graph. A question 300 input by a requester (e.g., a user) is shown as input to the system. At the top of the figure, the processing of the question is depicted by a conventional QA pipeline 340, which uses conventional methods to identify candidate answers and metadata (e.g., scores, etc.) related to such candidate answers, which are shown as stored in a memory area 345. In addition, the conventional QA pipeline identifies text paragraphs related to the question, where these paragraphs are stored in a memory area 350.
[0044] Utilizing knowledge graph data to discover candidate answers is shown to begin with process 310, which builds a knowledge graph for question 300. One or more candidate answers discovered through knowledge graph analysis may be the same as candidate answers discovered through traditional QA pipeline methods, where the scores of such candidate answers are improved. In addition, some candidate answers discovered through knowledge graph analysis may be new or may be different from candidate answers discovered through traditional QA methods, in which case such candidate answers are added to the list of possible candidate answers. The result of process 310 is a question knowledge graph 320. The example shown in figure 320 shows two "known" question entities (QE1 and QE2) provided by the question and a "missing" question entity (QE1) as a missing entity. m ), where the question seeks an answer to the missing entity. The relationships (associations) between the various entities are also shown. Although the initial question knowledge graph (320) and the initial paragraph knowledge graph (360) can be analyzed and used to identify candidate answers based on the knowledge graph, in one embodiment, the knowledge graph is "extended" using known reliable data (e.g., depicted as an online encyclopedia retrieved from an external data repository 330). The extended knowledge graph is used to identify additional entities and relationships that may not be easily found in the question and paragraph data. If knowledge graph expansion is being used, the process 325 is used to expand the question knowledge graph (320) to form an expanded question knowledge graph 335.
[0045] Regarding paragraphs, a knowledge graph is constructed for each paragraph identified by the traditional QA pipeline using process 355. Process 355 thereby forms a paragraph knowledge graph 360. Again, if graph expansion is being utilized, a process (process 365) is performed to expand each paragraph knowledge graph 360 to create an expanded paragraph knowledge graph 370.
[0046] Process 375 computes the similarity between the question knowledge graph (graph 320 or graph 335 if extensions are used) and each paragraph knowledge graph (graph 360 or graph 370 if extensions are used). This process attempts to identify entities in the paragraph knowledge graph that the analysis indicates correspond to entities that are "missing" from the question knowledge graph. Using the example shown, the "missing" entity (QE m) appears to correspond to PE3 in the passage knowledge graph shown based on other entities and relationships. Although PE3 is depicted in both the unexpanded knowledge graph and the expanded knowledge graph for simplicity of the diagram, different entities in the expanded knowledge graph may correspond well to the missing entity (e.g., new entity "QE5" not shown, etc.). The identification of additional candidate answers by processing 375 also calculates a similarity score, which in one embodiment indicates the similarity between the passage knowledge graph from which the candidate answer is found and the question knowledge graph, so that highly similar graphs are scored higher than less similar graphs. The identified candidate answers and their corresponding scores are stored in memory area 380.
[0047] Processing 385 combines the candidate answers identified by the traditional QA pipeline processing with the candidate answers identified by the above-mentioned knowledge graph analysis. In one embodiment, the candidate answers identified by both the traditional QA pipeline processing and the knowledge graph analysis processing have their scores "improved". In one embodiment, the amount of improvement in the traditional score of the candidate answer found in the memory area 345 is based on the score of the candidate answer according to the knowledge graph similarity stored in the memory area 380. The candidate answers and their "improved" scores are stored in the memory area 390. In one embodiment, if the candidate answer is only found in the memory area 380 (indicating that the candidate answer is found by the knowledge graph analysis processing, not by the traditional QA pipeline processing), the candidate answer is added to the list of possible candidate answers in the memory area 390, where the score of the candidate answer is based on the knowledge graph similarity score stored in the memory area 380. The traditional QA pipeline processing is shown as continuing at 395, where the pipeline processing uses the candidate answers and scores stored in the memory area 390, where some of these candidate answers and scores are affected by the above-mentioned knowledge graph analysis. Continued QA processing ultimately results in one or more candidate answers being selected as the most likely answers to the question initially input to the system (question 300).
[0048] Figure 4 is an illustration of a flow chart showing logic for exploiting entity relationships to discover answers using a knowledge graph. Figure 4 The process of begins at 400 and illustrates the steps taken by the process of exploiting entity relationships to discover answers using knowledge graph data. At step 410, a traditional question answering (QA) pipeline process is performed on the submitted question 300. The traditional QA pipeline generates candidate answers with scoring metadata, and the candidate answers and metadata are stored in the memory area 345. In addition, the traditional QA pipeline process also identifies relevant paragraphs used to generate the candidate answers, which are stored in the memory area 350.
[0049] At step 420, the process uses a traditional knowledge graph generator process to create a knowledge graph (KG) of the submitted question 300. The created question KG is stored in memory area 320. At step 430, the process selects the first paragraph identified by the traditional QA pipeline process from memory area 350. At step 440, the process uses a traditional knowledge graph generator to create a knowledge graph (KG) of the selected paragraph. The created paragraph KG is stored in memory area 360, where a memory area is allocated for each paragraph KG. The process determines whether there are more paragraphs to be processed and creates a paragraph knowledge graph (decision 450). If there are more paragraphs, decision 450 branches to the "yes" branch, which loops back to step 430 to select the next paragraph and create its knowledge graph. The loop continues until all paragraphs have been processed, at which time decision 450 branches to the "no" branch, thereby exiting the loop.
[0050] This process determines whether to use Figure 5 to “extend” the generated knowledge graph (decision 460) using the novel techniques shown in . Knowledge graph extension uses a known data set (e.g., an online encyclopedia) to add additional entities and relationships to the created knowledge graph collection. Discovery of additional candidate answers can be performed without knowledge graph extension. However, in some environments, extension of the knowledge graph may provide additional candidate answers that are not apparent from the original knowledge graph. In one embodiment, knowledge graph extension is an option, such as a configuration setting or runtime option that can be selected by an operator or requester. If knowledge graph extension is being used, decision 460 branches to the “yes” branch, which then, at predefined processing 470, executes the expansion KG routine (see for processing details). Figure 5 and corresponding text). On the other hand, if knowledge graph extension is not used, decision 460 branches to the “no” branch, thereby bypassing predefined processing 470.
[0051] At predefined processing 480, the process executes a "compute graph similarity" routine (see Figure 6 and the corresponding text). This routine uses the extended knowledge graph (if predefined processing is used) or the original knowledge graph and computes the graph similarity between the question KG and the paragraph KG to identify additional candidate answers.
[0052] At predefined processing 490, the process executes a "score candidate answers (CA)" routine (see Figure 7 and corresponding text). The routine scores the candidate answers identified by computing graph similarity. In one embodiment, the routine improves the scores of candidate answers found by both the graph similarity processing described herein and the traditional QA pipeline processing. Thereafter, Figure 4Processing ends at 495.
[0053] Figure 5 is an illustration of a flow chart showing logic for extending a knowledge graph using data from an external source. Figure 5 The process of begins at 500 and illustrates the steps taken by the process to extend a knowledge graph (KG) using one or more external data sources. At step 510, the process retrieves an external data source, such as an online encyclopedia or the like. In one embodiment, an external data source is selected that is relevant to the subject matter of the submitted question and the resulting passage. For example, if the question is related to the medical field, a medical external data source may be retrieved instead of or in addition to a general online encyclopedia.
[0054] At step 520, the process selects a first knowledge graph from the set of available knowledge graphs 525. The set of available knowledge graphs includes the original question KG 320 and Figure 4 The process shown in generates a set of original paragraph KG 360. At step 530, the process uses the selected knowledge graph to initialize the extended knowledge graph, wherein the set of extended knowledge graphs is stored in the memory area 540, and the set of extended knowledge graphs includes an extended question knowledge graph 335 and a set of extended knowledge graphs 370. In one embodiment, the initialization of the extended knowledge graph includes copying the original knowledge graph to the extended knowledge graph, so that the extended knowledge graph starts with the basis of the original knowledge graph, and the expansion adds entities and relationships to the original knowledge graph data. At step 550, the process selects the first entity from the selected knowledge graph. Next, the process determines whether the selected entity is found in the external data source (decision 560). If the selected entity is found in the external data source, decision 560 branches to the "yes" branch to execute steps 565 to 580. On the other hand, if the selected entity is not found in the external data source, decision 560 branches to the "no" branch, thereby bypassing steps 565 to 580.
[0055] In step 565, the process selects the first relationship (association) found in the external data source that references another entity from the entity. The process determines whether the selected relationship is also found in the selected knowledge graph (decision 570). If the selected relationship is also found in the selected knowledge graph, decision 570 skips the relationship and branches to the "yes" branch. On the other hand, if the selected relationship is not found in the selected knowledge graph, it means that a new relationship is found in the external data source, then decision 570 branches to the "no" branch, and then, in step 575, the process adds the newly found relationship to the extended knowledge graph, and also adds a new entity that connects the relationship to the existing relationship in the original knowledge graph, thereby adding new relationships and new entities that are not found in the original knowledge graph to the extended knowledge graph. The new relationship and new entity are added to the memory area 540 (if the original question KG is being processed, it is added to the extended question KG 335, or if one of the original paragraph KGs is being processed, it is added to one of the extended paragraph KGs 370).
[0056] The process determines whether there are more relationships in the external data related to the selected entity to be processed (decision 580). If there are more relationships to be processed, decision 580 branches to the “yes” branch which loops back to step 565 to select and process the next relationship as described above. The loop continues until all relationships with the selected entity have been processed, at which time decision 580 branches to the “no” branch, exiting the loop.
[0057] Next, the process determines whether there are more entities to be processed found in the selected knowledge graph (decision 585). If there are more entities to be processed, decision 585 branches to the "yes" branch, which loops back to step 550 to select and process the next entity as described above. The loop continues until all entities found in the selected knowledge graph have been processed, at which point decision 585 branches to the "no" branch, thereby exiting the loop. Finally, the process determines whether there are more original knowledge graphs stored in memory area 525 to be processed (decision 590). If there are more original knowledge graphs to be processed, decision 590 branches to the "yes" branch, which loops back to step 520 to select and process the next original knowledge graph as described above. The loop continues until all original knowledge graphs have been processed, at which point decision 590 branches to the "no" branch, thereby exiting the loop. Thereafter, Figure 5 Processing returns to the calling routine at 595 (see Figure 4 ).
[0058] Figure 6 is an illustration of a flow chart showing logic for computing similarities between knowledge graphs (KGs). Figure 6 The process of begins at 600 and illustrates the steps taken by the process of calculating the similarity between the question knowledge graph and various paragraph knowledge graphs. At step 610, the process selects a first paragraph knowledge graph. The paragraph knowledge graph can be the original paragraph knowledge graph 360, or, if graph expansion is utilized, the expanded paragraph knowledge graph 370 is selected.
[0059] At step 620, the process selects a first entity from the question knowledge graph. Similar to the paragraph knowledge graph, the question knowledge graph can be the original question knowledge graph 320, or, if graph expansion is utilized, the extended question knowledge graph 335 is selected. The process determines whether the selected entity is also found in the selected paragraph knowledge graph (decision 625). If the selected entity is also found in the selected paragraph knowledge graph, decision 625 branches to the "yes" branch, and then, at step 630, the process increases the score of the paragraph knowledge graph to reflect the similarity between the paragraph knowledge graph and the question knowledge graph. The score of the paragraph knowledge graph is stored in the memory area 640. On the other hand, if the selected entity in the question knowledge graph is not found in the selected paragraph knowledge graph, decision 625 branches to the "no" branch, thereby bypassing step 630.
[0060] The process determines whether there are more entities in the question knowledge graph to search for in the paragraph knowledge graph (decision 650). If there are more entities to search for in the paragraph knowledge graph, decision 650 branches to the "yes" branch, which loops back to step 620 to select the next entity from the question knowledge graph. The loop continues until all entities in the question knowledge graph have been processed, at which point decision 650 branches to the "no" branch, exiting the loop.
[0061] Steps 655 to 675 process similarities in entity relationships in a manner similar to steps 620 to 650 that process entity similarities. In step 655, the process selects the first relationship from the question knowledge graph (the original question KG 320 or the expanded question KG 335). The process determines whether the selected relationship is also found in the selected paragraph knowledge graph (decision 660). If the selected relationship is also found in the selected paragraph knowledge graph, decision 660 branches to the "yes" branch, and then, in step 670, the process increases the score of the paragraph knowledge graph to reflect the similarity between the paragraph knowledge graph and the question knowledge graph. The score of the paragraph knowledge graph is stored in the memory area 640. On the other hand, if the selected missing entity in the question knowledge graph is not found in the selected paragraph knowledge graph, decision 660 branches to the "no" branch, thereby bypassing step 670.
[0062] The process determines whether there are more relations in the question knowledge graph to be searched in the paragraph knowledge graph (decision 650). If there are more relations to be searched in the paragraph knowledge graph, decision 675 branches to the “yes” branch, which loops back to step 655 to select the next relation from the question knowledge graph as described above. The loop continues until all relations in the question knowledge graph have been processed, at which point decision 675 branches to the “no” branch, thereby exiting the loop.
[0063] The process determines whether there are more paragraph knowledge graphs to be processed to calculate their similarity with the question knowledge graph as described above (decision 680). If there are more paragraph knowledge graphs to be processed, decision 680 branches to the "yes" branch, which loops back to step 610 to select and process the next paragraph knowledge graph (original knowledge graph 360 or expanded knowledge graph 370) as described above. The loop continues until all paragraph knowledge graphs have been processed, at which time decision 680 branches to the "no" branch, thereby exiting the loop.
[0064] At step 690, the process compares the missing entities (Q m ) are added to the knowledge graph candidate answer set, where these entities found in the passage knowledge graph are used as possible candidate answers. In one embodiment, the scores of the passage knowledge graph (previously stored in memory area 640) are used to calculate the scores of the candidate answers, where the candidate answers found from the knowledge graph comparison are stored in memory area 380. Passage knowledge graphs that do not have entities that are substantially similar to the missing entities in the question knowledge graph are not used (are discarded). Thereafter, Figure 6 Processing returns to the calling routine at 695 (see Figure 4 ).
[0065] Figure 7 is an illustration of a flow chart showing logic for scoring candidate answers (CAs), including candidate answers generated by utilizing entity relationships found in a knowledge graph. Figure 7 The process begins at 700 and is shown by using Figure 6 The steps taken by the process to score the candidate answer (CA) based on the information derived from the knowledge graph comparison shown in . At step 710, the process applies a threshold (e.g., a minimum paragraph knowledge graph (KG) score for this implementation, etc.).
[0066] In step 725, the process selects Figure 6The first candidate answer generated by comparing the knowledge graph shown in . The knowledge graph candidate answer is retrieved from the memory area 380, and if a threshold is applied, the candidate answer retrieved from the memory area 380 is the candidate answer with a score that satisfies the threshold. At step 730, the process searches for the selected knowledge graph candidate answer in the candidate answer list generated by the traditional QA pipeline process, wherein the candidate answer from the traditional QA pipeline process is retrieved from the memory area 345.
[0067] Next, the process determines whether the selected candidate answer generated by the knowledge graph comparison process generates a candidate answer that was also generated by the traditional QA pipeline process (decision 740). If the selected candidate answer is found in both lists (generated by the knowledge graph comparison process and the traditional QA pipeline process), then decision 740 branches to the "yes" branch, and step 745 is then executed. In one embodiment, when the candidate answer is found in both lists, the score of the candidate answer is increased ("boosted") to reflect the discovery of the answer using both processes.
[0068] On the other hand, if a candidate answer is found only in the knowledge graph candidate answer list (memory area 380), and the candidate answer is not generated by traditional QA pipeline processing, then decision 740 branches to the "no" branch, and then, at step 750, the new candidate answer found by the knowledge graph comparison process is added to the list of potential candidate answers. The candidate answers and their corresponding scores are stored in memory area 755. In one embodiment, the scores of candidate answers found only by the candidate answer comparison process are based on the scores in the Figure 6 The score calculated in reflects the similarity between the paragraph knowledge graph from which the candidate answer is found and the question knowledge graph.
[0069] Next, the process determines whether there are more candidate answers to process in the list 380, which is obtained by Figure 6 380 (decision 760). If there are more candidate answers to process, decision 760 branches to the "yes" branch, which loops back to step 725 to select and process the next candidate answer in list 380 as described above. The loop continues until all candidate answers in list 380 have been processed, at which time decision 760 branches to the "no" branch, thereby exiting the loop.
[0070] At step 765, the process adds any candidate answers that were not in the knowledge graph candidate answer list (380) but were discovered only through traditional QA pipeline processing (stored in memory area 345 rather than in memory area 380). These additional candidate answers and their scores are copied to memory area 755 without enhancing (“improving”) their scores. At step 770, the process sorts the enhanced candidate answer scores from highest (best) score to lowest (worst) score. These sorted enhanced candidate answers and their corresponding scores are stored in memory area 775. At step 780, the process returns one or more “best” answers from the sorted enhanced candidate answer list now stored in memory area 775. The selected “best” answer is stored in memory area 785 and returned to the requester 790, where the requester is a process or user. Thereafter, Figure 7 Processing returns to the calling routine at 795 (see Figure 4 ).
[0071] Although specific embodiments of the present invention have been shown and described, it is obvious to those skilled in the art that, based on the teachings herein, changes and modifications can be made without departing from the present invention and its broader aspects. Therefore, the appended claims include all such changes and modifications within the true spirit and scope of the present invention within their scope. It will be understood by those skilled in the art that if a specific number of claim elements are intended to be introduced, such intentions will be explicitly recorded in the claims, and in the absence of such a record, there is no such limitation. For non-limiting examples, in order to help understand, the following appended claims include the use of introductory phrases "at least one" and "one or more" to introduce claim elements. However, the use of such phrases should not be interpreted as implying that the introduction of claim elements by the indefinite article "one" or "an" limits any specific claim containing such introduced claim elements to an invention containing only one such element, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article (such as "one" or "an"); the use of definite articles in the claims is also the same.
Claims
1. A method implemented by an information processing system, the information processing system comprising a processor and a memory accessible by the processor, the method include: Select original entities from the original knowledge graph; Accessing data sources external to the original knowledge graph; Searching the accessed data source for the original entity; Based on the search, identifying a relationship in the data source between the original entity and a new entity, wherein the new entity is missing from the original knowledge graph; receiving, from a question answering (QA) system, a paragraph and a set of candidate answers corresponding to a submitted question; creating a plurality of paragraph knowledge graphs, wherein each of the paragraph knowledge graphs corresponds to one of the paragraphs; and An extended knowledge graph formed by adding the new entity to the original knowledge graph is generated, and a candidate answer score is calculated based on a similarity between the extended knowledge graph and the paragraph knowledge graph.
2. The method according to claim 1, further comprising: include: The extended knowledge graph includes the relationship between the original entity and the new entity.
3. The method according to claim 1, in, The data source is an online encyclopedia.
4. The method according to claim 1, in, The original knowledge graph is the knowledge graph of the questions submitted to the question answering (QA) system.
5. The method according to claim 1, in, The original knowledge graph is a knowledge graph of a passage identified by a question answering (QA) system pipeline during a process of discovering one or more candidate answers in response to a question received at the QA system.
6. The method according to claim 1, further comprising: include: comparing the expanded knowledge graph corresponding to the question knowledge graph to each of the paragraph knowledge graphs corresponding to the paragraphs, wherein the comparison results in a paragraph score associated with each of the paragraph knowledge graphs and identification of one or more candidate answers; selecting one or more of the candidate answers based on candidate answer scores corresponding to the selected candidate answers; and The selected candidate answer is provided to the requester of the question.
7. An information processing system, include: one or more processors; a memory coupled to at least one of the processors; as well as A set of computer program instructions, stored in the memory and executed by at least one of the processors, to perform the actions of the method according to any one of claims 1-6.
8. A computer program product stored in a computer-readable storage medium, comprising computer program code, which, when executed by an information processing system, performs the actions of the method according to any one of claims 1 to 6.
9. A computer system comprising means for performing the steps of the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
System and method for providing answers to questions
US20130007033A1
Question answering to populate knowledge base
US20140280307A1
Knowledge Graph Generator Enabled by Diagonal Search
US20150095303A1
