Information processing device, information processing method, and program
By verifying the truth value of extracted relationships in natural language data, the method addresses inaccuracies in knowledge graph construction, resulting in a highly accurate knowledge graph that reflects the original data accurately.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- NEC CORP
- Filing Date
- 2024-11-20
- Publication Date
- 2026-06-01
AI Technical Summary
Existing methods for constructing knowledge graphs, such as those using large language models (LLM), often result in inaccuracies due to hallucinations, leading to knowledge graphs that do not accurately reflect the facts described in the original natural language data.
A process is introduced to verify the truth value of relationships extracted from natural language data using a large language model, ensuring only accurate triples are added to the knowledge graph by confirming their presence in the original data.
This approach enables the construction of a highly accurate knowledge graph that aligns with the facts in the original data, improving the reliability of knowledge representation.
Smart Images
Figure 2026089305000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a technique for constructing a knowledge graph.
Background Art
[0002] Knowledge graphs that represent knowledge in various fields have been developed and utilized. For example, Patent Document 1 proposes a method for constructing a knowledge graph customized for applications.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, it is not always possible to construct a highly accurate knowledge graph even with Patent Document 1.
[0005] One object of the present disclosure is to provide an information processing apparatus capable of constructing a highly accurate knowledge graph.
Means for Solving the Problems
[0006] In one aspect of the present disclosure, the information processing apparatus includes extraction means for extracting entities and relationships between entities from natural language data, judgment means for judging the truth or falsehood of the relationships between the entities, graph construction means for adding the relationships between the entities judged to be true by the judgment means to the knowledge graph and not adding the relationships between the entities judged to be false by the judgment means to the knowledge graph, and is provided with.
[0007] From another perspective of this disclosure, the information processing method is: A method of information processing performed by a computer, An extraction process is performed to extract entities and relationships between entities from natural language data. A determination process is performed to determine the truth value of the relationship between the aforementioned entities. The system performs a graph construction process that adds relationships between entities determined to be true by the aforementioned determination process to the knowledge graph, and does not add relationships between entities determined to be false by the aforementioned determination process to the knowledge graph.
[0008] In yet another aspect of this disclosure, the program is An extraction process that extracts entities and relationships between entities from natural language data, A determination process for determining the truth value of the relationship between the aforementioned entities, The computer is instructed to perform a graph construction process that adds the relationships between entities determined to be true by the aforementioned determination process to the knowledge graph, and does not add the relationships between entities determined to be false by the aforementioned determination process to the knowledge graph. [Effects of the Invention]
[0009] This disclosure makes it possible to provide an information processing device capable of constructing a highly accurate knowledge graph. [Brief explanation of the drawing]
[0010] [Figure 1] This is a diagram conceptually illustrating the information processing device related to this disclosure. [Figure 2] This is a block diagram showing the hardware configuration of the information processing device related to this disclosure. [Figure 3] This is a block diagram showing the functional configuration of the information processing device related to this disclosure. [Figure 4] This is an example of natural language data. [Figure 5] This is an example of an extracted entity. [Figure 6] This is an example of a prompt generated by the relation extraction unit. [Figure 7] An example of the answer of the LLM. [Figure 8] An example of the prompt by the relationship extraction confirmation unit. [Figure 9] Another example of the answer of the LLM. [Figure 10] A flowchart of the processing by the information processing apparatus according to the present disclosure. [Figure 11] A block diagram showing the functional configuration of another information processing apparatus according to the present disclosure. [Figure 12] A flowchart of the processing by another information processing apparatus according to the present disclosure.
Mode for Carrying Out the Invention
[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.
[0012] <First Embodiment> [Overall Configuration] In the fields of medicine and drug discovery, there is abundant data written in natural language, such as papers and electronic medical records. By constructing a knowledge graph from such natural language data, the relationships between the data can be expressed and utilized for advanced search and prediction tasks, etc.
[0013] The knowledge graph is constructed, for example, by using a large language model (LLM: Large Language Models). However, in the above method, when the LLM generates hallucinations (i.e., when the LLM generates misinformation), there is a possibility that a knowledge graph different from the facts described in the original natural language data is constructed.
[0014] Therefore, in the present embodiment, when constructing the knowledge graph, a process of confirming whether the knowledge graph matches the facts described in the original natural language data is included. Thereby, a highly accurate knowledge graph in accordance with the facts described in the original natural language data is constructed.
[0015] Figure 1 is a conceptual diagram of the information processing device of this embodiment. The information processing device 10 constructs a knowledge graph from natural language data such as input papers. First, the information processing device 10 extracts a list of triples (node, edge, node) from the natural language data using LLM. Nodes represent entities, and edges represent relationships between nodes. Next, the information processing device 10 uses LLM to determine the truth value of each triple. The information processing device 10 adds triples determined to be true to the knowledge graph, and excludes triples determined to be false. In this way, by including a process to verify the truth value of the information (triples) obtained from LLM, the information processing device 10 can construct a highly accurate knowledge graph.
[0016] In this embodiment, a knowledge graph is constructed in the field of medicine and drug discovery, but the target field is not limited to this, and it can be applied to other fields such as materials development and pesticide development.
[0017] [Hardware configuration] Figure 2 is a block diagram showing the hardware configuration of the information processing device 10 according to the first embodiment. The information processing device 10 is an example of an information processing device. As shown in the figure, the information processing device 10 includes an interface (I / F) 11, a processor 12, a memory 13, a recording medium 14, and a database (DB) 15.
[0018] I / F11 performs data input and output with external devices. Specifically, I / F11 acquires natural language data for use by the information processing device 10 from external devices.
[0019] Processor 12 is a computer such as a CPU (Central Processing Unit) and controls the entire information processing device 10 by executing a pre-prepared program. Processor 12 may be a GPU (Graphics Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof. Processor 12 executes the knowledge graph construction process described later.
[0020] Memory 13 consists of ROM (Read Only Memory), RAM (Random Access Memory), and other components. Memory 13 is also used as working memory while the processor 12 is executing various processes.
[0021] The recording medium 14 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the information processing device 10. The recording medium 14 stores various programs that the processor 12 executes. When the information processing device 10 executes various processes, the programs stored on the recording medium 14 are loaded into the memory 13 and executed by the processor 12. The DB 15 stores the knowledge graph.
[0022] In addition to the above, the information processing device 10 may also be equipped with a display device such as a liquid crystal display, and an input device such as a keyboard or mouse. These display devices and input devices are used, for example, by the administrator of the information processing device 10 to perform necessary management.
[0023] [Functional Configuration] Figure 3 is a block diagram showing the functional configuration of the information processing device 10 of the first embodiment. Functionally, the information processing device 10 comprises a named entity recognition unit 101, a relationship extraction unit 102, and a relationship extraction confirmation unit 103.
[0024] Natural language data is input to the information processing device 10 via the I / F 11. The natural language data is input to the named entity recognition unit 101, the relation extraction unit 102, and the relation extraction confirmation unit 103. Figure 4 shows an example of natural language data. The natural language data in Figure 4 is a medical paper and is obtained from a medical literature search database, etc.
[0025] The named entity recognition unit 101 extracts named entities from natural language data using a model such as LLM. Named entities include proper nouns and numerical representations such as dates and times. In the fields of medicine and drug discovery, named entities include, for example, disease names, drug names, gene names, and protein names. The named entities extracted by the named entity recognition unit 101 are treated as candidates for entities in the knowledge graph. The named entity recognition unit 101 outputs the extracted named entities (hereinafter also referred to as "entities") to the relational extraction unit 102.
[0026] Figure 5 shows an example of extracted entities. In Figure 5, the named entity recognition unit 101 extracts entities such as cholesterol, DNA, PCSK9, and R3500Q from the natural language data in Figure 4.
[0027] The relationship extraction unit 102 extracts relationships between entities from the natural language data based on the natural language data and the entities. The relationship extraction unit 102 extracts a list of triples as the relationships between entities.
[0028] Specifically, the relation extraction unit 102 creates a prompt as shown in Figure 6 and inputs the created prompt to the LLM. This prompt instructs the LLM to extract triples from the natural language data. The relation extraction unit 102 then obtains the response to the prompt (i.e., a list of triples) from the LLM. The relation extraction unit 102 outputs the LLM's response to the relation extraction confirmation unit 103.
[0029] Figure 6 shows an example of a prompt from the relationship extraction unit 102. The prompt in Figure 6 includes an instruction sentence 51, a specific example 52, and a context 53. Instruction sentence 51 is an instruction sentence for the LLM. Instruction sentence 51 includes sentences that instruct the LLM to infer relationships between entities from natural language data, and sentences that instruct the LLM to output the answer in triple format. Specific example 52 is an example of a process that we want the LLM to perform. By showing such an example, the accuracy of the LLM's answer can be improved. Context 53 is a source of information for the LLM to generate the answer, and includes entities input from the named entity recognition unit 101 and natural language data.
[0030] Figure 7 shows an example of an LLM response. As shown in Figure 7, the relation extraction unit 102 obtains a list of triples as the LLM response.
[0031] Returning to Figure 3, the relationship extraction and verification unit 103 determines the truth value of each triple based on the natural language data and the list of triples. "True" indicates that the triple (i.e., the relationship between entities) is described in the natural language data, while "false" indicates that the triple is not described in the natural language data. Based on the truth value determination results, the relationship extraction and verification unit 103 constructs a knowledge graph.
[0032] Specifically, the relationship extraction and verification unit 103 creates a prompt as shown in Figure 8 and inputs the created prompt to the LLM. This prompt instructs the LLM to determine the truth value of each triple. The relationship extraction and verification unit 103 then obtains the response to the prompt (i.e., the truth value determination result) from the LLM. If the triple is true (TRUE), the relationship extraction and verification unit 103 adds that triple to the knowledge graph. On the other hand, if the triple is false (FALSE), the relationship extraction and verification unit 103 excludes that triple without adding it to the knowledge graph.
[0033] Figure 8 shows an example of a prompt from the relation extraction and verification unit 103. The prompt in Figure 8 includes an instruction sentence 71 and a context 72. The instruction sentence 71 is an instruction sentence to the LLM and includes a sentence instructing it to determine the truth value of each triple based on natural language data. The context 72 is a source of information for the LLM to generate an answer and includes a list of triples input from the relation extraction unit 102 and natural language data.
[0034] Figure 9 shows an example of an LLM response. As shown in Figure 9, the relationship extraction and verification unit 103 obtains the truth value determination result for each triple as the LLM response. In the response shown in Figure 9, three triples are determined to be false. The relationship extraction and verification unit 103 determines that these three triples should be excluded from the knowledge graph.
[0035] The relationship extraction and verification unit 103 may also output the truth value determination result for each triple to the display device. The user can verify whether the determination made by the relationship extraction and verification unit 103 is correct by looking at the natural language data and the display on the display device.
[0036] Furthermore, the named entity recognition unit 101, the relation extraction unit 102, and the relation extraction verification unit 103 can use OpenAI's GPT (Generative Pre-trained Transformer) as the LLM. The LLMs used by the named entity recognition unit 101, the relation extraction unit 102, and the relation extraction verification unit 103 may be the same model or different models. For example, the named entity recognition unit 101 may use a pre-trained language model specialized for a domain (in this embodiment, the medical / drug discovery field).
[0037] In the above configuration, the named entity recognition unit 101 and the relationship extraction unit 102 are examples of extraction means, and the relationship extraction confirmation unit 103 is an example of a determination means and a graph construction means.
[0038] [Knowledge graph construction process] Next, we will explain the process of constructing the knowledge graph as described above. Figure 10 is a flowchart of the knowledge graph construction process by the information processing device 10. This process is realized when the processor 12 shown in Figure 2 executes a pre-prepared program and operates as each element shown in Figure 3.
[0039] First, natural language data is input to the information processing device 10 through the I / F 11 (step S101). The natural language data is input to the named entity recognition unit 101, the relation extraction unit 102, and the relation extraction confirmation unit 103.
[0040] Next, the named entity recognition unit 101 extracts entities from the natural language data using a model such as an LLM (step S102). The named entity recognition unit 101 outputs the extracted entities to the relational extraction unit 102.
[0041] Next, the relationship extraction unit 102 extracts a list of triples from the natural language data based on the natural language data and entities (step S103). The relationship extraction unit 102 outputs the list of triples to the relationship extraction confirmation unit 103.
[0042] Next, the relationship extraction and verification unit 103 determines the truth value of each triple based on the natural language data and the list of triples (step S104). Then, the relationship extraction and verification unit 103 adds the triples determined to be true to the knowledge graph and discards the triples determined to be false (step S105). The process then ends.
[0043] [Examples of use] The knowledge graph constructed by the information processing device 10 can be used, for example, in semantic search. Furthermore, by utilizing the constructed knowledge graph in machine learning, it can be applied to various prediction tasks.
[0044] <Second Embodiment> Figure 11 is a block diagram showing the functional configuration of the information processing device of the second embodiment. The information processing device 20 comprises an extraction means 201, a determination means 202, and a graph construction means 203.
[0045] Figure 12 is a flowchart of the processing performed by the information processing device of the second embodiment. Extraction means 201 extracts entities and relationships between entities from natural language data (step S201). Determination means 202 determines the truth value of the relationships between entities (step S202). Graph construction means 203 adds the relationships between entities determined to be true by the determination means to the knowledge graph, and does not add the relationships between entities determined to be false by the determination means to the knowledge graph (step S203).
[0046] According to the information processing device of the second embodiment, it is possible to construct a highly accurate knowledge graph.
[0047] Some or all of the above embodiments may also be described as follows, but are not limited to the following:
[0048] (Note 1) An extraction means for extracting entities and relationships between entities from natural language data, A determination means for determining the truth or falsity of the relationship between the entities, A graph construction means that adds relationships between entities determined to be true by the determination means to the knowledge graph, and does not add relationships between entities determined to be false by the determination means to the knowledge graph, An information processing device equipped with the following features.
[0049] (Note 2) The information processing device described in Appendix 1, wherein the determination means determines that the relationship between the entities is true if it matches the facts described in the natural language data, and false if it does not match the facts described in the natural language data.
[0050] (Note 3) The determination means determines the truth value of the relationship between the entities by inputting the generated prompt into a large-scale language model. The information processing device according to Appendix 2, wherein the prompt is a prompt that instructs a large-scale language model to determine the truth value of the relationship between entities based on the natural language data and the relationship between the entities extracted by the extraction means.
[0051] (Note 4) The extraction means extracts a list of triples as the relationships between the entities, The determination means determines the truth value of each triple based on the natural language data and the list of triples. The graph construction means is an information processing device according to Appendix 1, which adds triples determined to be true to the knowledge graph and does not add triples determined to be false to the knowledge graph.
[0052] (Note 5) The aforementioned natural language data includes the information processing device described in Appendix 1, which includes research paper data and electronic medical records.
[0053] (Note 6) The extraction means is an information processing device according to Appendix 1, which uses a large-scale language model to extract entities and relationships between entities.
[0054] (Note 7) The extraction means is an information processing device according to Appendix 6, which extracts the entities using a domain-specific, pre-trained, large-scale language model.
[0055] (Note 8) The determination means is an information processing device as described in Appendix 1, which outputs the result of the determination to a display device.
[0056] (Note 9) A method of information processing performed by a computer, An extraction process is performed to extract entities and relationships between entities from natural language data. A determination process is performed to determine the truth value of the relationship between the aforementioned entities. An information processing method that performs a graph construction process which adds relationships between entities determined to be true by the aforementioned determination process to the knowledge graph, and does not add relationships between entities determined to be false by the aforementioned determination process to the knowledge graph.
[0057] (Note 10) An extraction process that extracts entities and relationships between entities from natural language data, A determination process for determining the truth value of the relationship between the aforementioned entities, A program that causes a computer to perform a graph construction process that adds relationships between entities determined to be true by the aforementioned determination process to the knowledge graph, and does not add relationships between entities determined to be false by the aforementioned determination process to the knowledge graph.
[0058] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure. [Explanation of Symbols]
[0059] 10 Information Processing Devices 101 Named entity extraction unit 102 Relationship Extraction Unit 103 Relationship Extraction and Verification Unit
Claims
1. An extraction means for extracting entities and relationships between entities from natural language data, A determination means for determining the truth or falsity of the relationship between the entities, A graph construction means that adds relationships between entities determined to be true by the determination means to the knowledge graph, and does not add relationships between entities determined to be false by the determination means to the knowledge graph, An information processing device equipped with the following features.
2. The information processing apparatus according to claim 1, wherein the determination means determines that the relationship between the entities is true if it matches the facts described in the natural language data, and determines that it is false if it does not match the facts described in the natural language data.
3. The determination means determines the truth value of the relationship between the entities by inputting the generated prompt into a large-scale language model. The information processing apparatus according to claim 2, wherein the prompt is a prompt that instructs a large-scale language model to determine the truth value of the relationship between entities based on the natural language data and the relationship between the entities extracted by the extraction means.
4. The extraction means extracts a list of triples as the relationships between the entities, The determination means determines the truth value of each triple based on the natural language data and the list of triples. The information processing apparatus according to claim 1, wherein the graph construction means adds triples determined to be true to the knowledge graph and does not add triples determined to be false to the knowledge graph.
5. The information processing apparatus according to claim 1, which includes the aforementioned natural language data such as research paper data and electronic medical records.
6. The information processing apparatus according to claim 1, wherein the extraction means extracts entities and relationships between entities using a large-scale language model.
7. The information processing apparatus according to claim 6, wherein the extraction means extracts the entities using a domain-specific, pre-trained, large-scale language model.
8. The information processing apparatus according to claim 1, wherein the determination means outputs the result of the determination to a display device.
9. A method of information processing performed by a computer, An extraction process is performed to extract entities and relationships between entities from natural language data. A determination process is performed to determine the truth value of the relationship between the aforementioned entities. An information processing method that performs a graph construction process which adds relationships between entities determined to be true by the aforementioned determination process to the knowledge graph, and does not add relationships between entities determined to be false by the aforementioned determination process to the knowledge graph.
10. An extraction process that extracts entities and relationships between entities from natural language data, A determination process for determining the truth value of the relationship between the aforementioned entities, A program that causes a computer to perform a graph construction process that adds relationships between entities determined to be true by the aforementioned determination process to the knowledge graph, and does not add relationships between entities determined to be false by the aforementioned determination process to the knowledge graph.