Generation method, generation program, and information processing device.

Abstract

To improve the accuracy of similarity estimation. [Solution] The processing unit 12 obtains combinations of a first entity included in the knowledge graph 20 and a second entity included in the knowledge graph 21. The processing unit 12 inputs the first entity and the second entity to the machine learning model 30 and instructs it to reduce the similarity between the first entity and the second entity if the relationship between the first entity and the second entity is a predetermined relationship. The processing unit 12 obtains the similarity between the first entity and the second entity output by the machine learning model 30. The processing unit 12 generates a knowledge graph 50 by combining the knowledge graphs 20 and 21 as the same entity for the first entity and the second entity whose similarity is greater than a threshold.

Description

[Technical Field]

[0001] This invention relates to a generation method, a generation program, and an information processing device. [Background technology]

[0002] In computer systems, a type of information called a knowledge graph is sometimes used to represent certain knowledge. A knowledge graph is a graph-structured data set that has multiple nodes corresponding to multiple entities, and the relationships between entities are represented by edges connecting the nodes. Entities represent real-world objects and events.

[0003] For example, there is a proposed knowledge graph generator that generates graph data of a knowledge graph in which each word in multiple natural sentences is shown as a node and the relationships between each word in the word pairs within the natural sentences are shown as edges. This knowledge graph generator calculates the cosine similarity for each word pair in each natural sentence pair and connects similar nodes in each natural sentence pair based on the cosine similarity of each word pair in each natural sentence pair.

[0004] Furthermore, there is a proposed processor system that evaluates the semantic similarity between a first knowledge item included in a first knowledge graph and a second knowledge item included in a second knowledge graph, and generates an integrated graph by combining the first and second knowledge graphs if the similarity satisfies predetermined conditions. This processor system calculates the similarity between knowledge items using a correspondence dictionary in which semantic correspondences related to the notation of knowledge items (e.g., names of defect causes) are registered.

[0005] Furthermore, there is a proposed system that indirectly links the first and second knowledge graphs via a graph called a metalayer knowledge graph. In addition, there is a proposal for a knowledge graph fusion device that identifies the same or similar entities from multiple knowledge graphs and fuses the multiple knowledge graphs. The knowledge graph fusion device evaluates the similarity between two entities using the cosine similarity.

[0006] Furthermore, there is a proposal for a system that determines the similarity of subgraphs in each knowledge graph based on the structure of each concept object corresponding to a node and the edges connecting them in each knowledge graph.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Summary of the Invention

Problems to be Solved by the Invention

[0008] It is conceivable to use the sentences included in a document as entities and represent the causal relationships of events and things represented by each sentence in the document by a knowledge graph. By integrating the knowledge graphs corresponding to each document, the knowledge represented by the knowledge graph can be enriched.

[0009] In this case, evaluating the similarity between two entities included in each knowledge graph becomes problematic. For example, existing methods using cosine similarity have low accuracy in estimating similarity between texts. Therefore, existing methods may incorrectly determine that two entities are identical, for example, if one entity represents a first event and the other represents a second event resulting from the first event. Consequently, in the integration of knowledge graphs, these entities may be aggregated into a single entity. As a result, the information from the original knowledge graphs may not be properly reflected in the integrated knowledge graph.

[0010] In one aspect, the present invention aims to improve the accuracy of similarity estimation. [Means for solving the problem]

[0011] In one embodiment, a generation method is provided in which a computer performs the following processes: The computer obtains combinations of a first entity contained in the first knowledge graph and a second entity contained in the second knowledge graph from a first knowledge graph and a second knowledge graph, each of which shows causal relationships between multiple entities. The computer inputs the first and second entities into a machine learning model capable of outputting the similarity of two entities for input of two entities, and instructs the model to reduce the similarity of the first and second entities if the relationship between the first and second entities is a predetermined relationship. The computer obtains the similarity of the first and second entities output by the machine learning model. The computer generates a third knowledge graph by combining the first and second knowledge graphs as the same entity for first and second entities whose similarity is greater than a threshold.

[0012] In one embodiment, a generation program executed by a computer is provided. In another embodiment, an information processing device having a storage unit and a processing unit is provided. [Effects of the Invention]

[0013] In one respect, it can improve the accuracy of similarity estimation. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram illustrating the information processing device of the first embodiment. [Figure 2] This figure shows an example of the hardware of the information processing device according to the second embodiment. [Figure 3] This figure shows an example of a knowledge graph. [Figure 4] This figure shows an example of an integrated knowledge graph. [Figure 5] This figure shows an example of the functions of an information processing device. [Figure 6] The figure shows an example of a prompt. [Figure 7] This flowchart shows an example of generating an integrated knowledge graph. [Figure 8] This flowchart shows an example of the process for identifying identical entities. [Figure 9] This figure shows examples of similarity estimation methods. [Figure 10] This figure shows examples of similarity evaluation results using each similarity estimation method. [Figure 11] This figure shows an example of knowledge graph integration. [Figure 12] This is a diagram explaining ToT (Total of Time). [Figure 13] This is a diagram explaining ICL. [Figure 14] This flowchart shows a modified example of the process for identifying identical entities. [Figure 15] The figure shows an example of a prompt. [Figure 16] The following is a diagram showing an example of a prompt (continued). [Figure 17] This figure shows an example of the functions of the information processing device according to the third embodiment. [Figure 18] This figure shows an example of narrowing down the results using cosine similarity. [Figure 19] This flowchart shows an example of the process for identifying identical entities. [Figure 20] This figure shows an example of the functions of the information processing device according to the fourth embodiment. [Figure 21] This figure shows examples of selecting combinations of entities to be merged. [Figure 22] This flowchart shows an example of the process of joining identical entities. [Figure 23] This figure shows an example of the functions of the information processing device according to the fifth embodiment. [Figure 24] This is a diagram illustrating the setting of thresholds. [Figure 25] This figure shows an example of how to use an integrated knowledge graph. [Modes for carrying out the invention]

[0015] This embodiment will be described below with reference to the drawings. [First Embodiment] A first embodiment will be described.

[0016] Figure 1 is a diagram illustrating the information processing device of the first embodiment. The information processing device 10 performs processing to integrate multiple knowledge graphs. The information processing device 10 has a storage unit 11 and a processing unit 12.

[0017] The memory unit 11 may be a volatile semiconductor memory such as RAM (Random Access Memory), or a non-volatile storage such as an HDD (Hard Disk Drive) or flash memory. The memory unit 11 stores information that represents multiple knowledge graphs. Each of the multiple knowledge graphs contains multiple entities. Each of the multiple entities is a sentence extracted from a document or the like. However, some of the entities among the multiple entities may be words. The knowledge graph represents the causal relationships between each entity in the document or the like from which each entity was extracted. In the knowledge graph, entities are represented as nodes. Furthermore, the causal relationship between two entities is represented by an edge connecting the two nodes corresponding to the two entities.

[0018] The processing unit 12 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or DSP (Digital Signal Processor). However, the processing unit 12 may also include application-specific electronic circuits such as ASICs (Application Specific Integrated Circuits) or FPGAs (Field Programmable Gate Arrays). The processor executes programs stored in memory such as RAM (which may also be the storage unit 11). A collection of multiple processors is sometimes called a "multiprocessor" or simply a "processor."

[0019] The processing unit 12 acquires a first knowledge graph and a second knowledge graph based on information indicating multiple knowledge graphs stored in the storage unit 11. For example, the processing unit 12 acquires knowledge graphs 20 and 21. Knowledge graphs 20 and 21 are examples of the first and second knowledge graphs.

[0020] Knowledge graph 20 has entities A1 and A2, and shows the causal relationship between entities, where the cause of the event represented by entity A1 is the event represented by entity A2. Knowledge graph 21 has entities B1, B2, and B3, and shows the causal relationship between entities, where the cause of the events represented by entities B1 and B2 is the event represented by entity B3. Note that "event" may be considered a term that encompasses "things" and "phenomena".

[0021] The processing unit 12 retrieves combinations of a first entity included in the first knowledge graph and a second entity included in the second knowledge graph. For example, the processing unit 12 can retrieve all combinations of a first entity belonging to the knowledge graph 20 and a second entity belonging to the knowledge graph 21, namely (A1,B1), (A1,B2), (A1,B3), (A2,B1), (A2,B2), and (A2,B3).

[0022] The processing unit 12 uses a machine learning model 30 to estimate the similarity between two entities. The machine learning model 30 can output the similarity between two entities for the input of two entities. The similarity between two entities is the semantic similarity of the two sentences represented by the two entities. The higher the similarity, the greater the degree to which the two entities are similar. For example, the machine learning model 30 may be a Large Language Model (LLM). Examples of LLMs that can be used as machine learning model 30 include Llama 2, 3, Command r+, GPT-4 (registered trademark), and GEMINI (registered trademark). The machine learning model 30 may be stored in the memory unit 11, or it may be provided by another information processing device that communicates with the information processing device 10 via a network.

[0023] The processing unit 12 inputs the first entity and the second entity into the machine learning model 30 and instructs it to reduce the similarity between the first entity and the second entity if the relationship between the first entity and the second entity is a predetermined relationship.

[0024] Here, the specified relationships include, for example, cause and effect, simultaneous occurrence, and subject and predicate relationships. A cause and effect relationship is, for example, a relationship in which an event represented by a first entity causes an event represented by a second entity to occur. In this case, the first entity is the cause, and the second entity is the effect corresponding to that cause. A simultaneous occurrence relationship is a relationship in which an event represented by a second entity occurs at the same time as an event represented by a first entity. A subject and predicate relationship is a relationship in which, in a given sentence, the event represented by the first entity is the subject (or subject phrase), and the event represented by the second entity is the predicate (or predicate phrase).

[0025] Prompt 40 is an example of an instruction statement for the machine learning model 30, which is input to the machine learning model 30 by the processing unit 12. In one example, prompt 40 includes a statement that instructs the input of entities and the output of similarity, such as "Output the similarity between sentence X and sentence Y." Sentences X and Y correspond to two entities input to the machine learning model 30. Prompt 40 also includes a statement that instructs the definition of similarity, such as "The similarity is expressed as 0% to 100%, with 100% if the meanings are the same and 0% if the meanings are completely different."

[0026] Furthermore, prompt 40 includes a sentence that imposes a constraint: "Reduce the similarity if sentence X and sentence Y have a relationship ***." For example, processing unit 12 can specify at least one of the following as "relationship ***": "cause-and-effect relationship," "coincidence relationship," "subject-predicate relationship," etc.

[0027] For example, the processing unit 12 uses the prompt 40 to input the combination of entities A1 and B1, along with the definition and constraints of similarity, into the machine learning model 30. For example, if the machine learning model 30 is provided by another information processing device, the processing unit 12 can input the prompt 40 into the machine learning model 30 by transmitting the information of the prompt 40 to the other information processing device via the network.

[0028] Whether entities A1 and B1 have the relationship specified by the constraint is determined by the machine learning model 30. For example, when the processing unit 12 specifies a relationship such as "cause-and-effect relationship," "co-occurrence relationship," or "subject-predicate relationship" in the prompt 40, it may also include an example of two entities that have the specified relationship in the prompt 40. This allows the processing unit 12 to improve the accuracy of the machine learning model 30's determination of whether or not the relationship exists.

[0029] The processing unit 12 then obtains the similarity between the first entity and the second entity, which is output by the machine learning model 30. For example, by performing the above process for each combination of the two entities (A1, B1), (A1, B2), ... the processing unit 12 can obtain the similarity of each combination, such as the similarity of entities A1, B1, the similarity of entities A1, B2, ...

[0030] The processing unit 12 compares the similarity of each combination of the first entity and the second entity to a threshold. The processing unit 12 generates a third knowledge graph by combining the first and second knowledge graphs, treating the first and second entities whose similarity is greater than the threshold as the same entity. The processing unit 12 does not consider the first and second entities whose similarity is less than or equal to the threshold to be the same entity. Here, if the similarity between the first entity and the second entity is greater than the threshold, the first and second entities can be said to be entities with the same meaning that are common to the first and second knowledge graphs.

[0031] Knowledge Graph 50 is an example of a knowledge graph after combining knowledge graphs 20 and 21. For example, the processing unit 12 detects that the similarity between entities A1 and B1 is greater than a threshold. Then, the processing unit 12 generates knowledge graph 50 by combining knowledge graphs 20 and 21, treating entity A1 from knowledge graph 20 and entity B1 from knowledge graph 21 as the same entity. More specifically, the processing unit 12 generates knowledge graph 50 by combining entity A1 from knowledge graph 20 and entity B1 from knowledge graph 21 into a single entity. For example, the processing unit 12 aggregates entity A1 into entity B1 and leaves entity B1. In knowledge graph 20, entity A1 has a causal relationship with entity A2. Therefore, in knowledge graph 50, entity B1 has a causal relationship with entity A2 instead of entity A1.

[0032] The information in the knowledge graph 50 is stored in the storage unit 11. For example, the knowledge graph 50 indicates that the cause of an event represented by entity B1 is an event represented by either entity A2 or B3, and the cause of an event represented by entity B2 is an event represented by entity B3.

[0033] Furthermore, the process by the processing unit 12 to generate the third knowledge graph can also be described as a process that generates a third knowledge graph by combining the first and second knowledge graphs, for example, by fusing one of the first and second entities whose similarity is greater than a threshold with the other entity.

[0034] According to the information processing device 10, combinations of a first entity included in the first knowledge graph and a second entity included in the second knowledge graph are obtained from the first knowledge graph and the second knowledge graph. The first entity and the second entity are input to a machine learning model capable of outputting the similarity between two entities for the input of two entities. At the same time, the model is instructed to reduce the similarity between the first entity and the second entity if the relationship between them is a predetermined relationship. The similarity between the first entity and the second entity output by the machine learning model is obtained. The first entity and the second knowledge graph are combined as the same entity for the first entity and the second entity whose similarity is greater than a threshold, thereby generating a third knowledge graph.

[0035] This allows the information processing device 10 to improve the accuracy of estimating the similarity between two entities. Specifically, the information processing device 10 can reduce the possibility that the first entity and the second entity are judged to be identical when they have predetermined relationships such as cause and effect, simultaneous occurrence, or subject and predicate. Therefore, when the first entity and the second entity have such predetermined relationships, the information processing device 10 can reduce the possibility that these entities will be aggregated into a single entity in the integrated knowledge graph. In other words, the information processing device 10 can suppress unnecessary aggregation of entities in the knowledge graph merging process and appropriately reflect the information held by each knowledge graph before merging in the merged knowledge graph.

[0036] Furthermore, the processing unit 12 can further combine another knowledge graph with the knowledge graph 50 using the method exemplified in the first embodiment. By sequentially repeating this combination for each knowledge graph, the processing unit 12 can create a knowledge graph that integrates multiple knowledge graphs. As a result of improved accuracy in estimating the similarity between two entities, the information processing device 10 can appropriately integrate the knowledge graphs.

[0037] The integrated knowledge graph can be used, for example, as training data for machine learning models such as LLMs used in question-answering systems that respond to natural language questions from users. In this way, the question-answering system can output detailed answers based on knowledge integrated from the contents of multiple documents, or multiple patterns of answers.

[0038] [Second Embodiment] Next, a second embodiment will be described. Figure 2 shows an example of the hardware of the information processing device according to the second embodiment.

[0039] The information processing device 100 performs a process to integrate multiple knowledge graphs. A knowledge graph is a graph-structured data that has multiple nodes corresponding to multiple entities, and the relationships between entities are represented by edges connecting the nodes.

[0040] The information processing device 100 includes a processor 101, RAM 102, HDD 103, GPU 104, input interface 105, media reader 106, and communication interface 107. These units of the information processing device 100 are connected to a bus internally. The processor 101 corresponds to the processing unit 12 in the first embodiment. The RAM 102 or HDD 103 corresponds to the storage unit 11 in the first embodiment. The information processing device 100 may also be a computer.

[0041] The processor 101 is an arithmetic unit that executes program instructions. The processor 101 is, for example, a CPU. The processor 101 loads at least a portion of the program and data stored in the HDD 103 into the RAM 102 and executes the program. The processor 101 may include multiple processor cores. The information processing device 100 may have multiple processors. The processor that executes one of the multiple processes performed by the information processing device 100 may be different from the processor that executes a different process from the multiple processes. A collection of multiple processors is sometimes called a "multiprocessor" or simply a "processor". A processor may also be called a "processor circuitry".

[0042] RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by the processor 101 and data used by the processor 101 for calculations. The information processing device 100 may also be equipped with other types of memory, and may be equipped with multiple types of memory.

[0043] HDD103 is a non-volatile storage device that stores software programs such as the OS (Operating System), middleware, and application software, as well as data. The information processing device 100 may also be equipped with other types of storage devices such as flash memory or SSD (Solid State Drive), and may be equipped with multiple non-volatile storage devices.

[0044] The GPU 104 outputs an image to the display 111 connected to the information processing unit 100, according to instructions from the processor 101. Any type of display can be used as the display 111, such as a CRT (Cathode Ray Tube) display, an LCD (Liquid Crystal Display), a plasma display, or an organic electro-luminescence (OEL) display.

[0045] The input interface 105 acquires input signals from the input device 112 connected to the information processing device 100 and outputs them to the processor 101. The input device 112 can be a pointing device such as a mouse, touch panel, touchpad, or trackball, or a keyboard, remote controller, or button switch. Furthermore, multiple types of input devices may be connected to the information processing device 100.

[0046] The media reader 106 is a reading device that reads programs and data recorded on the recording medium 113. The recording medium 113 can be, for example, a magnetic disk, an optical disk, a magneto-optical disk (MO), or semiconductor memory. Magnetic disks include flexible disks (FD) and HDDs. Optical disks include CDs (Compact Discs) and DVDs (Digital Versatile Discs).

[0047] The media reader 106 copies programs and data read from the recording medium 113 to other recording media such as RAM 102 or HDD 103. The read programs are executed by the processor 101, for example. The recording medium 113 may be a portable recording medium and may be used for distributing programs and data. The recording medium 113 and HDD 103 are sometimes referred to as computer-readable recording media.

[0048] The communication interface 107 is connected to the network 114 and communicates with other information processing devices via the network 114. The communication interface 107 may be a wired communication interface connected to a wired communication device such as a switch or router, or a wireless communication interface connected to a wireless communication device such as a base station or access point.

[0049] Figure 3 shows an example of a knowledge graph. For example, a knowledge graph can be created from documentation about failure cases in computer systems. However, a knowledge graph may also be created from documentation about events other than failures in computer systems. In the following, a knowledge graph may be abbreviated as KG (Knowledge Graph).

[0050] Figure 3(A) shows KG60 created from failure case 1. Figure 3(B) shows KG70 created from failure case 2. In the example in Figure 3, entities are created by dividing the failure case text, which includes the failure content and cause, into sentences that represent a single content. However, some entities in the knowledge graph may be words.

[0051] KG60 includes entities 61, 62, 63, 64, and 65. KG60 has a node corresponding to each of entities 61 through 65, and the causal relationships between entities 61 through 65, such as causes or suggestions, are represented by directed edges connecting the nodes.

[0052] KG70 includes entities 71, 72, 73, 74, and 75. KG70 has a node corresponding to each of entities 71 through 75, and causal relationships, such as causes or suggestions, between entities 71 through 75 are represented by directed edges connecting the nodes.

[0053] The information processing device 100 can combine KG60 and KG70 by aggregating entities representing the same event from each pair of entities in KG60 and KG70 into a single entity. The KG generated by combining multiple KGs is called an integrated KG.

[0054] In the example of KG60,70, entities 64 and 73 represent the same event. Entities 64 and 73 are entities with the same meaning. Also, entities 65 and 74 represent the same event. That is, entities 65 and 74 are entities with the same meaning.

[0055] Figure 4 shows an example of an integrated knowledge graph. Integrated KG80 is an example of a KG generated by combining KG60 and 70. The information processing device 100 generates Integrated KG80 by aggregating entities 64 and 73 and entities 65 and 74 in KG60 and 70. Integrated KG80 includes entities 81, 82, 83, 84, 85, 86, 87, and 88. Entities 81, 82, 83, 86, 87, and 88 correspond to entities 61, 62, 63, 71, 72, and 75, respectively. Entity 84 is an aggregate of entities 64 and 73. Entity 85 is an aggregate of entities 65 and 74. When aggregating, one of the sentences of the entity pair before aggregation (for example, the sentence on the Integrated KG side) is used as the sentence of the entity after aggregation.

[0056] In this way, the information processing device 100 can integrate multiple KGs to reach the root cause of a failure that cannot be determined from individual failure cases alone. Therefore, the information processing device 100 provides a function to accurately identify combinations of entities with the same meaning from each KG and create an appropriate integrated KG.

[0057] Figure 5 shows an example of the functions of an information processing device. The information processing device 100 includes an entity combination creation unit 120, an identical entity determination unit 130, and an identical entity joining unit 140. The entity combination creation unit 120, the identical entity determination unit 130, and the identical entity joining unit 140 are realized by the execution of a program stored in RAM 102 by the processor 101. Although not shown in the figures, the information processing device 100 also has a data storage unit realized using the storage areas of RAM 102 and HDD 103, and stores data used for processing by the information processing device 100 in the data storage unit.

[0058] The entity combination creation unit 120 accepts input from multiple KGs to be integrated and creates combinations (pairs) between the entities of one of the KGs (input KG) and the entities of the integrated KG that is currently being created. If the integrated KG has not yet been created, the entity combination creation unit 120 creates entity combinations from two of the multiple KGs that are input. The entity combination creation unit 120 extracts one entity from each of the two relevant KGs and creates all possible pairs.

[0059] The identical entity determination unit 130 determines the identity of two entities belonging to each pair of entities created by the entity combination creation unit 120. The identical entity determination unit 130 includes a prompt generation unit 131 and an LLM similarity estimation unit 132.

[0060] The prompt generation unit 131 generates prompts to send to the LLM server 200. The LLM server 200 is a server computer that can communicate with the information processing device 100 via the network 114. The LLM server 200 provides an LLM that outputs answers to questions entered in natural language.

[0061] The prompt generation unit 131 generates prompts that instruct the user to respond regarding the similarity of two entities. The prompt generation unit 131 includes instructions in the prompt to lower the similarity of the two entities if they have a predetermined relationship such as cause and effect, simultaneous occurrence, or subject and predicate, or to consider the two entities as different.

[0062] The LLM similarity estimation unit 132 estimates the similarity between two entities using LLM provided by the LLM server 200, based on the prompt generated by the prompt generation unit 131. The similarity estimated using LLM is called LLM similarity. The LLM similarity estimation unit 132 inputs a prompt to the LLM server 200 and obtains the LLM similarity estimated by LLM based on the prompt from the LLM server 200.

[0063] The LLM provided by the LLM server 200 here is, for example, GPT-4 and GEMINI. However, the information processing device 100 may have LLMs in a local environment such as Llama2,3 or Command r+. In that case, the identical entity determination unit 130 may use the LLMs in that local environment to estimate the LLM similarity.

[0064] The identical entity determination unit 130 determines whether two entities are identical based on their LLM similarity. For example, the identical entity determination unit 130 compares the LLM similarity of the two entities with a predetermined threshold. If the LLM similarity is greater than the threshold, the identical entity determination unit 130 determines that the two entities are identical. If the LLM similarity is less than or equal to the threshold, the identical entity determination unit 130 determines that the two entities are not identical.

[0065] The identical entity joining unit 140 joins the two KGs by fusing the two entities that the identical entity determination unit 130 has determined to be identical. The entity combination creation unit 120 then creates a combination of two entities again using the next input KG from the multiple input KGs and the integrated KG that is currently being created. The processes of the identical entity determination unit 130 and the identical entity joining unit 140 are then repeated. In this way, when the integration of the multiple KGs into the integrated KG is complete, the identical entity joining unit 140 outputs information about the completed integrated KG.

[0066] Figure 6 shows an example of a prompt. Prompt 131a is an example of a prompt generated by the prompt generation unit 131. The prompt generation unit 131 generates prompt 131a for two entities, namely, sentence X and sentence Y.

[0067] Prompt 131a includes a definition of LLM similarity and instructions to reduce the similarity (LLM similarity) between two entities if the relationship between the two entities falls into any of the predefined relationship lists (e.g., cause and effect, parallel, subject and predicate).

[0068] For example, prompt 131a includes the instruction, "Output the similarity between sentence X and sentence Y," as an instruction to input entities and output LLM similarity. Prompt 131a also includes the definition of LLM similarity, "Similarity is expressed as 0 to 100%, with 100% if the meanings are the same and 0% if the meanings are completely different." Prompt 131a also includes the constraint on LLM similarity, "Reduce the similarity if sentence X and sentence Y are in a cause-and-effect, parallel, or subject-and-predicate relationship."

[0069] The LLM similarity estimation unit 132 estimates LLM similarity for two entities (texts X and Y) using LLM based on the prompt 131a. Next, the processing procedure of the information processing device 100 will be explained.

[0070] Figure 7 is a flowchart showing an example of the generation of an integrated knowledge graph. (S10) The information processing device 100 repeats steps S11 to S13 for the KGs (input KGs) that are input sequentially (input KG loop). When step S10 is executed for the first time, the information processing device 100 considers one of the first two input KGs to be the KG to be integrated (integrated KG) and executes steps S11 to S13.

[0071] (S11) The entity combination creation unit 120 creates combinations of entities from the input KG and the integrated KG. For example, the entity combination creation unit 120 comprehensively creates all possible combinations.

[0072] (S12) The identical entity determination unit 130 performs identical entity determination processing based on LLM similarity. Details of the identical entity determination processing will be described later. (S13) The identical entity joining unit 140 incorporates the input KG into the integrated KG by joining the entities that the identical entity determination unit 130 has determined to be identical. In other words, the identical entity joining unit 140 aggregates the entities of the input KG that have been determined to be identical to the entities of the integrated KG into the corresponding entities of the integrated KG. As a result, the corresponding entities of the integrated KG will have the same connection relationships with other entities as the aggregated entities in the input KG.

[0073] (S14) When the information processing device 100 has completed the integration of all of the multiple KGs into the integrated KG, it completes the input KG loop and proceeds to step S15. (S15) The identical entity joining unit 140 outputs the information of the generated integrated KG.

[0074] Figure 8 is a flowchart showing an example of the process for identifying identical entities. The process of identifying identical entities corresponds to step S12. (S20) The identical entity determination unit 130 repeats steps S21 to S26 for each combination of entities.

[0075] (S21) The prompt generation unit 131 generates a prompt containing instructions to output the LLM similarity for the two entities in question, and includes the definition of LLM similarity in the prompt.

[0076] (S22) The prompt generation unit 131 includes a constraint in the prompt that the LLM similarity between the two entities will be reduced if the relationship between the two entities is one of the following: cause and effect, parallel, subject and predicate.

[0077] (S23) The LLM similarity estimation unit 132 uses LLM to estimate the LLM similarity S for the two entities in question.LLM is estimated. For example, the LLM similarity estimation unit 132 inputs the prompt generated by the prompt generation unit 131 into the LLM server 200, and obtains the LLM similarity S LLM from the LLM server 200.

[0078] (S24) The same entity determination unit 130 determines whether the LLM similarity S LLM is greater than the threshold value S LLM th . If the LLM similarity S LLM is greater than the threshold value S LLM th , the process proceeds to step S25. If the LLM similarity S LLM is less than or equal to the threshold value S LLM <\ th , the process proceeds to step S26.

[0079] (S25) The same entity determination unit 130 determines that the two entities are the same. Then, the process proceeds to step S27. (S26) The same entity determination unit 130 determines that the two entities are different. Then, the process proceeds to step S27.

[0080] (S27) The same entity determination unit 130 repeats steps S21 to S26 for all combinations of entities created from the input KG and the integrated KG, and then completes the repetition. Then, the same entity determination unit 130 notifies the same entity combination unit 140 of the information of the two entities determined to be the same in step S25, and ends the same entity discrimination process.

[0081] Note that, for example, it may be determined that there is no combination of the same entity between the input KG and the integrated KG. In that case, the same entity combination unit 140 can incorporate the input KG into the integrated KG without aggregating the entities with the integrated KG. In this case, the integrated KG will include multiple systems of KG including the existing integrated KG and the input KG.

[0082] In this way, the information processing device 100 can improve the accuracy of LLM similarity estimation by including a constraint in the prompt when estimating LLM similarity that reduces the LLM similarity if the relationship between the two entities is a predetermined relationship.

[0083] Figure 9 shows a comparison of similarity estimation methods. As a first example of a method for estimating similarity between two entities, we can consider a method for estimating cosine similarity. Figure 9(A) shows an example of obtaining cosine similarity for texts X and Y. When calculating cosine similarity, first, text X is converted into text vector x and text Y is converted into text vector y by the text vector transformation model 90. The text vector transformation model 90 is a machine learning model that vectorizes text. The text vector transformation model 90 can also be said to be a language model that vectorizes text. Text vector x is a vector whose elements are multiple features related to text X. Text vector y is a vector whose elements are multiple features related to text Y.

[0084] Then, the cosine similarity calculation unit 91, implemented in the information processing device 100, calculates the cosine similarity between the text vectors x and y. For example, equation (1) is used to calculate the cosine similarity.

[0085]

number

[0086] Furthermore, as a second example of a similarity estimation method for two entities, one can consider using a "naive prompt" as the prompt for LLM similarity estimation. A "naive prompt" is a prompt that does not include the constraints exemplified in prompt 131a. Figure 9(B) shows an example of a naive prompt 92. Prompt 92 does not include the said constraints on LLM similarity.

[0087] Figure 10 shows examples of the similarity evaluation results using each similarity estimation method. Figure 10 illustrates the results of evaluating two entities whose semantic agreement / disagreement is known in advance, using cosine similarity, LLM similarity based on a naive prompt, and LLM similarity based on a prompt generated by the information processing device 100.

[0088] Figure 10(A) shows the evaluation result 93 using cosine similarity. Figure 10(B) shows the evaluation result 94 using LLM similarity for a naive prompt. Figure 10(C) shows the evaluation result 95 using LLM similarity for a prompt with constraints generated by the information processing device 100. The horizontal axis of evaluation result 93 is cosine similarity. The horizontal axis of evaluation results 94 and 95 is LLM similarity. In evaluation results 93, 94 and 95, pairs of entities with matching meanings are plotted as rectangles, and pairs of entities with different meanings are plotted as circles.

[0089] Evaluation result 93 indicates that cosine similarity cannot adequately separate pairs of entities with matching meanings from pairs of entities with different meanings. Evaluation result 94 indicates that LLM similarity obtained for naive prompts cannot adequately separate pairs of entities with matching meanings from pairs of entities with different meanings.

[0090] On the other hand, evaluation result 95 shows that by using the LLM similarity obtained for a prompt that includes the constraint of LLM similarity, it is possible to appropriately separate pairs of entities with the same meaning from pairs of entities with different meanings. In this way, the information processing device 100 can improve the accuracy of estimating LLM similarity for two entities.

[0091] Figure 11 shows an example of knowledge graph integration. In calculating LLM similarity, there are cases where two entities with a cause-and-effect relationship are judged to have similar meanings. However, in the case of KG, where the goal is to identify causal relationships, it is preferable to exclude cause-and-effect relationships from consideration when estimating LLM similarity.

[0092] For example, KG301 indicates that the cause of the entity "my stomach rumbled" is the entity "I hit my stomach." KG302 indicates that the cause of the entity "my stomach rumbled" is the entity "I was hungry." Also, KG302 indicates that the cause of the entity "I ate a lot" is the entity "I was hungry."

[0093] When integrating KG301 and 302, the "stomach rumbled" entity in KG301 may be judged to have the same meaning as the "stomach rumbled" entity in KG302. In that case, the "stomach rumbled" entity in KG301 may also be judged to have a similar meaning to the "hungry" entity in KG302, which is the cause of the "stomach rumbled" entity.

[0094] As a result, as shown in the example of integration failure, KG303 may be created in which the three entities "stomach rumbled," "stomach rumbled," and "hungry" are merged into a single entity, "hungry." In KG303, the causal relationship between the entity "stomach rumbled" and the entity "hungry" (or between the entity "stomach rumbled" and the entity "hungry") is lost.

[0095] Therefore, the information processing device 100 can improve the accuracy of LLM similarity estimation by including a prompt that, along with the output instruction for LLM similarity, lowers the LLM similarity score when the two entities have a predetermined relationship such as cause and effect, parallelism, or subject and predicate.

[0096] Specifically, in the examples of KG301 and KG302, the entity "my stomach rumbled" in KG301 and the entity "my stomach rumbled" in KG302 can be judged as entities with the same meaning. In this case, the entity "I'm hungry" in KG302 is judged to be the cause of the entity "my stomach rumbled" in KG301, which is identified with the entity "my stomach rumbled" in KG302. As a result, the estimated LLM similarity for the pair of the entity "my stomach rumbled" in KG301 and the entity "I'm hungry" in KG302, which is the cause of that entity, becomes relatively low. For this reason, the information processing device 100 can create KG304 which includes the causal relationship between the entity "my stomach rumbled" and the entity "I'm hungry" (or the causal relationship between the entity "my stomach rumbled" and the entity "I'm hungry"), as shown in the successful integration example.

[0097] Here, the information processing device 100 can use methods such as ToT (Tree of Thought) or ICL (In Context Learning) when writing prompts in order to further improve the accuracy of LLM similarity.

[0098] Figure 12 is a diagram illustrating ToT. ToT is a prompting method that prompts an LLM to offer multiple opinions on a given question, repeatedly self-evaluating and exploring each opinion in depth, and ultimately outputting a conclusion.

[0099] Figure 12(A) shows Flow 311, which represents a simple question-and-answer example. Flow 311 illustrates the process by which the LLM outputs the answer "Yes" in response to a very simple prompt, "Are A and B the same?". For very simple prompts, the LLM directly outputs the answer to the prompt and does not verify the answer.

[0100] Figure 12(B) shows Flow 312, which represents an example of ToT. Flow 312 illustrates the process in which the LLM outputs the answer "They are different" in response to the prompt "Three experts should give their opinions on whether A and B are the same, evaluate each other, and finally reach a conclusion." Flow 312 includes multiple opinions 1-3, 1', 2', 2'' output by the LLM in response to the answer. Opinions 1, 2, 2'' are opinions that are in favor. Opinions 1', 2', 3 are opinions that are rejected. In this way, the information processing device 100 may obtain a consensus-based answer from the LLM using a ToT-based prompt. By using the ToT method, the information processing device 100 can further improve the accuracy of estimating the LLM similarity between entities.

[0101] Figure 13 is a diagram illustrating ICL. ICL is a technique that improves the accuracy of LLM answers by including hints for answering questions, such as example answers, in the prompts.

[0102] Prompt 313 shows an example of an instruction statement using ICL. The prompt generator 131 can provide an example answer to the question, "Please output the similarity between sentence X and sentence Y," for example, that the similarity between sentence 1, "Alice is not at school," and sentence 2, "Alice has not come to school," is "95%." The information processing device 100 can further improve the accuracy of LLM similarity estimation between entities by using the ICL method.

[0103] Figure 14 is a flowchart showing a modified version of the identical entity identification process. The process of identifying identical entities corresponds to step S12. The procedure in Figure 14 differs from the procedure illustrated in Figure 8 in that step S21a is executed instead of steps S21 and S22. Therefore, the following will mainly explain step S21a, and the explanations of steps S20, S23 to S27 will be omitted. Step S21a is executed within the loop for the entity combinations in step S20.

[0104] (S21a) The prompt generation unit 131 generates a prompt similar to ToT. The prompt generation unit 131 may generate a prompt similar to ICL instead of ToT. The prompt generation unit 131 may generate a prompt that combines the methods of ToT and ICL. By using a prompt that combines the methods of ToT and ICL, the prompt generation unit 131 may be able to improve the estimation accuracy of LLM similarity compared to using ToT or ICL alone. In step S21a, the prompt generation unit 131 also includes the similarity definition from step S21 and the constraints from step S22 in the prompt. Then the process proceeds to step S23.

[0105] Next, we will explain specific examples of prompts generated by the prompt generation unit 131 that conform to ToT and ICL. Figure 15 shows an example of a prompt.

[0106] Prompt 131b is an example of a prompt similar to ToT. Prompt 131b includes, for example, instructions for a consensus-based response by three experts. Prompt 131c is an example prompt that includes input of two entities, a definition of similarity, and a description of constraints.

[0107] In prompt 131c, "Event 1" and "Event 2" each correspond to entities. In prompt 131c, "Description 1" is the description before and after "Event 1" in the first document (the section from which "Event 1" was extracted). Similarly, "Description 2" is the description before and after "Event 2" in the second document (the section from which "Event 2" was extracted).

[0108] The prompt generation unit 131, by specifying 100% as the upper limit of the LLM similarity and 0% as the lower limit of the LLM similarity via prompt 131c, can obtain LLM similarity values ​​ranging from 0% to 100%.

[0109] Prompt 131c includes the following statement as a "relational constraint": Even when there is a causal relationship or one represents another, the similarity score will be low because the specific elements are different. For example, if "event A indicates event B," "event A occurs when event B occurs," or "event A is caused by event B," the similarity score will be low. Also, for example, in the sentence "Alice is a doctor," the similarity score between "Alice" and "doctor" will be low because these two elements are different. Prompt 131c includes a description of specific examples of relationships between entities in accordance with ICL in the "Relationship Constraints" section. "Event A indicates Event B" is a specific example of a relationship where "Event A" suggests "Event B". "Event A occurs when Event B occurs" is a specific example of a case where "Event A" and "Event B" have a relationship of simultaneous occurrence. "Event A is caused by Event B" is a specific example of a relationship where "Event B" is the cause and "Event A" is the effect. "Alice is a doctor" is a specific example of a relationship where "Alice" (Event A or Event B) is the subject and "Doctor" (Event B or Event A) is the predicate.

[0110] Figure 16 shows an example of a prompt (continued). Prompt 131d is an example of a prompt in accordance with ICL. Prompt 131d includes descriptions of specific examples of "Event 1" extracted from "Description 1" and "Event 2" extracted from "Description 2," as well as examples of responses for the LLM similarity between "Event 1" and "Event 2." As shown in prompt 131d, the prompt generation unit 131 may also include an example output of the reason for determining the LLM similarity along with the example responses for the LLM similarity.

[0111] In this way, the information processing device 100 can further improve the accuracy of LLM similarity estimation between entities by using the ToT and ICL methods. The information processing device 100 can use prompts 131b, 131c, and 131d in any combination as appropriate. For example, the information processing device 100 may obtain LLM similarity using prompts 131c and 131d without using prompt 131b. The information processing device 100 may obtain LLM similarity using prompts 131b and 131c without using prompt 131d. The information processing device 100 may obtain LLM similarity using only prompt 131c among prompts 131b, 131c, and 131d. If prompt 131d is not used, the prompt generation unit 131 does not need to include sentences related to "Description 1" and "Description 2" in prompt 131c.

[0112] [Third Embodiment] Next, a third embodiment will be described. The differences from the second embodiment described above will be explained primarily, and the common aspects will be omitted.

[0113] The information processing device 100 of the third embodiment provides a function to narrow down the combination of entities for which LLM similarity is to be obtained. Figure 17 shows an example of the functions of the information processing device according to the third embodiment.

[0114] The information processing device 100 includes an entity combination creation unit 120, an identical entity determination unit 130, and an identical entity joining unit 140. In this third embodiment, the identical entity determination unit 130 differs from the second embodiment in that, in addition to the prompt generation unit 131 and the LLM similarity estimation unit 132, it also includes a cosine similarity estimation unit 133.

[0115] The cosine similarity estimation unit 133 estimates the cosine similarity for the combinations of entities created by the entity combination creation unit 120. The cosine similarity estimation unit 133 compares the cosine similarity with a threshold and selects combinations whose cosine similarity is greater than the threshold as targets for LLM similarity estimation.

[0116] The cosine similarity estimation unit 133, as illustrated in Figure 9(A), uses a text vector transformation model to transform each entity into a text vector and calculates the cosine similarity of each text vector using equation (1).

[0117] The prompt generation unit 131 generates prompts for combinations of entities whose cosine similarity is greater than a threshold. The LLM similarity estimation unit 132 then uses these prompts to estimate the LLM similarity for combinations of entities whose cosine similarity is greater than a threshold.

[0118] Figure 18 shows an example of narrowing down results using cosine similarity. Graphs 321 and 322 show example plots of cosine similarity and LLM similarity obtained for combinations of entities with matching meanings and combinations of entities with different meanings. The horizontal axis of Graphs 321 and 322 represents cosine similarity. The vertical axis of Graphs 321 and 322 represents LLM similarity.

[0119] Graph 321 shows the threshold S for cosine similarity. cos th This is described. Graph 322 shows the threshold S for LLM similarity. LLM th It is described as follows. As shown in Graph 321, the information processing device 100 uses cosine similarity and threshold S cos thBy comparing with the others, the number of candidate combinations of entities with matching meanings can be narrowed down. The information processing device 100 can reduce the number of combinations of entities to be estimated for LLM similarity by narrowing down the candidates using cosine similarity, thereby reducing the processing load related to LLM similarity estimation. In addition, the information processing device 100 can determine combinations of entities with matching meanings at high speed. Furthermore, the information processing device 100 can improve the accuracy of determining combinations of entities with matching meanings.

[0120] Figure 19 is a flowchart showing an example of the process for identifying identical entities. The information processing device 100 of the third embodiment performs the following procedure instead of the procedure of the second embodiment illustrated in Figure 8. The identical entity determination process corresponds to step S12.

[0121] (S30) The identical entity determination unit 130 repeats steps S31 to S37 for each combination of entities. (S31) The cosine similarity estimation unit 133 calculates the cosine similarity S for the two entities in question. cos Calculate.

[0122] (S32) The cosine similarity estimation unit 133 calculates the cosine similarity S cos is the threshold S cos th Determine whether it is greater than or less than the cosine similarity S. cos is the threshold S cos th If it is greater than, the process proceeds to step S33. Cosine similarity S cos is the threshold S cos th In the following cases, the process proceeds to step S37.

[0123] (S33) The prompt generation unit 131 generates a prompt similar to ToT. The prompt generation unit 131 may generate a prompt similar to ICL instead of ToT. The prompt generation unit 131 may generate a prompt that combines the methods of ToT and ICL. By using a prompt that combines the methods of ToT and ICL, the prompt generation unit 131 may be able to improve the estimation accuracy of LLM similarity compared to using ToT or ICL alone. In step S33, the prompt generation unit 131 also includes the similarity definition from step S21 and the constraints from step S22 in the prompt.

[0124] (S34) The LLM similarity estimation unit 132 uses LLM to estimate the LLM similarity S for the two entities in question. LLM For example, the LLM similarity estimation unit 132 inputs the prompt generated by the prompt generation unit 131 to the LLM server 200, and the LLM similarity S is estimated from the LLM server 200. LLM Obtain it.

[0125] (S35) The identical entity determination unit 130 determines the LLM similarity S LLM is the threshold S LLM th Determine whether it is greater or less. LLM similarity S LLM is the threshold S LLM th If it is greater, the process proceeds to step S36. LLM Similarity S LLM is the threshold S LLM th In the following cases, the process proceeds to step S37.

[0126] (S36) The identical entity determination unit 130 determines that both entities are identical. Then the process proceeds to step S38. (S37) The identical entity determination unit 130 determines that the two entities are inconsistent. Then, the process proceeds to step S38.

[0127] (S38) The identical entity determination unit 130 repeats steps S31 to S37 for all combinations of entities created from the input KG and the integrated KG, and then completes the repetition. The identical entity determination unit 130 then notifies the identical entity joining unit 140 of the information of the two entities determined to be identical in step S36, and terminates the identical entity determination process.

[0128] In addition, in the procedure shown in Figure 19, the prompt generation unit 131 may execute steps S21 and S22 instead of step S33. In other words, the prompt generation unit 131 does not need to use the ToT or ICL method in generating prompts.

[0129] In this way, the information processing device 100 can reduce the number of combinations of entities to be estimated for LLM similarity by narrowing down the candidate combinations of entities to be estimated for LLM similarity using cosine similarity. Therefore, the information processing device 100 can reduce the processing load on the LLM server 200 related to LLM similarity estimation. In addition, the information processing device 100 can omit steps S33 to S35 by the amount by which the number of combinations of entities to be estimated for LLM similarity has decreased. Therefore, the information processing device 100 can speed up the determination of combinations of entities that have matching meanings. As a result, the information processing device 100 can speed up the generation of the integrated KG.

[0130] [Fourth Embodiment] Next, a fourth embodiment will be described. The differences from the second and third embodiments described above will be explained primarily, and the common aspects will be omitted.

[0131] Here, the information processing device 100 determines that entities whose LLM similarity is greater than a threshold are identical, and creates a unified KG by fusing those entities in the two KGs. In this case, if there are multiple entities in the other KG that are determined to be identical to an entity in the other KG, those multiple entities in the other KG may be aggregated into a single entity in the unified KG. As a result, an excessive amount of information that the original KG held may be lost in the unified KG.

[0132] Therefore, the information processing device 100 of the fourth embodiment provides a function to integrate KGs such that the information from the original KGs is appropriately reflected in the integrated KGs. Figure 20 shows an example of the functions of the information processing device according to the fourth embodiment.

[0133] The information processing device 100 includes an entity combination creation unit 120, an identical entity determination unit 130, and an identical entity joining unit 140. In this fourth embodiment, the identical entity joining unit 140 differs from the third embodiment in that it includes a first maximum similarity selection unit 141 and a second maximum similarity selection unit 142.

[0134] The first maximum similarity selection unit 141 obtains combinations (pairs) of entities determined by the identical entity determination unit 130 to have an LLM similarity greater than a threshold. From all the obtained combinations, the first maximum similarity selection unit 141 selects the pair with the highest LLM similarity between each entity on the input KG side and the entity on the integrated KG side.

[0135] The second maximum similarity selection unit 142 selects from each pair selected by the first maximum similarity selection unit 141 the pair that has the highest LLM similarity between each entity on the integrated KG side and the entity on the input KG side.

[0136] The same entity joining unit 140 merges the entities of the input KG belonging to the pair selected by the second maximum similarity selection unit 142 into the entities of the integrated KG, thereby joining the input KG to the integrated KG.

[0137] Figure 21 shows an example of selecting a combination of entities to be merged. In Figure 21, the three entities "Entity 1, 2, 3" are entities of the integrated KG. The two entities "Entity a, b" are entities of the input KG. Assume that the identical entity joining unit 140 obtained four combinations of entities, (2,a), (2,b), (3,a), and (3,b), from the identical entity determination unit 130.

[0138] The LLM similarity of combination (2,a) is 0.9. The LLM similarity of combination (2,b) is 0.87. The LLM similarity of combination (3,a) is 0.89. The LLM similarity of combination (3,b) is 0.86.

[0139] First, the first maximum similarity selection unit 141 selects the combination with the highest LLM similarity for each entity on the input KG side from all acquired combinations (2,a), (2,b), (3,a), and (3,b). Of the combinations (2,a) and (3,a) for "entity a", the combination with the highest LLM similarity is (2,a). Of the combinations (2,b) and (3,b) for "entity b", the combination with the highest LLM similarity is (2,b). Therefore, the first maximum similarity selection unit 141 selects combinations (2,a) and (2,b). Diagram 331 shows an example of selection by the first maximum similarity selection unit 141.

[0140] The second maximum similarity selection unit 142 selects the combination with the highest LLM similarity for each entity on the integrated KG side from among the combinations (2,a) and (2,b) selected by the first maximum similarity selection unit 141. Of the combinations (2,a) and (2,b) for "Entity 2", the combination with the highest LLM similarity is (2,a). Therefore, the second maximum similarity selection unit 142 selects combination (2,a). Diagram 332 shows an example of selection by the second maximum similarity selection unit 142.

[0141] Figure 22 is a flowchart showing an example of the process of joining identical entities. The process of joining identical entities corresponds to step S13. (S40) The first maximum similarity selection unit 141 obtains combinations (pairs) of entities from the identical entity determination unit 130 whose LLM similarity is determined to be greater than a threshold. The first maximum similarity selection unit 141 performs the process of selecting the maximum similarity to the integrated KG. Specifically, from all combinations obtained from the identical entity determination unit 130, the first maximum similarity selection unit 141 selects the pair with the highest LLM similarity between each entity on the input KG side and the entity on the integrated KG side.

[0142] (S41) The second maximum similarity selection unit 142 performs the process of selecting the maximum similarity from the integrated KG. Specifically, from among the pairs selected in the first maximum similarity selection unit 141, the second maximum similarity selection unit 142 selects the pair with the maximum LLM similarity between each entity on the integrated KG side and the entity on the input KG side.

[0143] (S42) The same entity joining unit 140 combines (merges) the entities of the input KG and the entities of the integrated KG that belong to the pair selected by the second maximum similarity selection unit 142, thereby incorporating the input KG into the integrated KG.

[0144] According to the information processing device 100 of the fourth embodiment, the pairs of entities from the input KG and the integrated KG to be combined are appropriately narrowed down. As a result, the information processing device 100 can appropriately reflect the information held by the input KG in the integrated KG. For example, the information processing device 100 can reduce the possibility that the integrated KG will be lacking in information due to excessive aggregation of entities from each input KG during the KG integration process. In this way, the information processing device 100 can efficiently create an integrated KG with an appropriate amount of information.

[0145] [Fifth Embodiment] Next, a fifth embodiment will be described. The differences from the second to fourth embodiments described above will be explained primarily, and the common aspects will be omitted. The information processing device 100 may use the LLM similarity estimation result from the LLM similarity estimation unit 132 to perform fine tuning of the cosine similarity estimation unit 133.

[0146] Figure 23 shows an example of the functions of the information processing device according to the fifth embodiment. The information processing device 100 includes an entity combination creation unit 120, an identical entity determination unit 130, an identical entity joining unit 140, a determination result storage unit 150, and a fine-tuning execution unit 160. In this fifth embodiment, the information processing device 100 differs from the third and fourth embodiments in that it includes a determination result storage unit 150 and a fine-tuning execution unit 160.

[0147] The determination result storage unit 150 stores the identity determination results for each combination of entities, as determined by the identical entity determination unit 130. For example, the determination result storage unit 150 stores a dataset that includes combinations of entities with matching meanings and combinations of entities with different meanings. This dataset may also include LLM similarity estimated for each combination of entities.

[0148] The fine-tuning execution unit 160 uses the dataset stored in the judgment result storage unit 150 as training data to perform fine-tuning of the text vector transformation model in the cosine similarity estimation unit 133. In fine-tuning, the text vector transformation model is further trained using the training data, and the parameters used in the text vector transformation model are adjusted.

[0149] This allows the information processing device 100 to improve the accuracy of cosine similarity estimation by the cosine similarity estimation unit 133. As a result, the information processing device 100 can improve the accuracy of narrowing down the combination of entities using cosine similarity. Therefore, the information processing device 100 can further improve the efficiency of the integrated KG creation process.

[0150] [Differentiation] Next, modifications of the third to fifth embodiments will be described. The information processing device 100 has a cosine similarity threshold S as follows. cos th and the threshold S for LLM similarity LLM th You may set it to that.

[0151] Figure 24 is a diagram illustrating the setting of thresholds. For example, the identical entity determination unit 130 experimentally prepares datasets of combinations of entities with the same meaning and combinations of entities with different meanings in advance, and calculates cosine similarity and LLM similarity based on these datasets. The identical entity determination unit 130 then identifies a region in the graph 341 plotted using the calculated cosine similarity and LLM similarity that contains only combinations of entities with the same meaning, using a threshold S. cos th and threshold S LLM th This is determined. Here, in Graph 341, the horizontal axis is cosine similarity and the vertical axis is LLM similarity.

[0152] In Graph 341, the threshold S cos thand threshold S LLM th The region of graph 341 is divided into four parts by the lines representing each of the four elements. In this case, the upper right region of the divided region in graph 341 is a region where both the cosine similarity and LLM similarity are relatively large. Therefore, the identical entity determination unit 130 sets a threshold S to make the upper right region of graph 341 a region that contains only combinations of entities with matching meanings. cos th and threshold S LLM th You just need to decide that.

[0153] Furthermore, the identical entity determination unit 130 may, in creating the dataset, pre-prepare only combinations of entities with matching meanings and then use LLM or the like to process those entities to create combinations of entities with different meanings.

[0154] Furthermore, the identical entity determination unit 130 may create a dataset by using an LLM with higher accuracy than the LLM provided by the LLM server 200 to distinguish between combinations of entities with the same meaning and combinations of entities with different meanings. In this case, the identical entity determination unit 130 may recreate the dataset at a predetermined timing and set a threshold S cos th and threshold S LLM th The threshold can be dynamically updated by resetting it. Here, the high-precision LLM used to create the dataset can accurately determine the semantic identity between entities, but the processing time can be very long. In that case, the threshold setting using the high-precision LLM is performed in advance before creating the integrated KG. By setting each threshold in advance in this way, the information processing device 100 can determine combinations of entities with matching meanings with higher accuracy.

[0155] Next, we will explain specific examples of how to use integrated KGs. Figure 25 shows an example of how to use an integrated knowledge graph. For example, the information processing device 100 creates a Key Value Group (KG) that shows the causal relationship of a failure from each document describing the failure event, integrates each KG into a knowledge database, and enables highly accurate failure cause analysis. In one example, the information processing device 100 creates a KG from each of the documents 410 and 420 that describe the failure event. The information processing device 100 integrates the KG corresponding to document 410 and the KG corresponding to document 420 to create an integrated KG 430 that shows the causal relationship of the failure. The document name of document 410 is document E. The document name of document 420 is document F.

[0156] The information processing device 100 then inputs the integrated KG430 into the LLM440, for example, and trains the LLM440 using machine learning based on the integrated KG430. This ensures that the knowledge gained from a combined interpretation of documents 410 and 420 is reflected in the LLM440. The LLM440 may be provided by another information processing device (e.g., an LLM server 200) that can communicate with the information processing device 100. The information processing device 100 may also have its own LLM440.

[0157] Operator 450 can input a natural language question into LLM440 using a terminal device, and obtain an answer to that question from LLM440. For example, operator 450 inputs the question "What causes the router to become overloaded?" into LLM440. Based on the knowledge gained from a combined interpretation of documents 410 and 420, LLM440 then outputs the following answer.

[0158] The following causes are possible: - Cable degradation (see Document E) -XX configuration error (see Document F) The response output by the LLM440 described above is displayed, for example, on the screen of a terminal device used by operator 450.

[0159] In this way, the information processing device 100 can support the operator 450 in fault cause analysis by creating the integrated KG430. Furthermore, by appropriately creating the integrated KG430, the information processing device 100 can contribute to the efficiency of fault cause analysis.

[0160] As described above, the information processing device 100 of the second to fifth embodiments performs the following processing. The entity combination creation unit 120 obtains combinations of a first entity included in the first knowledge graph and a second entity included in the second knowledge graph from the first knowledge graph and the second knowledge graph. Each of the first and second knowledge graphs shows the causal relationships between multiple entities. The identical entity determination unit 130 inputs the first entity and the second entity to a machine learning model capable of outputting the similarity of two entities for input of two entities, and instructs the model to lower the similarity between the first entity and the second entity if the relationship between the first entity and the second entity is a predetermined relationship. The identical entity determination unit 130 obtains the similarity between the first entity and the second entity output by the machine learning model. The identical entity determination unit 130 compares the similarity with a threshold and determines that the first entity and the second entity whose similarity is greater than the threshold are entities with the same meaning. The identical entity merging unit 140 generates a third knowledge graph by merging the first knowledge graph and the second knowledge graph, treating the first entity and the second entity whose similarity is greater than the threshold as identical entities, i.e., entities with the same meaning.

[0161] This allows the information processing device 100 to improve the accuracy of its similarity estimation between two entities. Furthermore, the information processing device 100 can appropriately generate a third knowledge graph by integrating the first and second knowledge graphs. The LLM provided by the LLM server 200 is an example of a machine learning model. The LLM similarity is an example of the similarity output by the machine learning model.

[0162] Here, at least one of the first entity and the second entity is a document. The information processing device 100 is particularly useful when documents are included in the knowledge graph as entities, as it can improve the accuracy of the similarity between the two entities.

[0163] For example, a predetermined relationship includes at least one of a cause-and-effect relationship, a co-occurrence relationship, and a subject-predicate relationship. This makes it difficult for the first entity and the second entity to be aggregated into a single entity when they have a cause-and-effect relationship, a co-occurrence relationship, or a subject-predicate relationship. For this reason, the information processing device 100 can create a third knowledge graph that appropriately reflects the information of the first knowledge graph and the second knowledge graph.

[0164] Furthermore, instructions for the machine learning model may include, for example, instructions for determining similarity by consensus, examples of a given relationship, and examples of similarity responses for two example sentences.

[0165] This allows the information processing device 100 to further improve the accuracy of estimating the similarity between the first entity and the second entity. The aforementioned prompt 131b based on ToT is an example of an instruction indicating that the similarity will be determined by a consensus among multiple hypothetical responding entities (e.g., experts). The prompt 131c based on ICL is an example of an instruction that includes an example of a predetermined relationship. The prompt 131d based on ICL is an example of an instruction that includes an example of a similarity response for two example sentences ("Event 1" and "Event 2"). Instructions for the machine learning model may also include a description of the definition of similarity.

[0166] Furthermore, the identical entity determination unit 130 may select a combination of entities to be used to obtain similarity scores using the machine learning model, based on the cosine similarity between the first entity and the second entity. This allows the information processing device 100 to obtain similarity scores using the machine learning model quickly and with high accuracy.

[0167] Furthermore, in generating the third knowledge graph, the same entity joining unit 140 may acquire multiple combinations of the first entity and the second entity obtained from the first knowledge graph and the second knowledge graph, where the similarity of each of these combinations is greater than a threshold. The same entity joining unit 140 may select the combination with the highest similarity for each first entity from among these multiple combinations. The same entity joining unit 140 may also select the combination with the highest similarity for each second entity from among the combinations selected for each first entity. The same entity joining unit 140 may determine that the first entity and the second entity belonging to the combination selected for each second entity are the same entity.

[0168] This allows the information processing device 100 to efficiently select pairs of entities that are considered identical. Furthermore, the information processing device 100 can appropriately reflect the information from the first and second knowledge graphs into the third knowledge graph.

[0169] Furthermore, the identical entity determination unit 130 may obtain a determination result of semantic identity between the first entity and the second entity belonging to a given combination, based on a comparison between the similarity and a threshold for each of multiple combinations of the first entity and the second entity obtained from the first knowledge graph and the second knowledge graph. The determination result of semantic identity can also be said to be a determination result of whether or not the meanings of the first entity and the second entity are identical. The determination result of semantic identity may include the similarity between the first entity and the second entity output by a machine learning model such as LLM. The fine-tuning execution unit 160 may use the determination result to perform fine-tuning of the language model used to calculate cosine similarity.

[0170] As a result, the information processing device 100 can improve the accuracy of cosine similarity and thus improve the accuracy of narrowing down the combinations of the first and second entities. The text vector transformation model 90 is an example of a language model used to calculate cosine similarity.

[0171] Furthermore, the entity combination creation unit 120 obtains combinations of the third entity included in the third knowledge graph and the fourth entity included in the fourth knowledge graph from the third knowledge graph and the fourth knowledge graph. The identical entity determination unit 130 obtains the similarity between the third entity and the fourth entity using a machine learning model. The identical entity joining unit 140 generates a fifth knowledge graph by joining the third knowledge graph and the fourth knowledge graph, treating the third entity and the fourth entity whose similarity is greater than a threshold as the same entity.

[0172] In this way, the information processing device 100 can sequentially integrate multiple knowledge graphs and appropriately generate an integrated knowledge graph. The information processing in the first embodiment can be achieved by having the processing unit 12 execute a program. The information processing in the second embodiment can be achieved by having the processor 101 execute a program. The program can be recorded on a computer-readable recording medium 113.

[0173] For example, a program can be distributed by distributing a recording medium 113 on which the program is stored. Alternatively, the program may be stored on another computer and distributed via a network. A computer may, for example, store (install) a program stored on the recording medium 113 or a program received from another computer into a storage device such as RAM 102 or HDD 103, and then read and execute the program from that storage device. [Explanation of Symbols]

[0174] 10 Information Processing Devices 11 Storage section 12 Processing Units 20, 21, 50 Knowledge Graph 30 Machine Learning Models 40 Prompts

Claims

1. Computers From a first knowledge graph and a second knowledge graph, each representing a causal relationship between multiple entities, a combination of a first entity included in the first knowledge graph and a second entity included in the second knowledge graph is obtained. A machine learning model capable of outputting the similarity between two entities for the input of two entities is given the first entity and the second entity as inputs, and is instructed to lower the similarity between the first entity and the second entity if the relationship between the first entity and the second entity is a predetermined relationship. The similarity between the first entity and the second entity, output by the machine learning model, is obtained. A third knowledge graph is generated by combining the first and second knowledge graphs as the same entity, where the similarity of the first and second entities is greater than the threshold. Generation method.

2. The aforementioned predetermined relationship includes at least one of the following: a cause-and-effect relationship, a relationship of simultaneous occurrence, and a subject-and-predicate relationship. The generation method according to claim 1.

3. The instructions for the machine learning model include at least one of the following: instructions for determining the similarity by consensus, examples of the predetermined relationship, and examples of responses for the similarity of two example sentences. The generation method according to claim 1.

4. The aforementioned computer, Based on the cosine similarity of the first entity and the second entity, the machine learning model selects the combination for which to obtain the similarity. The generation method according to claim 1.

5. The aforementioned computer, In the generation of the third knowledge graph described above, A plurality of combinations of the first entity and the second entity obtained from the first knowledge graph and the second knowledge graph, wherein the similarity of each of the plurality of combinations is greater than the threshold, From among the plurality of combinations, select the combination with the highest similarity for each of the first entities. From the combinations selected for each of the first entities, select the combination with the highest similarity for each of the second entities. For each of the aforementioned second entities, the aforementioned first entity and the aforementioned second entity belonging to the selected combination are determined to be the same entity. The generation method according to claim 1.

6. The aforementioned computer, Based on a comparison of the similarity and threshold for each of the multiple combinations of the first entity and the second entity obtained from the first and second knowledge graphs, a determination result of the semantic identity of the first entity and the second entity belonging to that combination is obtained. Using the aforementioned determination results, fine-tuning of the language model used to calculate the cosine similarity is performed. The method of production according to claim 4.

7. The aforementioned computer, From the third knowledge graph and the fourth knowledge graph, obtain the combination of the third entity included in the third knowledge graph and the fourth entity included in the fourth knowledge graph. Using the machine learning model, the similarity between the third entity and the fourth entity is obtained. A fifth knowledge graph is generated by combining the third and fourth knowledge graphs as the same entity, where the similarity of the third and fourth entities is greater than the threshold. The generation method according to claim 1.

8. At least one of the first entity and the second entity is a document. The generation method according to claim 1.

9. On the computer, From a first knowledge graph and a second knowledge graph, each representing a causal relationship between multiple entities, a combination of a first entity included in the first knowledge graph and a second entity included in the second knowledge graph is obtained. A machine learning model capable of outputting the similarity between two entities for the input of two entities is given the first entity and the second entity as inputs, and is instructed to lower the similarity between the first entity and the second entity if the relationship between the first entity and the second entity is a predetermined relationship. The similarity between the first entity and the second entity, output by the machine learning model, is obtained. A third knowledge graph is generated by combining the first and second knowledge graphs as the same entity, where the similarity of the first and second entities is greater than the threshold. A generation program that executes a process.

10. A storage unit that stores information from a first knowledge graph and a second knowledge graph, each of which shows the causal relationships between multiple entities, A processing unit that obtains combinations of a first entity included in the first knowledge graph and a second entity included in the second knowledge graph, and instructs a machine learning model capable of outputting the similarity of two entities for input of two entities to input the first entity and the second entity, and to lower the similarity of the first entity and the second entity if the relationship between the first entity and the second entity is a predetermined relationship, obtains the similarity of the first entity and the second entity output by the machine learning model, and generates a third knowledge graph by combining the first knowledge graph and the second knowledge graph as the same entity for the first entity and the second entity whose similarity is greater than a threshold, An information processing device having

Images