Generation method, generation program, and information processing device.

Abstract

It accurately generates knowledge graphs that show cause and effect relationships. [Solution] The information processing device 10 extracts an entity 15 representing the final event from the text 13. Based on the extraction result of entity 15, the information processing device 10 extracts an entity 16 representing the root cause event from the text 13. Based on the extraction results of entities 15 and 16, the information processing device 10 extracts entities 17a and 17b representing intermediate events on the causal relationship path from the root cause event to the final event from the text 13. The information processing device 10 determines the similarity between the knowledge graph 14 showing the causal relationship between entities 15, 16, 17a, and 17b and the text 13. If the similarity exceeds a threshold, the information processing device 10 outputs the knowledge graph 14.

Description

Technical Field

[0001] The present invention relates to a generation method, a generation program, and an information processing apparatus.

Background Art

[0002] A computer may want to estimate the cause corresponding to a newly occurred event by using a knowledge database that records past cases. For example, a computer may want to estimate the cause corresponding to a newly occurred failure by using a knowledge database that records past failure cases of an information processing system. By the computer assisting in the analysis of the failure cause, it is expected that the recovery of the information processing system from the failure will be accelerated.

[0003] There is a technology for assisting in the generation of data that defines recovery procedures for failures of an information processing system. There is also a technology for storing the analysis results of data collected from individual data sources in a database and cross-searching the analysis results of multiple data sources. There is also a technology for generating a knowledge graph from structured data, generating word vectors from unstructured data, and using the knowledge graph and word vectors to output a response sentence for an input sentence. There is also a technology for generating an input to a machine learning model called generative AI (Artificial Intelligence) using a template or another natural language processing model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

[0005] To facilitate the utilization of insights gained from past cases, computers can generate knowledge graphs that show causal relationships between multiple events from text written in natural language describing past cases. However, natural language text is unstructured data, and accurately determining causal relationships between events is not easy. Therefore, in one aspect, the present invention aims to accurately generate knowledge graphs that show causal relationships. [Means for solving the problem]

[0006] In one aspect, a computer-generated method is provided in which the process involves extracting a first entity from the text that represents the final event; extracting a second entity from the text that represents the root cause event corresponding to the final event, based on the extraction results of the first entity; extracting a third entity from the text that represents intermediate events on the causal path from the root cause event to the final event, based on the extraction results of the first and second entities; determining the similarity between the text and a knowledge graph that shows the causal relationships between the first, second, and third entities; and outputting the knowledge graph if the similarity exceeds a threshold. [Effects of the Invention]

[0007] In one respect, it can accurately generate knowledge graphs that show causal relationships. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating the information processing device of the first embodiment. [Figure 2] This figure shows an example of the hardware of the information processing device according to the second embodiment. [Figure 3] This is a diagram showing the first example of a case document. [Figure 4] It is a diagram showing a first example of a causal knowledge graph. [Figure 5] It is a diagram showing a first example of a cause investigation screen. [Figure 6] It is a diagram showing a second example of a case document. [Figure 7] It is a diagram showing a second example of a causal knowledge graph. [Figure 8] It is a diagram showing a second example of a cause investigation screen. [Figure 9] It is a diagram showing an example of a causal knowledge graph inconsistent with a case document. [Figure 10] It is a diagram showing an example of a graph template. [Figure 11] It is a diagram showing an example of an evaluation of extraction result text. [Figure 12] It is a diagram showing a third example of a case document. [Figure 13] It is a diagram showing a third example of a causal knowledge graph. [Figure 14] It is a diagram showing an example of the structure of a large language model. [Figure 15] It is a diagram showing a fourth example of a case document. [Figure 16] It is a diagram showing an example of input / output data of a large language model. [Figure 17] It is a diagram (continuation 1) showing an example of input / output data of a large language model. [Figure 18] It is a diagram (continuation 2) showing an example of input / output data of a large language model. [Figure 19] It is a diagram (continuation 3) showing an example of input / output data of a large language model. [Figure 20] It is a diagram (continuation 4) showing an example of input / output data of a large language model. [Figure 21] It is a block diagram showing an example of the functions of an information processing apparatus. [Figure 22] It is a flowchart showing an example of a graph generation procedure.

Modes for Carrying Out the Invention

[0009] This embodiment will be described below with reference to the drawings. (a) First embodiment Figure 1 is a diagram illustrating an information processing device of a first embodiment. The information processing device 10 of the first embodiment generates a knowledge graph showing the causal relationships between events from text. The information processing device 10 may be, for example, a client device or a server device. The information processing device 10 may also be called, for example, a computer or a generating device.

[0010] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile memory such as RAM (Random Access Memory). Alternatively, the storage unit 11 may be a non-volatile storage such as an HDD (Hard Disk Drive) or SSD (Solid State Drive).

[0011] The processing unit 12 is a processor, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). However, the processing unit 12 may also include electronic circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processor executes a program stored in memory, such as RAM. The processor is sometimes called a processor circuit. A collection of processors is sometimes called a multiprocessor or simply a "processor." Different processing steps among multiple processing steps may be executed by different processors.

[0012] The memory unit 11 stores text 13. Text 13 is a case text that describes past cases in natural language and includes descriptions of multiple events that have a causal relationship. For example, text 13 is a failure case text that describes past failure cases that occurred in an information processing system in natural language and includes descriptions of multiple events related to the cause and effect of the failure. The information processing system may be a network system that provides communication services.

[0013] Multiple events include the final event, the root cause event corresponding to the final event, and intermediate events that occur in the process from the root cause event to the final event. The chain of causal relationships from the root cause event to the final event is not necessarily linear, but may have branches. There may not be just one root cause event or one final event. Due to the nature of natural language, these events are not described according to a strict format.

[0014] The processing unit 12 generates a knowledge graph 14 based on the text 13. The knowledge graph 14 shows the causal relationships between multiple entities that represent different events. The knowledge graph 14 may be represented using nodes that represent entities and directed edges that show the causal relationships between entities. Alternatively, the knowledge graph 14 may be represented as a list of causal relationships in which one event causes another. Furthermore, the knowledge graph 14 may be described using a descriptive language such as JSON (JavaScript Object Notation) or RDF (Resource Description Framework).

[0015] Here, due to the limitations of natural language processing accuracy, attempting to extract multiple events simultaneously from text 13 may result in lower accuracy for the knowledge graph 14. Therefore, the processing unit 12 extracts multiple events from text 13 using a specific procedure described below. The processing unit 12 also outputs a knowledge graph 14 that meets certain evaluation criteria.

[0016] First, the processing unit 12 extracts a first entity representing the final event from the text 13. In the example in Figure 1, entity 15 represents the final event. At this time, the processing unit 12 may use a machine learning model trained by machine learning, or it may use a natural language processing model such as a so-called Large Language Model (LLM). For example, the processing unit 12 specifies the text 13 to the natural language processing model, causing the natural language processing model to extract entity 15 from the text 13.

[0017] Next, based on the extraction result of the first entity, the processing unit 12 extracts a second entity from the text 13 that represents the root cause event corresponding to the final event. In the example in Figure 1, entity 16 represents the root cause event. At this time, the processing unit 12 may use a natural language processing model. For example, the processing unit 12 specifies the text 13 and entity 15 to the natural language processing model, causing the natural language processing model to extract entity 16 from the text 13.

[0018] Next, based on the extraction results of the first and second entities, the processing unit 12 extracts a third entity representing an intermediate event from the text 13. An intermediate event is an event that lies on the causal path from the root cause event to the final event. In the example in Figure 1, entities 17a and 17b represent intermediate events. At this time, the processing unit 12 may use a natural language processing model. For example, the processing unit 12 specifies the text 13 and entities 15 and 16 to the natural language processing model, causing the model to extract entities 17a and 17b from the text 13.

[0019] Next, the processing unit 12 generates a knowledge graph 14 that shows the causal relationships between the first entity, the second entity, and the third entity. For example, the knowledge graph 14 shows that entity 16 causes entities 17a and 17b, and entities 17a and 17b cause entity 15.

[0020] The processing unit 12 determines the similarity between the knowledge graph 14 and the text 13. Similarity is an evaluation value determined, for example, from the perspective of whether the causal relationships shown in the knowledge graph 14 are consistent with the content of the text 13. A higher similarity is preferable. If one of the causal relationships included in the knowledge graph 14 cannot be said to be described in the text 13, the similarity may be low. Also, if a causal relationship not covered by the knowledge graph 14 can be derived from the text 13, the similarity may be low.

[0021] In this case, the processing unit 12 may use a natural language processing model. For example, the processing unit 12 may specify the text 13 and the knowledge graph 14 to the natural language processing model to cause the model to determine the similarity. The processing unit 12 may also cause the natural language processing model to generate text explaining the reason for the similarity, along with the similarity.

[0022] The processing unit 12 outputs the knowledge graph 14 when the similarity exceeds a threshold. The processing unit 12 may save the knowledge graph 14 to non-volatile storage, display it on a display device, or transmit it to another information processing device. The information processing device 10 or other information processing devices may use the knowledge graph 14 to perform cause analysis, such as failure cause analysis. For example, the information processing device 10 accepts the specification of a new event and selects entities representing the specified event from the knowledge graph 14. The information processing device 10 searches for the root cause entity by tracing the causal relationship backward from the selected entity.

[0023] If the similarity does not exceed the threshold, the processing unit 12 may regenerate the knowledge graph 14 from the text 13, and may repeat the regeneration of the knowledge graph 14 until the similarity exceeds the threshold. The processing unit 12 may re-extract the first entity, the second entity, and the third entity so that the extraction results of the first entity, the second entity, and the third entity change. In this case, the processing unit 12 may change the random numbers used for natural language processing. If the natural language processing model uses random numbers, the processing unit 12 may generate a different knowledge graph 14 from the previous one by calling the natural language processing model again according to the above procedure. The processing unit 12 may also have the natural language processing model modify the knowledge graph 14 by specifying the text 13 and the reason for the similarity to the natural language processing model.

[0024] As described above, the information processing device 10 of the first embodiment extracts an entity 15 representing the final event from the text 13. Based on the extraction result of entity 15, the information processing device 10 extracts an entity 16 representing the root cause event corresponding to the final event from the text 13. Based on the extraction results of entities 15 and 16, the information processing device 10 extracts entities 17a and 17b representing intermediate events on the causal relationship path from the root cause event to the final event from the text 13.

[0025] The information processing device 10 determines the similarity between the knowledge graph 14, which shows the causal relationships between entities 15, 16, 17a, and 17b, and the text 13. If the similarity exceeds a threshold, the information processing device 10 outputs the knowledge graph 14.

[0026] As a result, the information processing device 10 can generate a knowledge graph 14 that is useful for cause analysis, such as fault cause analysis. Furthermore, when various events are extracted simultaneously from the text 13, insufficient recognition of intermediate events may result in the generation of a knowledge graph that skips intermediate events and directly connects the root cause event to the final event. In addition, insufficient recognition of the causal chain may result in the generation of multiple fragmented and disconnected knowledge graphs. By executing the extraction procedure of the first embodiment, insufficient recognition of intermediate events and insufficient recognition of the causal chain are suppressed, and the accuracy of the knowledge graph 14 is improved.

[0027] (b) Second embodiment Figure 2 shows an example of the hardware of the information processing device according to the second embodiment. The information processing device 100 of the second embodiment generates a causal knowledge graph showing the causal relationships between events from case documents in which cases are described in natural language. The information processing device 100 also uses the causal knowledge graph to estimate the root cause corresponding to a new event. However, the generation of the causal knowledge graph and the estimation of the root cause may be performed by different information processing devices. The information processing device 100 corresponds to the information processing device 10 of the first embodiment.

[0028] The information processing device 100 includes a CPU 101, RAM 102, HDD 103, GPU 104, input interface 105, media reader 106, and communication interface 107. The CPU 101 corresponds to the processing unit 12 of the first embodiment. The RAM 102 or HDD 103 corresponds to the storage unit 11 of the first embodiment.

[0029] The CPU 101 is a processor that executes program instructions. The CPU 101 loads the program and data from the HDD 103 into the RAM 102 and executes the program. The information processing device 100 may have multiple processors.

[0030] RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by CPU 101 and data used for calculations by CPU 101. The information processing device 100 may have a type of volatile memory other than RAM.

[0031] The HDD 103 is a non-volatile storage device that stores software programs such as operating systems, middleware, and application software, as well as data. The information processing device 100 may have other types of non-volatile storage, such as an SSD or flash memory.

[0032] The GPU 104 works in conjunction with the CPU 101 to perform image processing and outputs the image to the display device 111 connected to the information processing device 100. The display device 111 is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, or a projector.

[0033] Furthermore, the GPU 104 may be used as a GPGPU (General Purpose Computing on Graphics Processing Unit). The GPU 104 can execute programs in response to instructions from the CPU 101. The information processing device 100 may have volatile semiconductor memory other than RAM 102 as GPU memory.

[0034] The input interface 105 receives input signals from an input device 112 connected to the information processing device 100. The input device 112 is, for example, a mouse, a touch panel, or a keyboard. Multiple input devices may be connected to the information processing device 100.

[0035] The media reader 106 is a reading device that reads programs and data recorded on the recording medium 113. The recording medium 113 is, for example, a magnetic disk, an optical disk, or semiconductor memory. Magnetic disks include flexible disks (FD) and HDDs. Optical disks include CDs (Compact Discs) and DVDs (Digital Versatile Discs). The media reader 106 copies the programs and data read from the recording medium 113 to other recording media such as RAM 102 or HDD 103. The read programs may be executed by the CPU 101.

[0036] The recording medium 113 may be a portable recording medium. The recording medium 113 may be used for distributing programs and data. The recording medium 113 and the HDD 103 may also be referred to as computer-readable recording media.

[0037] The communication interface 107 communicates with other information processing devices via the network 114. The communication interface 107 may be a wired communication interface connected to a wired communication device such as a switch or router, or a wireless communication interface connected to a wireless communication device such as a base station or access point.

[0038] In the second embodiment, the example mainly assumes a failure in a wireless communication network. A wireless communication network may include various types of components, and these components may be from different vendors. For example, multiple antenna base stations (RUs: Radio Units) may be connected to a base station control unit (DU: Distributed Unit), and an RU from one vendor may be connected to a DU from a different vendor. In this case, a failure may occur in the connection between the RU and the DU.

[0039] In wireless communication networks, the increasing complexity of the configuration can lead to a diversification of failure causes. For example, a failure in RU / DU connectivity can have various causes, such as hardware failure, hardware incompatibility, software incompatibility, or incorrect identifier settings. Therefore, the information processing device 100 supports failure cause analysis.

[0040] Figure 3 shows a first example of a case document. The information processing device 100 has a case database 130. The case database 130 stores multiple case documents. Each case document contains one or more case texts describing a failure case in natural language. If a single case document contains case texts for two or more failure cases, those case texts are separated.

[0041] Case study texts are created by engineers when a failure occurs. These texts include descriptions of the malfunctions caused by the failure. They also include descriptions of the root cause of the failure as determined through the recovery process. Case study texts may include descriptions of phenomena observed regarding components of the wireless communication network during the recovery process. Furthermore, case study texts may include descriptions of the root cause analysis and recovery work performed by the engineers.

[0042] Furthermore, the case text may include log strings extracted from log files output by components of the wireless communication network. These log strings are sections of the log file that engineers have determined to be relevant to the fault and useful for understanding the fault. Due to the dependencies between multiple components included in a wireless communication network, various events may occur in a chain reaction. Therefore, the case text may explicitly or implicitly explain the chain of causal relationships between multiple events.

[0043] For example, case database 130 stores case documents 131 and 132. Case document 131 describes a malfunction in which router A becomes overloaded. Case document 131 also describes a phenomenon in which retransmissions frequently occurred between router A and device B. Case document 131 also describes the root cause as cable degradation.

[0044] Case study document 132 describes a malfunction in device B where data transmission is impossible. Case study document 132 also describes a phenomenon where retransmissions frequently occur between router A and device B. Furthermore, case study document 132 describes the root cause as a configuration error in item X. The information processing device 100 generates a causal knowledge graph from the case study database 130 in order to utilize the insights gained from these multiple case studies in a cross-sectional manner.

[0045] Figure 4 shows the first example of a causal knowledge graph. Causal knowledge graph 141 is generated from the above case documents 131 and 132. Causal knowledge graph 141 is a directed graph that includes nodes representing events and directed edges representing the relationships between events. Causal knowledge graph 141 includes nodes 141a, 141b, 141c, 141d, and 141e.

[0046] Node 141a indicates a cable degradation issue. Node 141b indicates a configuration error for item X. Node 141c indicates frequent retransmissions between router A and device B. Node 141d indicates router A is overloaded. Node 141e indicates that device B is unable to transmit data. Nodes 141a, 141c, and 141d have the document name of case document 131 appended to indicate that the issues are described in case document 131. Similarly, nodes 141b, 141c, and 141e have the document name of case document 132 appended.

[0047] The causal knowledge graph 141 shows a causal relationship in which the event at node 141a causes the event at node 141c. Furthermore, the causal knowledge graph 141 shows a causal relationship in which the event at node 141b causes the event at node 141c. Furthermore, the causal knowledge graph 141 shows a causal relationship in which the event at node 141c causes the event at node 141d. Furthermore, the causal knowledge graph 141 shows a causal relationship in which the event at node 141c causes the event at node 141e.

[0048] By generating a causal knowledge graph 141 from multiple failure cases, the causal knowledge graph 141 may reveal potential causal relationships that are not apparent from individual failure cases alone. For example, in the failure case described in case document 131, cable degradation caused an overload on router A. Also, in the failure case described in case document 132, a misconfiguration of item X caused device B to be unable to transmit data.

[0049] However, both cable degradation and misconfiguration of item X cause the same intermediate event. Therefore, potentially, cable degradation could cause device B to be unable to transmit data, and misconfiguration of item X could cause router A to be overloaded. Thus, causal knowledge graph 141 shows cross-sectional insights obtained from multiple failure cases.

[0050] Figure 5 shows a first example of the cause investigation screen. The information processing device 100 uses a causal knowledge graph 141 to support the engineer in analyzing the cause of the failure. The information processing device 100 displays the cause investigation screen 150 on the display device 111. The engineer enters a message 151 on the cause investigation screen 150. Message 151 is a query text inquiring about the cause of the router becoming overloaded.

[0051] The information processing device 100 identifies node 141d from the causal knowledge graph 141, which represents the event described in message 151. The information processing device 100 searches for the uppermost node that can not be reached by tracing the edges in reverse from node 141d. The uppermost node represents the event corresponding to the root cause. Here, nodes 141a and 141b are detected from the causal knowledge graph 141.

[0052] The information processing device 100 generates message 152 and displays it on the cause investigation screen 150. Message 152 is a response text indicating that cable degradation indicated by node 141a and misconfiguration of item X indicated by node 141b are candidate causes of the failure. Message 152 includes the document names attached to nodes 141a and 141b to show the basis for the cause estimation. Message 152 presents the document name of case document 131 regarding cable degradation and the document name of case document 132 regarding misconfiguration of item X.

[0053] If case documents 131 and 132 are referred to individually, the information processing device 100 will only present cable degradation, as described in case document 131, as the cause of router overload. In contrast, by using the causal knowledge graph 141, the information processing device 100 can present a potential cause of failure, namely a misconfiguration of item X. Next, another example of a causal knowledge graph will be explained.

[0054] Figure 6 shows a second example of a case document. Case document 133 describes a failure in a wireless communication network. Case document 133 includes a description of the malfunction, the phenomena observed in the components, the work performed by the engineer, and the method used to resolve the failure.

[0055] Figure 7 shows a second example of a causal knowledge graph. Causal knowledge graph 142 is an ideal causal knowledge graph that is expected to be generated from the above case document 133. Causal knowledge graph 142 includes nodes 142a, 142b, 142c, 142d, 142e, and 142f.

[0056] An event at node 142a triggers an event at node 142c. An event at node 142b triggers an event at node 142c. An event at node 142c can arise from an event at either node 142a or 142b. To represent this logical relationship, the causal knowledge graph 142 has labels indicating the logical OR attached to the two edges from nodes 142a and 142b to node 142c.

[0057] An event at node 142c triggers an event at node 142d. An event at node 142d triggers an event at node 142e. Also, an event at node 142d triggers an event at node 142f. An event at node 142d can simultaneously trigger events at both nodes 142e and 142f. To represent this logical relationship, labels indicating a logical AND are attached to the two edges from node 142d to nodes 142e and 142f.

[0058] Figure 8 shows a second example of the cause investigation screen. The engineer enters message 153 on the cause investigation screen 150. Message 153 is a query text inquiring about the cause of the PTP unlock. If a causal knowledge graph 142 has been generated, the information processing device 100 searches the causal knowledge graph 142 for node 142e that represents the event described in message 153.

[0059] The information processing device 100 searches for nodes 142a and 142b that indicate the root cause by tracing the edge in reverse from node 142e. The information processing device 100 generates message 154 and displays it on the cause investigation screen 150. Message 154 is the response text indicating that the events at nodes 142a and 142b are candidate causes of the failure.

[0060] As will be described later, the information processing device 100 generates causal knowledge graphs such as causal knowledge graphs 141 and 142 using a large-scale language model, which is a type of machine learning model. The large-scale language model is a generative natural language processing model that receives prompt text and outputs response text corresponding to the prompt text.

[0061] However, case texts describe various events, and these multiple events are connected by complex causal relationships. The number of events and the branching structure of the causal relationship paths differ depending on the case text. Therefore, if a large-scale language model extracts all events at once, it may generate a causal knowledge graph with low accuracy. For example, if there are omissions in the extraction of causal relationships, a causal knowledge graph containing multiple disconnected subgraphs may be generated. In addition, a causal knowledge graph may be generated that skips intermediate events and directly connects the root cause to the final result.

[0062] Figure 9 shows an example of a causal knowledge graph that is inconsistent with the case document. Causal knowledge graph 143 is an example of another causal knowledge graph that can be generated from the aforementioned case document 133. Causal knowledge graph 143 includes nodes 143a, 143b, 143c, 143d, 143e, and 143f. The event at node 143a causes the event at node 143b. The event at node 143b causes the event at node 143c.

[0063] Furthermore, an event at node 143d triggers an event at node 143e, and an event at node 143e triggers an event at node 143f. Therefore, nodes 143a, 143b, and 143c form one subgraph, and nodes 143d, 143e, and 143f form another subgraph. These two subgraphs are disconnected and separated. An inaccurate causal knowledge graph reduces the accuracy of the failure cause analysis. Therefore, the information processing device 100 generates a highly accurate causal knowledge graph by following the procedure described below.

[0064] Figure 10 shows an example of a graph template. Graph template 144 is a type of causal knowledge graph that defines the types of events to be extracted from the case text and the order in which they are extracted. Graph template 144 includes nodes 144a, 144b, 144c, 144d, 144e, and 144f.

[0065] Node 144a represents the final event. The final event corresponds to the final defect caused by the failure. Node 144b represents the causal event. The causal event corresponds to the root cause of the failure. Nodes 144c and 144d represent intermediate events. Intermediate events correspond to intermediate events that occurred at intermediate stages in the chain of causal relationships from the causal event to the final event. The number of intermediate events and the branching structure of the causal relationships related to the intermediate events are variable.

[0066] Node 144e indicates an "other event." An "other event" is an event that deviates from the causal path from the causal event to the final event. An "other event" is an event that was recognized during the fault recovery process but was not directly related to the malfunction. Such events may still be important in other fault cases. Node 144f indicates a "log event." A "log event" indicates that a specific log string is included in the log file. A "log event" supports the occurrence of one of the following events: causal event, intermediate event, final event, or other event.

[0067] As an example of edges connecting these multiple nodes, graph template 144 includes an edge from node 144b to nodes 144c and 144e. These two edges indicate a causal relationship. Node 144c can reach node 144d via one or more edges. Graph template 144 also includes an edge from node 144f to node 144d. This edge indicates an indicative relationship. The log event at node 144f suggests the occurrence of an intermediate event at node 144d. Graph template 144 also includes an edge from node 144d to node 144e. This edge indicates a causal relationship.

[0068] The extraction order for the final event is first. The extraction order for the causal event is second. The extraction order for the intermediate event is third. The extraction order for other events is fourth. The extraction order for log events is fifth. By determining the causal and final events before extracting the intermediate events, the risk of skipping intermediate events and directly linking the causal and final events, as well as the risk of splitting the causal chain, is reduced.

[0069] First, the information processing device 100 extracts multiple case texts from the case database 130, each representing a single failure case. If the case documents are written in a specific format, the information processing device 100 may determine the range of text representing a single failure case according to that format. Alternatively, the information processing device 100 may have a large-scale language model determine the boundaries of the case texts.

[0070] Next, the information processing device 100 generates a causal knowledge graph for each case text, corresponding to one failure case. At this time, the information processing device 100 uses a large-scale language model to extract the causal event, intermediate event, final event, other events, and log events from the case text in the order indicated by the graph template 144.

[0071] First, the information processing device 100 extracts the final event from the case text. The information processing device 100 generates prompt text that includes the case text and an instruction sentence that instructs it to extract the final event. The information processing device 100 inputs the prompt text into a large-scale language model and obtains response text indicating the final event from the large-scale language model.

[0072] Secondly, the information processing device 100 extracts causal events from the case text. The information processing device 100 specifies the case text and the previously extracted final event, and generates prompt text that includes an instruction to extract causal events related to the final event. The information processing device 100 inputs the prompt text into a large-scale language model and obtains response text indicating the causal events from the large-scale language model.

[0073] Thirdly, the information processing device 100 extracts intermediate events from the case text. The information processing device 100 specifies the case text and the previously extracted final and causal events, and generates prompt text that includes an instruction to extract intermediate events between the causal and final events. The information processing device 100 inputs the prompt text into a large-scale language model and obtains response text from the large-scale language model that shows the causal relationships regarding the intermediate events.

[0074] Fourthly, the information processing device 100 extracts other events from the case text. The information processing device 100 specifies the case text and the previously extracted final event, causal event, and intermediate event, and generates prompt text that includes an instruction to extract other events. The information processing device 100 inputs the prompt text into a large-scale language model and obtains response text from the large-scale language model that shows the causal relationships regarding the other events.

[0075] Fifthly, the information processing device 100 extracts log events from the case text. The information processing device 100 specifies the case text and the previously extracted final event, causal event, intermediate event, and other events, and generates prompt text that includes an instruction to extract log events. The information processing device 100 inputs the prompt text into a large-scale language model and obtains response text from the large-scale language model that indicates the suggestive relationships of the log events.

[0076] Next, the information processing device 100 uses a large-scale language model to calculate the similarity between the original case text and the list showing the extracted events. The similarity is evaluated in terms of the accuracy of the causal relationships between the events. A higher similarity is preferable.

[0077] Figure 11 shows an example of the evaluation of the extracted text. The information processing device 100 calculates the similarity between the case document 133 and the extracted text 134. The extracted text 134 contains a list of causal relationships and suggestive relationships between events. In the example in Figure 11, the extracted text 134 shows the causal relationship from one causal event to one intermediate event, and the causal relationship from that intermediate event to two final events.

[0078] The information processing device 100 generates prompt text that includes the case text of the case document 133, the list shown in the extracted result text 134, and an instruction sentence that instructs the calculation of similarity. The information processing device 100 inputs the prompt text into a large-scale language model and obtains response text from the large-scale language model that shows the calculated similarity.

[0079] Similarity is expressed as an integer between 0% and 100%, for example. If the causal relationships shown in the extracted text 134 are described completely and accurately in the case document 133, the similarity may be 100%. If the extracted text 134 contains causal relationships that are not described in the case document 133, the similarity may be low. Also, if there is a high probability that there are causal relationships remaining in the case document 133 that were not extracted in the extracted text 134, the similarity may be low.

[0080] The information processing device 100 determines whether the similarity exceeds a threshold. The threshold is predetermined, for example, 90%. If the similarity does not exceed the threshold, the information processing device 100 re-extracts the causal event, intermediate event, final event, other event, and log event from the event text. As will be described later, large-scale language models may use random numbers to ensure diversity in response text. Therefore, different response texts may be generated from the same prompt text. The information processing device 100 repeats the extraction of causal events, intermediate events, final events, other events, and log event until the similarity exceeds the threshold.

[0081] When the similarity exceeds a threshold, the information processing device 100 converts the list of causal relationships and suggestive relationships into a data format for a causal knowledge graph. For example, the information processing device 100 outputs the causal knowledge graph as JSON data. The information processing device 100 may also use a large-scale language model to convert the data format.

[0082] Furthermore, when calculating the similarity score, the information processing device 100 may have the large-scale language model generate the reason for the calculated similarity score along with the similarity score itself. Before comparing the similarity score with the threshold, the information processing device 100 may feed back the reason to the large-scale language model to refine the list of extraction results in the large-scale language model.

[0083] For example, the information processing device 100 generates prompt text that includes instructions to improve the list, specifying the example text, the previously extracted list, and the reason for the similarity. The information processing device 100 inputs the prompt text into a large-scale language model and obtains response text from the large-scale language model showing the updated list. The information processing device 100 calculates the similarity of the updated list and compares it to a threshold.

[0084] The information processing device 100 integrates multiple causal knowledge graphs generated from multiple case texts. At this time, the information processing device 100 detects nodes representing the same event among the multiple causal knowledge graphs. The information processing device 100 generates an integrated causal knowledge graph by replacing the multiple nodes representing the same event with a single node.

[0085] The generated causal knowledge graph can be used, for example, as follows: The information processing device 100 generates prompt text that includes the input query text and an instruction that instructs the extraction of the final event from the query text. The information processing device 100 inputs the prompt text into a large-scale language model and obtains response text indicating the final event. Outside the large-scale language model, the information processing device 100 searches for a node indicating the final event in the causal knowledge graph and searches for a node indicating the causal event in the causal knowledge graph by following the edges.

[0086] The information processing device 100 may refer to log events to narrow down the cause of the failure, or it may prioritize the candidate causes of the failure. For example, the input query text may contain log strings extracted from a log file. When tracing the causal relationship path from the final event, the information processing device 100 may narrow down the path to the path associated with the log event corresponding to the input log string. The information processing device 100 may also give a higher priority to the cause event reached by the path associated with the log event corresponding to the input log string than to the cause event reached by other paths. Next, another example of a causal knowledge graph will be described as a use case for failure cause analysis.

[0087] Figure 12 shows the third example of a case document. Case document 135 describes a failure case in which the application terminates immediately after startup. Case document 135 describes the final event, intermediate events, and causal events. Although omitted in Figure 12, case document 135 also includes log events.

[0088] Figure 13 shows a third example of a causal knowledge graph. Causal knowledge graph 145 is generated from the above case document 135. Causal knowledge graph 145 includes nodes 145a, 145b, 145c, 145d, and 145e.

[0089] Node 145a indicates the root cause, which is the download of the UI application from the internet. Node 145b indicates the log event, which is that version incompatibility is recorded in the log. Node 145c indicates the intermediate event, which is version incompatibility of the UI application. Node 145d indicates the intermediate event, which is version incompatibility of the analysis module. Node 145e indicates the final event, which is the termination of the application.

[0090] The causal event at node 145a triggers an intermediate event at node 145c. The intermediate event at node 145c triggers an intermediate event at node 145d. The log event at node 145b suggests intermediate events at nodes 145c and 145d. The intermediate event at node 145d triggers the final event at node 145e.

[0091] First, the information processing device 100 extracts the final event of node 145e from case document 135. Next, the information processing device 100 extracts the causal event of node 145a from case document 135. Next, the information processing device 100 extracts the intermediate events of nodes 145c and 145d from case document 135. Finally, the information processing device 100 extracts the log event of node 145b from case document 135.

[0092] Next, the structure and input / output data of the large-scale language model will be described. The large-scale language model may be a neural network or it may be implemented using a transformer with an attention mechanism. Transformers are also described in the following non-patent document: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, and Lukasz Kaiser, "Attention Is All You Need", Proc. of the 31st International Conference on Neural Information Processing Systems (NIPS 2017), pages 6000-6010, December 2017.

[0093] Figure 14 shows an example of the structure of a large-scale language model. The large-scale language model 160 is an encoder-decoder type neural network. The large-scale language model 160 has embedding layers 161, 162, position coding layers 163, 164, encoder 165, decoder 166, linear layer 167, and softmax layer 168.

[0094] The embedding layer 161 converts each of the multiple words contained in the input text into a word vector called an embedding representation or distributed representation. A word vector is a numerical vector with a fixed number of dimensions, such as 512 or 1024 dimensions. Similar word vectors are assigned to words used in similar contexts. The correspondence between words and word vectors is determined by the neural network. The embedding layer 161 may be trained together with the other layers of the large-scale language model 160, or it may be pre-trained.

[0095] The embedding layer 162 converts each of the one or more words that have been determined so far from the words that should be included in the output text into a word vector. In the large-scale language model 160, the words that should be included in the output text are determined one by one from the beginning. The same correspondence between words and word vectors as in the embedding layer 161 is used.

[0096] The position encoding layer 163 adds a position vector corresponding to the word's position to the word vector output by the embedding layer 161. This addition of position vectors is sometimes called position encoding. The position vector is a numerical vector with the same number of dimensions as the word vector. For each of the multiple words in the input text, the position encoding layer 163 calculates the numerical values ​​for each dimension of the position vector using a sine or cosine function, based on a non-negative integer indicating the word's position from the beginning.

[0097] The position coding layer 164 adds a position vector corresponding to the word's position to the word vector output by the embedding layer 162. The method for calculating the position vector is the same as that of the position coding layer 163. For each of the one or more words in the output text, the position coding layer 164 calculates the numerical values ​​for each dimension included in the position vector using a sine or cosine function, based on a non-negative integer indicating the word's position from the beginning.

[0098] Encoder 165 transforms multiple vectors corresponding to multiple words. Encoder 165 includes, in order, a self-attention layer 165a, a normalization layer 165b, a feedforward layer 165c, and a normalization layer 165d. A large-scale language model 160 may have multiple encoders 165 stacked in series. In that case, the first encoder receives vectors from the position coding layer 163, and the last encoder outputs vectors to the decoder 166.

[0099] The self-attention layer 165a transforms vectors using an attention mechanism. The self-attention layer 165a has a query matrix, a key matrix, and a value matrix as trained parameter values. The self-attention layer 165a selects one word of interest from among multiple words contained in the input text.

[0100] The self-attention layer 165a transforms the vector of the word of interest using a query matrix to calculate a vector called the query. The self-attention layer 165a also transforms the vectors of multiple words using a key matrix to calculate a vector called the key. The self-attention layer 165a calculates the dot product of the query and the key as the attention score for each word. The attention score indicates the degree of relevance between the word of interest and each other word.

[0101] The self-attention layer 165a transforms the vectors of multiple words using a value matrix to calculate a vector called a value. The self-attention layer 165a uses the attention score as a weight to calculate a weighted sum of values ​​among multiple words, and outputs the calculated weighted sum as the transformed vector for the word of interest. The self-attention layer 165a repeats the above process while changing the word of interest.

[0102] The normalization layer 165b normalizes the vector output by the self-attention layer 165a so that the numerical values ​​for each dimension follow a constant distribution. The feedforward layer 165c is a forward neural network. The feedforward layer 165c transforms vectors of multiple words individually using trained parameter values. The normalization layer 165d normalizes the vector output by the feedforward layer 165c in the same way as the normalization layer 165b.

[0103] Decoder 166 converts the vectors of one or more words that have been determined so far from the words that should be included in the output text. Decoder 166 includes, in order, a self-attention layer 166a, a normalization layer 166b, an attention layer 166c, a normalization layer 166d, a feedforward layer 166e, and a normalization layer 166f. The large-scale language model 160 may have multiple decoders 166 stacked in series. In that case, the first decoder receives a vector from the position coding layer 164, and the last decoder outputs a vector to the linear layer 167.

[0104] The self-attention layer 166a transforms the vector using an attention mechanism similar to that of the self-attention layer 165a. The query, key, and value are calculated from the word vectors of the output text. The normalization layer 166b normalizes the vector output by the self-attention layer 166a in the same way as the normalization layer 165b.

[0105] The attention layer 166c uses an attention mechanism to transform the word vectors in the output text. However, the attention layer 166c calculates the query from the word vectors in the output text and calculates the key and value from the word vectors in the input text. This allows the degree of relevance between the words in the output text and the words in the input text to be determined.

[0106] The attention layer 166c selects one word of interest from one or more words contained in the output text. The attention layer 166c calculates a query by transforming the vector of the word of interest using a query matrix. The attention layer 166c also receives vectors of multiple words contained in the input text from the encoder 165. The attention layer 166c calculates a key by transforming the vector of each word using a key matrix, and calculates a value by transforming the vector of each word using a value matrix.

[0107] The attention layer 166c calculates the dot product of the query and the key as an attention score for each word in the input text. The attention score indicates the degree of relevance between each word in the input text and the word in the output text of interest. The attention layer 166c uses the attention scores as weights to calculate a weighted sum of values ​​among multiple words in the input text. The attention layer 166c outputs the calculated weighted sum as a transformed vector for the word in the output text of interest.

[0108] The normalization layer 166d normalizes the vector output by the attention layer 166c in the same way as the normalization layer 165b. The feedforward layer 166e transforms the word vectors of the output text individually using the trained parameter values. The normalization layer 166f normalizes the vector output by the feedforward layer 166e in the same way as the normalization layer 165b.

[0109] The linear layer 167 uses the numerical values ​​contained in the vector output by the decoder 166 to calculate scores for various words listed in the dictionary. The words listed in the dictionary are those to which word vectors are assigned by the embedding layers 161 and 162. For example, the word vectors from embedding layers 161 and 162 are referenced in the calculation of the scores.

[0110] The softmax layer 168 converts the scores of various words into probabilities between 0 and 1. The large-scale language model 160 selects one word based on the probability and adds the selected word to the end of the output text. The large-scale language model 160 generates the output text by repeating the process of the decoder 166 described above.

[0111] The large-scale language model 160 uses random numbers when selecting words for the output text to ensure diversity in the output text. The large-scale language model 160 randomly selects one word from several words in order of probability, starting with the words with the highest probability. Therefore, the word with the highest probability is not necessarily selected. The extent to which lower-ranking words are included as selection candidates is adjusted by the hyperparameters of the large-scale language model 160.

[0112] Figure 15 shows the fourth example of a case document. Case document 136 describes a failure case in which S-plane packets are not accepted by the RU. Case document 136 includes a description of the malfunction, the observed phenomena regarding the components of the wireless communication network, the work performed by the engineer, and the method used to resolve the failure. Case document 136 also includes log strings extracted by the engineer from the log file. In Figure 15, the log strings included in case document 136 are underlined.

[0113] Figure 16 shows an example of input and output data for a large-scale language model. First, the information processing device 100 generates prompt text 171a and inputs it to the large-scale language model 160. Prompt text 171a includes the case text described in case document 136. Prompt text 171a also includes an instruction to extract the final event from this case text. Furthermore, prompt text 171a includes a note indicating that multiple events described in the case text may be connected by a chain of causal relationships, and that only the final event should be extracted, and other events should not be extracted.

[0114] Furthermore, prompt text 171a includes a cautionary note instructing against extracting intermediate tasks performed by engineers for fault analysis as events. In Figure 16, this cautionary note included in prompt text 171a is underlined. Additionally, prompt text 171a includes a cautionary note instructing against splitting time-series events described using logical AND or OR into multiple events. In Figure 16, this cautionary note included in prompt text 171a is underlined.

[0115] A time-series event is a sequence of events in which multiple phenomena occur consecutively. While time-series events may include logical AND or OR operations, they do not necessarily indicate a causal relationship between the multiple phenomena connected by these operations. For example, in the time-series event "synchronization is lost, and then synchronization is restored," there is no causal relationship between "synchronization is lost" and "synchronization is restored." Therefore, prompt text 171a instructs the large-scale language model 160 to extract the time-series event as a single event without splitting it.

[0116] The large-scale language model 160 generates response text 171b from prompt text 171a. Response text 171b indicates two final events. Upon receiving response text 171b, the information processing device 100 generates prompt text 172a and inputs it to the large-scale language model 160. Prompt text 172a includes an instruction that instructs the device to extract a causal event based on the previously input case text and the previously extracted final event.

[0117] Furthermore, prompt text 172a instructs the user to refer to the countermeasures section as a cautionary note to determine the cause of the event. In Figure 16, this cautionary note included in prompt text 172a is underlined. For example, the statement "The value exceeded the threshold" suggests two possible root causes: "The value exceeded the threshold" and "An error in setting the threshold." If the countermeasures section describes the action of "Increasing the threshold," it is preferable to determine that "An error in setting the threshold" is the root cause. However, the action described in the countermeasures section itself is not extracted as an event.

[0118] The large-scale language model 160 generates response text 172b from prompt text 172a. Response text 172b indicates a single causal event. Figure 17 is a diagram (continued 1) showing an example of input and output data for a large-scale language model. The information processing device 100 receives the response text 172b and generates prompt text 173a, which it inputs to the large-scale language model 160. The prompt text 173a includes an instruction that tells the device to extract a list of causal relationships, including intermediate events, based on the previously input case text and the previously extracted causal and final events. The prompt text 173a also instructs the device to explicitly indicate logical AND or OR relationships if such relationships exist between multiple causal relationships.

[0119] Furthermore, prompt text 173a indicates, as a caution, that the example text may contain a chain of causal relationships in which a causal event triggers an intermediate event, which in turn triggers another intermediate event, and which in turn triggers a final event. Prompt text 173a also indicates that the causal event and the final event are indirectly connected. The number of intermediate events between the causal event and the final event, and the branching structure of the path between the causal event and the final event, vary depending on the example text. Due to the above caution, the information processing device 100 ensures that the large-scale language model 160 extracts all causal relationships related to intermediate events from the event text without omission.

[0120] The large-scale language model 160 generates response text 173b from prompt text 173a. Response text 173b represents one intermediate event and shows three causal relationships between one causal event, one intermediate event, and two final events. Response text 173b shows that the causal event causes the intermediate event, and that the intermediate event causes the two final events. Furthermore, response text 173b shows that the two causal relationships from the intermediate event to the two final events have a logical AND relationship.

[0121] Figure 18 is a diagram (continued 2) showing an example of input and output data for a large-scale language model. The information processing device 100 receives the response text 173b, generates prompt text 174a, and inputs it to the large-scale language model 160. The prompt text 174a includes an instruction that instructs the device to extract other events and add causal relationships to the list based on the previously input case text and the previously extracted list of causal relationships.

[0122] The large-scale language model 160 generates response text 174b from prompt text 174a. Response text 174b shows four other events and seven causal relationships between one causal event, one intermediate event, two final events, and the four other events. The list shown in response text 174b is generated by adding causal relationships to the list shown in response text 173b.

[0123] Response text 174b, together with the two final events, shows that the intermediate event triggers the first other event. Response text 174b also shows that the intermediate event triggers the second other event, the second other event triggers the third other event, and the third other event triggers the fourth other event.

[0124] The information processing device 100 receives the response text 174b, generates prompt text 175a, and inputs it to the large-scale language model 160. The prompt text 175a includes an instruction that instructs the device to extract log events and add suggestive relationships to the list based on the previously input case text and the previously extracted list of causal relationships.

[0125] The large-scale language model 160 generates response text 175b from prompt text 175a. Response text 175b shows three log events and, in addition to the seven causal relationships shown in response text 174b, shows three suggestive relationships. Response text 175b shows that the first log event suggests the first other event. Response text 175b also shows that the second log event suggests the second other event. Response text 175b also shows that the third log event suggests the third other event.

[0126] Figure 19 is a diagram (continued 3) showing an example of input and output data for a large-scale language model. The information processing device 100 receives the response text 175b, generates prompt text 176a, and inputs it to the large-scale language model 160. The prompt text 176a includes the example text of the example document 136 and the list of the response text 175b. The prompt text 176a also includes an instruction that calculates the similarity between the example text and the list, and outputs the similarity along with the reason why the similarity was calculated.

[0127] The large-scale language model 160 generates response text 176b from prompt text 176a. Response text 176b indicates a similarity of 100%. Furthermore, response text 176b indicates that the example text and the list provide identical information regarding causal relationships, and that there is no difference or irrelevant information.

[0128] The information processing device 100 may omit list refinement if the similarity is sufficiently high. For example, the information processing device 100 omits list refinement if the similarity is 100% or exceeds a certain value. The information processing device 100 compares the similarity shown in response text 176b with a threshold. Since the similarity exceeds the threshold, the information processing device 100 adopts the list shown in response text 175b. The information processing device 100 converts the list in response text 175b into a causal knowledge graph.

[0129] Figure 20 is a diagram (continued 4) showing an example of input and output data for a large-scale language model. As described above, the information processing device 100 generates prompt text 176a and inputs it to the large-scale language model 160. Here, we will explain the case where the similarity is lower than that shown in Figure 19.

[0130] The large-scale language model 160 generates response text 176c from prompt text 176a. Response text 176c indicates a similarity of 90%. Response text 176c also indicates that the causal relationships included in the list as additional causal relationships, which are not irrelevant to the final event but rather important for the analysis of the final event, are the reason for the deduction in similarity.

[0131] Since the list in response text 175b is worthy of modification, the information processing device 100 receives response text 176c and generates prompt text 177a, which is then input to the large-scale language model 160. Prompt text 177a includes instructions to modify the list based on the previously calculated similarity and the reasons for that similarity.

[0132] The large-scale language model 160 generates response text 177b from prompt text 177a. Response text 177b contains a modified list from response text 175b. Response text 177b shows the relationship between one causal event, two intermediate events, five final events, and two log events.

[0133] The information processing device 100 calculates the similarity of the list in response text 177b and compares it to a threshold. If the similarity exceeds the threshold, the information processing device 100 adopts the list indicated in response text 177b. The information processing device 100 converts the list in response text 177b into a causal knowledge graph. On the other hand, if the similarity does not exceed the threshold, the information processing device 100 discards the list in response text 177b. The information processing device 100 re-executes the query to the large-scale language model 160 from prompt text 171a. Due to the influence of random numbers, the large-scale language model 160 may generate a different response text than the previous one. Next, the functions and processing procedures of the information processing device 100 will be described.

[0134] Figure 21 is a block diagram showing an example of the functions of an information processing device. The information processing device 100 includes a document storage unit 121, a model storage unit 122, a graph storage unit 123, a model access unit 124, an event extraction unit 125, a causal evaluation unit 126, a graph generation unit 127, and a graph inference unit 128. The document storage unit 121, the model storage unit 122, and the graph storage unit 123 are implemented using, for example, RAM 102 or HDD 103. The model access unit 124, the event extraction unit 125, the causal evaluation unit 126, the graph generation unit 127, and the graph inference unit 128 are implemented using, for example, a CPU 101, a GPU 104, and a program.

[0135] The document storage unit 121 stores case documents. The document storage unit 121 corresponds to the case database 130. The model storage unit 122 stores large-scale language models. However, it is sufficient that the information processing device 100 can utilize the large-scale language models, and the large-scale language models may be stored in other information processing devices. The graph storage unit 123 stores causal knowledge graphs.

[0136] The model access unit 124 performs natural language processing to generate response text from prompt text using a trained large-scale language model stored in the model storage unit 122. However, the model access unit 124 may also use a large-scale language model stored in another information processing device. In that case, the model access unit 124 sends the prompt text to the other information processing device and receives the response text from the other information processing device.

[0137] The event extraction unit 125 reads a case document from the document storage unit 121 and extracts a case text from the case document that represents one failure case. The event extraction unit 125 then sequentially extracts the final event, causal event, intermediate events, other events, and log events from the case text.

[0138] First, the event extraction unit 125 generates prompt text for extracting the final event from the case text and passes it to the model access unit 124, and obtains the extraction result of the final event from the model access unit 124. Next, the event extraction unit 125 generates prompt text for extracting the causal event from the case text and passes it to the model access unit 124, and obtains the extraction result of the causal event from the model access unit 124. Next, the event extraction unit 125 generates prompt text for extracting intermediate events from the case text and passes it to the model access unit 124, and obtains the extraction result of the intermediate events from the model access unit 124.

[0139] Next, the event extraction unit 125 generates prompt text for extracting other events from the case text and passes it to the model access unit 124, and obtains the extraction results of other events from the model access unit 124. Finally, the event extraction unit 125 generates prompt text for extracting log events from the case text and passes it to the model access unit 124, and obtains the extraction results of log events from the model access unit 124.

[0140] The causal evaluation unit 126 generates prompt text to calculate the similarity between the case text and the list of extracted results and passes it to the model access unit 124, and obtains the similarity from the model access unit 124. The causal evaluation unit 126 compares the similarity with a threshold. If the similarity exceeds the threshold, the causal evaluation unit 126 determines that the extraction of the causal relationship was successful. If the similarity does not exceed the threshold, the causal evaluation unit 126 determines that the extraction of the causal relationship failed. In the case of extraction failure, the causal evaluation unit 126 requests the event extraction unit 125 to re-extract.

[0141] The graph generation unit 127 converts the extraction results whose similarity exceeds the threshold into a causal knowledge graph format. The graph generation unit 127 saves the causal knowledge graph to the graph storage unit 123. If there is a causal knowledge graph generated from another failure case, the graph generation unit 127 integrates the causal knowledge graph generated this time into the existing causal knowledge graph.

[0142] The graph inference unit 128 receives a query text inquiring about the cause of the failure. The graph inference unit 128 generates prompt text to extract the final event from the query text and passes it to the model access unit 124, and retrieves the final event from the model access unit 124. The graph inference unit 128 searches for a node indicating the final event from the causal knowledge graph stored in the graph storage unit 123 and searches for a node indicating the cause event by tracing the causal relationship. The graph inference unit 128 generates and outputs an answer text indicating the estimated cause event. The graph inference unit 128 may save the answer text to non-volatile storage, display it on the display device 111, or transmit it to another information processing device.

[0143] Figure 22 is a flowchart showing an example of the graph generation procedure. In step S10, the event extraction unit 125 uses a large-scale language model to extract the final event from the case text. In step S11, the event extraction unit 125 uses a large-scale language model to extract the causal events corresponding to the final event from the case text.

[0144] In step S12, the event extraction unit 125 uses a large-scale language model to extract intermediate events between the causal event and the final event from the case text. In step S13, the event extraction unit 125 uses a large-scale language model to extract other events from the case text. In step S14, the event extraction unit 125 uses a large-scale language model to extract log events from the case text.

[0145] In step S15, the causal evaluation unit 126 uses a large-scale language model to evaluate the similarity between the extraction results from steps S10 to S14 and the case text. The causal evaluation unit 126 also uses the large-scale language model to generate a report showing the reasons for the evaluation. In step S16, the causal evaluation unit 126 uses the large-scale language model to modify the extraction results to reflect the reasons for the evaluation. The causal evaluation unit 126 uses the large-scale language model to evaluate the similarity between the modified extraction results and the case text.

[0146] In step S17, the causal evaluation unit 126 determines whether the similarity exceeds a threshold. If the similarity exceeds the threshold, the process proceeds to step S18; otherwise, the process returns to step S10. In step S18, the graph generation unit 127 converts the extraction results into a causal knowledge graph format and saves the causal knowledge graph.

[0147] As described above, the information processing device 100 of the second embodiment receives a query text inquiring about the cause of a failure, and outputs a response text indicating candidate causes of the failure, referring to past failure cases. This allows the information processing device 100 to support engineers in analyzing the cause of failures and to expedite the failure recovery process.

[0148] Furthermore, the information processing device 100 generates a causal knowledge graph from the case text that shows the failure cases. The information processing device 100 estimates the causal event corresponding to the final event by tracing edges on the causal knowledge graph. This allows the information processing device 100 to easily utilize the knowledge gained from the failure cases and improve the accuracy of causal estimation.

[0149] Furthermore, the information processing device 100 extracts various events from the case text in a fixed order: extraction of the final event, extraction of the causal event, extraction of intermediate events, extraction of other events, and extraction of log events. This reduces the risk of skipping intermediate events and directly connecting the causal event and the final event, as well as the risk of splitting the chain of causal relationships. As a result, the accuracy of the causal knowledge graph is improved.

[0150] Furthermore, the information processing device 100 calculates the similarity between the extracted results and the original case text. If the similarity is low, the information processing device 100 re-extracts various events from the case text, utilizing the property that the output of the large-scale language model depends on random numbers. This improves the accuracy of the causal knowledge graph. In addition, the information processing device 100 causes the large-scale language model to generate reasons for the similarity and feeds these reasons back to refine the extracted results. This reduces the number of iterations required to obtain an extracted result with a similarity exceeding a threshold, thereby shortening the execution time required to generate the causal knowledge graph.

[0151] Furthermore, the causal knowledge graph includes nodes that represent log events. This allows the information processing device 100 to refer to log events and narrow down or prioritize causal relationship paths. As a result, the accuracy of cause estimation using the causal knowledge graph is improved.

[0152] Furthermore, the information processing device 100 instructs the system in the prompt text not to extract analysis work or countermeasures performed by engineers as events. It also instructs the system in the prompt text not to split time-series events, which are lists of multiple phenomena that occurred consecutively, even if logical AND or OR expressions are used. Finally, the system instructs the system in the prompt text to refer to the countermeasures section when interpreting the causal events. This improves the accuracy of the extraction of various events by the large-scale language model. [Explanation of Symbols]

[0153] 10 Information Processing Devices 11 Storage section 12 Processing Units 13 Text 14. Knowledge Graph 15,16,17a,17b Entities

Claims

1. From the text, extract the first entity that represents the final event, Based on the extraction results of the first entity, a second entity representing the root cause event corresponding to the final event is extracted from the text. Based on the extraction results of the first and second entities, a third entity is extracted from the text that represents an intermediate event on the causal path from the root cause event to the final event. The similarity between the knowledge graph showing the causal relationships between the first entity, the second entity, and the third entity, and the text is determined. If the similarity exceeds the threshold, the knowledge graph is output. A generation method in which a computer performs the processing.

2. The process for extracting the first entity includes specifying the text to the natural language processing model, thereby causing the natural language processing model to extract the first entity. The process for extracting the second entity includes specifying the text and the first entity to the natural language processing model, thereby causing the natural language processing model to extract the second entity. The process for extracting the third entity includes specifying the text, the first entity, and the second entity to the natural language processing model, thereby causing the natural language processing model to extract the third entity. The generation method according to claim 1.

3. If the similarity does not exceed the threshold, the computer further performs a process to re-extract the first entity, the second entity, and the third entity so that the extraction results of the first entity, the second entity, and the third entity change. The generation method according to claim 1.

4. The aforementioned re-extraction process includes a process to change the random numbers used for natural language processing of the text. The generation method according to claim 3.

5. The process for determining the similarity includes specifying the knowledge graph and the text to the natural language processing model, thereby causing the natural language processing model to determine the similarity. The generation method according to claim 1.

6. The process for determining the similarity includes causing the natural language processing model to generate other text that indicates the reason for determining the similarity, The computer further performs a process to cause the natural language processing model to modify the knowledge graph based on the aforementioned other text. The generation method according to claim 5.

7. Based on the extraction results of the first entity, the second entity, and the third entity, the computer further performs a process to extract a fourth entity from the text that represents other events not on the path. The knowledge graph further includes the fourth entity: The generation method according to claim 1.

8. The aforementioned text includes log strings output by the information processing system. Based on the extraction results of the first entity, the second entity, and the third entity, the computer further performs a process of associating the log string with an entity that represents the event corresponding to the log string. The generation method according to claim 1.

9. The aforementioned text is a case study text describing a failure in an information processing system and the cause of the said failure. The generation method according to claim 1.

10. From the text, extract the first entity that represents the final event, Based on the extraction results of the first entity, a second entity representing the root cause event corresponding to the final event is extracted from the text. Based on the extraction results of the first and second entities, a third entity is extracted from the text that represents an intermediate event on the causal path from the root cause event to the final event. The similarity between the knowledge graph showing the causal relationships between the first entity, the second entity, and the third entity, and the text is determined. If the similarity exceeds the threshold, the knowledge graph is output. A generation program that causes a computer to perform a process.

11. A memory unit that stores text, A processing unit extracts a first entity representing the final event from the text, extracts a second entity representing the root cause event corresponding to the final event from the text based on the extraction result of the first entity, extracts a third entity representing an intermediate event on the causal relationship path from the root cause event to the final event from the text based on the extraction results of the first entity and the second entity, determines the similarity between the knowledge graph showing the causal relationship between the first entity, the second entity and the third entity and the text, and outputs the knowledge graph if the similarity exceeds a threshold. An information processing device having

Images