Identification method, identification program, and information processing device

Abstract

An information processing device (1) exchanges information with an LLM (40) that indicates a tool to be executed by an execution program (20) for identifying an answer to a question, such tool being from among a plurality of tools including a tool for acquiring information on a file on the basis of information outputted from the execution program (20), a tool for creating an overview of a file, and a tool for extracting a relevant section in a file. The present invention includes identifying an answer that contains a section that affects failure of the file, by repeating the exchange of information with the LLM (40) until the LLM (40) outputs an answer. The processing of exchanging the information includes causing, on the basis of the instruction by the LLM (40), a tool corresponding to the instruction to execute the instruction, and outputting information indicating a result corresponding to the executed instruction to the LLM (40). The processing of this information processing device (1) can be applied, for example, to failure analysis for a source program using the LLM (40).

Description

Identification method, identification program, and information processing device

[0001] The present invention relates to a specification method and the like.

[0002] In recent years, a large-scale language model known as an LLM (Large Language Model) has been attracting attention. A method has been proposed in which an LLM is combined with a program to collect necessary information and then generate an answer. One such method is called ReAct (see, for example, Non-Patent Document 1). ReAct is a broad technology that repeatedly uses external tools to collect information and generate an answer.

[0003] JP 2023-086678 A JP 2023-018624 A U.S. Patent Application Publication No. 2019 / 0250979 U.S. Patent Application Publication No. 2014 / 0278290

[0004] Shunyu Yao, et al. “ReAct: Synergizing Reasoning and Acting in Language Models”

[0005] However, conventional ReAct is a method of generating answers by repeatedly using external tools to collect information, and it is necessary to design appropriate tools to be used. If appropriate tools are not designed, there is a problem that the accuracy of the answers will decrease.

[0006] In one aspect, the present invention aims to improve the accuracy of answers in ReAct.

[0007] In one aspect, in the identification method, a program that identifies an answer to a question exchanges information with a first learning model that instructs the program to execute one of a plurality of tools, including a tool that obtains file information based on information output from the program, a tool that creates a file summary, and a tool that extracts relevant parts of the file, and identifies the answer including the part that will affect the file failure by repeating the exchange of information with the first learning model until the first learning model outputs the answer using information obtained by having the program execute the tool, and the process of exchanging information is performed by a computer, which process includes having a tool corresponding to the instruction execute the instruction based on the instruction of the first learning model, and outputting information indicating the result corresponding to the executed instruction to the first learning model.

[0008] According to one embodiment, ReAct can improve the accuracy of answers.

[0009] FIG. 1 is a diagram illustrating an example of the flow of a specification method according to the first embodiment. FIG. 2 is a diagram illustrating an example of the functional configuration of an information processing device according to the first embodiment. FIG. 3 is a diagram illustrating an example of a prompt used in an execution program according to the first embodiment. FIG. 4 is a diagram illustrating an example of the flow of an information acquisition process according to the first embodiment. FIG. 5 is a diagram illustrating an example of the flow of an outline creation process according to the first embodiment. FIG. 6 is a diagram illustrating an example of the flow of a related part extraction process according to the first embodiment. FIG. 7 is a diagram illustrating an example of a flowchart of an execution program according to the first embodiment. FIG. 8 is a diagram illustrating an example of the functional configuration of an information processing device according to the second embodiment. FIG. 9 is a diagram illustrating an example of a computer that executes a specification program. FIG. 10 is a reference diagram for explaining ReAct. FIG. 11 is a diagram illustrating a reference example of input / output between a ReAct execution program and an LLM.

[0010] Hereinafter, embodiments of the identification method, the identification program, and the information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments.

[0011] First, ReAct will be described with reference to Figures 10 and 11. ReAct is a technique that combines a program and an LLM to collect necessary information and then generate a response. ReAct is a broad technology that repeatedly uses external tools to collect information and generate a response.

[0012] Fig. 10 is a reference diagram for explaining ReAct. In Fig. 10, for example, a ReAct execution program identifies an answer to a question such as "What is the cause of XX?" One example of XX is when the output differs from what is expected when source code is executed.

[0013] The ReAct execution program receives a question from the user, presents the question and available tools to the LLM, and asks the LLM to use the tools to come up with an answer to the question. In this example, the ReAct execution program presents the LLM with a question such as "Tell me the cause of ____" and a set of available tools, including a file reading tool and a search tool. The target files are, for example, source code or development documents.

[0014] The LLM outputs to the ReAct execution program which tool to execute in the next step, and the ReAct execution program executes the output tool and returns the results to the LLM. The LLM then outputs a final answer to the question after using the tool several times. The ReAct execution program outputs the answer output by the LLM to the user.

[0015] Figure 11 is a diagram showing a reference example of input / output between the ReAct execution program and the LLM. As shown in Figure 11, input / output between the ReAct execution program and the LLM is performed using prompts. Here, prompts representing input from the ReAct execution program to the LLM are indicated by symbols p1, p2, and p3. Responses representing output from the LLM are indicated by symbols r1, r2, and r3.

[0016] First, the input and output shown in FIG. 11 is a case where a calculation problem is used as a question. The first prompt p1 represents input from the ReAct execution program to the LLM. This prompt p1 includes available tools, an explanation of the rules of the conversation, and the problem to be solved. In this example, the available tools are "terminal" and "Calculator." The problem to be solved is "Please calculate the square of the capacity (GB) of the storage mounted in / boot on this machine." The part indicated by the symbol a1 is implemented, for example, using the open source software (OSS) LangChain.

[0017] Upon receiving prompt p1, the LLM interprets it and outputs which tool to execute. The response output from the LLM is indicated by r1. In this example, response r1 describes "terminal" as the tool and "df -h / boot" as the argument.

[0018] The ReAct execution program, which has received the response r1, executes the tool specified in the response r1 with an argument and returns the execution result to the LLM. The prompt that the ReAct execution program gives to the LLM is indicated by p2. Here, " / dev / sda2 2.0G..." is displayed.

[0019] The LLM, which receives prompt p2, interprets it and outputs which tool to execute. The response output from the LLM is indicated by r2. In this example, response r2 indicates "Calculator" as the tool and "2.0 * 2.0" as the argument.

[0020] The ReAct execution program, which has received response r2, executes the tool specified in response r2 with an argument and provides the execution result to the LLM. The prompt that the ReAct execution program provides to the LLM is indicated by p3, which contains the value "4.0."

[0021] Then, upon receiving prompt p3, the LLM outputs the final answer to the question. The response output from the LLM is indicated by r3. Here, prompt p6 describes "Final Answer: 4.0." When "Final Answer" is output, the ReAct execution program ends. In other words, after using the tool several times, the LLM outputs the final answer to the question, and the ReAct execution program ends.

[0022] Such a ReAct execution program repeatedly uses tools to gather information and generate answers, but it is necessary to design appropriate tools to use.

[0023] For example, suppose the question is to identify the parts of the source code of a program under development that affect a failure. In such a case, if a tool is used that simply reads and searches files, a single file containing the source code of the program under development may implement multiple functions, and may contain a lot of information that is not important to the failure that has occurred. Similarly, if a tool is used that simply reads and searches files, the source code of the program under development may span multiple files, and may contain a lot of information that is not important to the failure that has occurred. If a lot of unimportant information is included, the accuracy of the answer will be reduced when identifying the parts that cause the failure.

[0024] Therefore, in the following embodiment, a method for accurately identifying locations that will be affected by a fault from the source code of a program under development will be described.

[0025] (Flow of Identification Method) FIG. 1 is a diagram illustrating an example of the flow of the identification method according to the first embodiment. As illustrated in FIG. 1 , the execution program 20 presents a question and available tools to a cause analysis LLM (LLM(A)) and engages in a conversation with the LLM(A) about the question. In response to instructions from the LLM(A), the execution program 20 retrieves data from the storage / DB 51 and issues instructions to the associated Agent (LLM). The execution program then ultimately identifies an answer corresponding to the question in response to the output from the LLM(A). Note that the identification method according to the first embodiment may be an extension of the existing ReAct, or may be an extension of an existing technology that combines the execution program 20 and the LLM(A) and further repeatedly uses tools to gather necessary information and then generate an answer.

[0026] In FIG. 1 , for example, the question is, "What is the reason why the output is different from the expected when the source code is executed?" Also, the available tools are a file information acquisition tool, a file overview creation agent (LLM(B)), and a file-related portion extraction agent (LLM(C)). LLM(B) and LLM(C) are independent LLMs separate from LLM(A), but they may be the same LLM as LLM(A). In other words, by performing file overview creation and file-related portion extraction in a sub-LLM, noise information is prevented from being input to LLM(A), which performs the main cause analysis, and the accuracy of the answers is prevented from decreasing.

[0027] Under these circumstances, the execution program 20 converses with the cause analysis LLM (LLM(A)) about the question (S100). For example, the LLM(A) outputs an instruction to the execution program 20, such as "Show me a list of program files (source code)" (S110). As an example, the LLM(A) instructs the execution program 20 to use a file information acquisition tool to execute the instruction. The file information acquisition tool is used to grasp the entire program file (source code).

[0028] In response to the instruction from LLM(A), the execution program 20 uses the file information acquisition tool to acquire data from the storage / DB 51 (S120).The execution program 20 then outputs the results of the instruction to LLM(A) (S130).

[0029] Next, the LLM (A) that has received the result of the instruction outputs an instruction to the execution program 20, such as "File xx seems to be related, please give me an overview" (S140). As an example, the LLM (A) instructs the execution program 20 to use a file outline creation agent to execute the instruction. The file outline creation agent is an LLM (B) that is separate and independent from the LLM (A), and is used to grasp the overview of the likely related parts.

[0030] In response to an instruction from LLM(A), the execution program 20 acquires a file summary created using the file summary creation agent (S150), and then outputs the results of the instruction to LLM(A) (S160).

[0031] Next, the LLM(A) that has received the result of the instruction further outputs an instruction to the execution program 20, saying, "Please also give me an overview of the file XX" (S170). As an example, the LLM(A) instructs the execution program 20 to use a file overview creation agent that is used to execute the instruction. The file overview creation agent is used to grasp an overview of parts that may be related.

[0032] In response to an instruction from LLM(A), the execution program 20 acquires a file summary created using the file summary creation agent (S180), and then outputs the result of the instruction to LLM(A) (S190).

[0033] Next, the LLM (A) that has received the result of the instruction outputs an instruction to the execution program 20, such as "Extract only the part that processes XX from △△" (S200). As an example, the LLM (A) instructs the execution program 20 to use the file-related part extraction agent to execute the instruction. The file-related part extraction agent is an LLM (C) that is separate and independent from the LLM (A), and is used to grasp the details of only the parts that seem to be related.

[0034] In response to an instruction from LLM(A), the execution program 20 obtains information that extracts relevant parts using the file relevant part extraction agent (S210), and then outputs the results of the instruction to LLM(A) (S220).

[0035] Next, the LLM (A) that has received the result of the instruction outputs an instruction to the execution program 20, such as "Extract the part where XX processing is performed at the call destination of XX" (S230). As an example, the LLM (A) instructs the execution program 20 to use a file-related part extraction agent to execute the instruction. The file-related part extraction agent is used to grasp the details of only the parts that seem to be related.

[0036] In response to an instruction from LLM(A), the execution program 20 obtains information that extracts relevant parts using the file relevant part extraction agent (S240), and then outputs the results of the instruction to LLM(A) (S250).

[0037] Then, the LLM(A) generates an answer to the question using each piece of information acquired, and outputs the generated answer to the execution program 20 (S260).

[0038] As described above, the execution program 20 provides the LLM(A) with relevant information and an answer to the cause of the problem while having the LLM(A) consider the order in which to focus on the file information acquisition tool, file summary creation agent, and file related part extraction agent, and the order in which to focus on which files.The execution program 20 then uses the tools and agents to provide the LLM(A) with an overall picture of the file from the file name and structure, the implementation status of surrounding parts from the summary, and detailed information on only parts that seem to be related, thereby reducing the amount of information according to importance and efficiently inputting information into the LLM(A).

[0039] As a result, by using the file information acquisition tool, file summary creation agent, and file related section extraction agent, the execution program 20 can efficiently collect information needed to obtain an answer in a step-by-step manner, enabling accurate identification of the cause of the problem. Furthermore, by having an LLM separate from the LLM (A) perform the file summary creation and related section extraction, the execution program 20 can prevent noise information from being input to the LLM (A) that performs the main cause analysis, thereby preventing a decrease in the accuracy of the answer. In other words, the execution program 20 can improve the accuracy of the answer to the question.

[0040] (Functional Configuration of Information Processing Apparatus) Fig. 2 is a diagram illustrating an example of the functional configuration of the information processing apparatus according to Example 1. As illustrated in Fig. 2, the information processing apparatus 1 converses with the LLM 40 and identifies an answer to a question.

[0041] The LLM 40 is an artificial intelligence (AI) that analyzes causes. The LLM 40 corresponds to the cause analysis LLM (LLM(A)) in FIG. 1 . The LLM 40 receives a request for an answer to a question from the execution program 20, including information on multiple executable tools and an explanation of the rules of conversation with the execution program 20. The tool information includes how to use the tools. The LLM 40 also interprets the tool information and the rules of conversation to infer the order in which the tools should be used and which files to focus on. The LLM 40 then instructs the execution program 20 to execute a specific tool from among the multiple tools to obtain the information necessary to answer the question. The LLM 40 also generates an answer to the question using the execution results corresponding to the instructions output from the execution program 20 and outputs the generated answer to the execution program 20. Note that the LLM 40 may be, for example, an existing ReAct LLM. Furthermore, LLM40 may be written as LLM(A). Furthermore, LLM40 is an example of a first learning model.

[0042] The information processing device 1 has a control unit 10 and a storage unit 50. The storage unit 50 has a storage / DB (Data Base) 51, an LLM 52, and an LLM 53. The LLM 52 and the LLM 53 may be configured by other information processing devices (not shown) different from the information processing device 1. In this case, the other information processing device and the information processing device 1 may communicate with each other as needed to exchange information.

[0043] The storage / DB 51 stores, for example, development documents 511 and source code 512. Note that the objects stored in the storage / DB 51 vary depending on the question.

[0044] LLM 52 is, for example, an AI that creates a summary of a file. LLM 52 corresponds to the file summary creation agent (LLM(B)) in FIG. 1. LLM 53 is, for example, an AI that extracts relevant parts of a file. LLM 53 corresponds to the file relevant part extraction agent (LLM(C)) in FIG. 1. Note that LLM 52 and LLM 53 may be written as LLM(B) and LLM(C), respectively.

[0045] The control unit 10 has an execution program 20. The execution program 20 has a conversation processing unit 21, a response output unit 22, and a tool group 30. The conversation processing unit 21 is an example of an input / output unit. The response output unit 22 is an example of an identification unit.

[0046] The conversation processing unit 21 processes conversations with the LLM 40 .

[0047] For example, when the conversation processing unit 21 receives a question from a user, it outputs to the LLM 40 a request for an answer to the question, including information on a plurality of executable tools.

[0048] The conversation processing unit 21 also receives an instruction to execute a tool from the LLM 40. Based on the instruction from the LLM 40, the conversation processing unit 21 causes the tool corresponding to the instruction to execute the instruction. For example, if the instruction from the LLM 40 is to acquire a list of target files (source code 512), the conversation processing unit 21 causes the information acquisition unit 31 (described later) to execute the instruction. For another example, if the instruction from the LLM 40 is to create an outline of the target file, the conversation processing unit 21 causes the outline creation unit 32 (described later) to execute the instruction. For another example, if the instruction from the LLM 40 is to extract relevant passages from the target file, the conversation processing unit 21 causes the relevant passage excerpting unit 33 (described later) to execute the instruction. The conversation processing unit 21 then outputs the results of the instructions executed by the tool to the LLM 40.

[0049] Then, the conversation processing unit 21 repeats the exchange of information with the LLM 40 until the LLM 40 outputs a response.

[0050] The answer output unit 22 outputs the answer to the user. For example, when the answer output unit 22 receives information from the LLM 40, the answer output unit 22 identifies an answer from the information. Then, the answer output unit 22 outputs the identified answer to the user.

[0051] The tool group 30 includes, for example, an information acquisition unit 31 , an outline creation unit 32 , and a relevant part extraction unit 33 .

[0052] The information acquisition unit 31 acquires information. For example, the information acquisition unit 31 acquires a list of target files (source code 512) from the storage / DB 51 in response to an instruction from the conversation processing unit 21. The information acquisition unit 31 then returns the acquired list of target files (source code 512) to the conversation processing unit 21.

[0053] The summary creation unit 32 creates a summary. For example, in response to an instruction from the conversation processing unit 21, the summary creation unit 32 reads a target file from the storage / DB 51 and causes the LLM 52 to create a summary of the read target file. The summary creation unit 32 then returns the created summary of the target file to the conversation processing unit 21.

[0054] The related passage excerpting unit 33 excerpts related passages. For example, the related passage excerpting unit 33 reads a target file from the storage / DB 51 in response to an instruction from the conversation processing unit 21, and causes the LLM 53 to excerpt related passages from the read target file. The related passage excerpting unit 33 then returns the excerpted related passages to the conversation processing unit 21.

[0055] (Prompts Used in Execution Program) Fig. 3 is a diagram showing an example of a prompt used in the execution program according to Example 1. Input and output between the execution program 20 and the LLM 40 is performed using prompts, as shown in Fig. 3. Fig. 3 shows a prompt that indicates the first input from the execution program 20 to the LLM.

[0056] This prompt includes available tools t0, an explanation of the rules of the conversation, and a problem to be solved. In this example, available tools t0 include, for example, "show_program_file_list," "show_overview_of_program_file," and "get_related_code."

[0057] "show_program_file_list" is a file information acquisition tool and corresponds to the information acquisition unit 31. "show_overview_of_program_file" is a file overview creation agent and corresponds to the overview creation unit 32. "get_related_code" is a file related part extraction agent and corresponds to the related part extraction unit 33. Note that the available tool t0 is not limited to the tools shown in FIG. 3, but changes for each question.

[0058] An explanation of the rules of the conversation and the problem to be solved are then input as explained in the reference example of input / output between the ReReAct execution program and the LLM in Fig. 11. After this prompt, the conversation continues between the execution program 20 and the LLM 40, as in the reference example of input / output between the ReReAct execution program and the LLM in Fig. 11.

[0059] Here, the flow of the information acquisition process, the summary creation process, and the relevant part extraction process executed by the execution program 20 will be described with reference to FIGS.

[0060] (Flow of Information Acquisition Process) FIG. 4 is a diagram illustrating an example of the flow of information acquisition process according to the first embodiment. Assume that the LLM (A) receives a request for an answer to a question from the execution program 20. Assume that the question means, "Please tell me why the output is different from the expected when the source code is executed." The prompt indicating the received request contains information on multiple executable tools, an explanation of the rules of the conversation, the question, and the like.

[0061] Then, LLM(A) outputs an execution instruction meaning "Show me a list of source code" to the execution program 20 (s1). Here, the prompt indicating the output execution instruction describes information such as "show_program_file_list" as the tool and nothing as the argument.

[0062] Upon receiving the execution instruction, the execution program 20 obtains a list of the target source code 512 from the storage / DB 51. As an example, the path of the directory in which the files are saved is defined in advance in the storage unit 50. The execution program 20 then reads a list of files under the directory path of the target source code 512 from the storage / DB 51 (s2).

[0063] Then, the execution program 20 outputs a list of the source code 512 that has been read to the LLM(A).

[0064] 5 is a diagram showing an example of the flow of the outlining process according to Example 1. It is assumed that the LLM (A) receives a list of target source code 512 from the execution program 20.

[0065] LLM(A) then outputs to the execution program 20 an execution instruction meaning "The source code 'moduleA / main.py' seems to be related, so please give me an overview" (s3). In this case, the prompt indicating the output execution instruction describes "show_overview_of_program_file" as the tool and "moduleA / main.py" as the argument.

[0066] Upon receiving the execution instruction, the execution program 20 reads the source code 512 called "moduleA / main.py" from the storage / DB 51 (s4). The execution program 20 then instructs the LLM (B) to execute the following for the read source code 512 (s5). The execution instruction here is "Please explain the classes and functions written in the source code 512 called "moduleA / main.py"."

[0067] Then, upon receiving the execution instruction, LLM(B) creates an outline of the classes and functions written in the source code 512 called "moduleA / main.py." LLM(B) then returns the created outline to the execution program 20. The execution program 20 then outputs the created outline to LLM(A).

[0068] 6 is a diagram illustrating an example of the flow of the related part extraction process according to Example 1. It is assumed that the LLM (A) receives an outline of the source code 512 called “moduleA / main.py” from the execution program 20.

[0069] LLM(A) then outputs to the execution program 20 an execution instruction meaning "Extract the part where XX processing is being executed from the source code 'moduleA / main.py'" (s6). Here, the prompt indicating the output execution instruction describes "get_related_code" as the tool and "'moduleA / main.py', 'the part where XX processing is being executed'" as arguments.

[0070] Upon receiving the execution instruction, the execution program 20 reads the source code 512 called "moduleA / main.py" from the storage / DB 51 (s7). The execution program 20 then instructs the LLM (C) to execute the following for the read source code 512. The execution instruction here is "Please extract the part of the Source Code that matches the content written in the Information." The Source here is "moduleA / main.py." The Information here is "the part where the XX process is being executed."

[0071] Then, upon receiving the execution instruction, LLM(C) extracts the "portion where XX processing is being executed" from "moduleA / main.py". Then, LLM(C) returns the extracted portion to the execution program 20. Then, the execution program 20 outputs the extracted portion to LLM(A).

[0072] (Flowchart of Execution Program) Here, a flowchart of the execution program 20 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a flowchart of the execution program according to the first embodiment.

[0073] As shown in FIG. 7 , the execution program 20 determines whether a question has been received from a user (step S11). If it is determined that a question has not been received (step S11; No), the execution program 20 repeats the determination process until a question is received. On the other hand, if it is determined that a question has been received (step S11; Yes), the execution program 20 presents the question and available tools to the LLM 40 (step S12). Here, the question is assumed to be, for example, a question meaning "Please tell me the cause of the bug in the source code." The available tools are assumed to be, for example, a file information acquisition tool, a file summary creation agent, and a file-related part extraction agent.

[0074] The execution program 20 then executes file information acquisition processing (step S13). For example, when the execution program 20 receives an instruction from the LLM 40 to acquire a list of target files (source code) using a file information acquisition tool, the execution program 20 uses the file information acquisition tool to acquire the list of target files (source code 512) from the storage / DB 51. The execution program 20 then outputs the acquired list of target files (source code 512) to the LLM 40.

[0075] The execution program 20 then executes the file summary creation process (step S14). For example, when the execution program 20 receives an instruction from the LLM 40 to create a summary of a target file using the file summary creation agent, the execution program 20 reads the target file from the storage / DB 51 and causes the LLM 52 to create a summary of the read target file. The execution program 20 then outputs the created summary of the target file to the LLM 40.

[0076] The execution program 20 then executes the file related portion extraction process (step S15). For example, when the execution program 20 receives an instruction from the LLM 40 to extract related portions from a target file using the file related portion extraction agent, the execution program 20 reads the target file from the storage / DB 51 and causes the LLM 53 to extract the related portions from the read target file. The execution program 20 then outputs the extracted related portions to the LLM 40.

[0077] The execution program 20 then identifies the answer (step S16). For example, the execution program 20 receives the answer from the LLM 40 and identifies the location of the bug in the source code 512. The execution program 20 then terminates.

[0078] In the first embodiment, the execution program 20 collects information about the source code 512 and identifies bugs in the source code 512. However, the execution program 20 is not limited to this. It may also collect information about files other than the source code 512, such as the development document 511, to identify inconsistencies between the source code 512 and the development document 511. In such a case, for example, the question may be, "Please tell me the reason why the output differs from the expected when the source code is executed, in relation to the development document." The execution program 20 then outputs a question including information about multiple tools to the LLM 40 and, based on instructions from the LLM 40, causes the corresponding tool to execute the instruction. As an example, an existing search tool is added to the tool group 30. When the execution program 20 receives an instruction from the LLM 40 to search the target development document 511, it causes the search tool to read the target development document 511 and outputs the execution result to the LLM 40. 4 to 6, the execution program 20 uses a tool to collect information about the source code 512 in response to instructions from the LLM 40, and outputs the collected results to the LLM 40. The execution program 20 then repeatedly exchanges information with the LLM 40 until the LLM 40 outputs an answer, thereby identifying an answer to the question. For example, the identified answer might be, "The XX document in the development documentation states XX, but the source code states △△."

[0079] Furthermore, each of the steps of the file information acquisition process (step S13), the file summary creation process (step S14), and the file related portion extraction process (step S15) may be performed multiple times or in a different order according to instructions from the LLM 40. For example, after the file information acquisition process (step S13) is performed once, the file summary creation process (step S14) and the file related portion extraction process (step S15) may be performed for a certain file, and then the file summary creation process (step S14) and the file related portion extraction process (step S15) may be performed for another file, after which the answer identification (step S16) may be performed.

[0080] Furthermore, when presenting the question and available tools to the LLM 40 (step S12), when executing the file summary creation process (step S14), or when executing the file relevant section excerpt process (step S15), the execution program 20 may include information about persona settings in the information exchanged with each LLM for the LLMs 40, 52, and 53 shown in FIG. 2 . One example of the information about persona settings is, "You are a software development expert." Such personas may be the same for the LLMs 40, 52, and 53, or different personas may be set for each LLM depending on the content of the question and the processing content of the available tools. This allows the LLMs 40, 52, and 53 to interpret the question and the execution content of the tools more specifically and professionally, thereby improving the accuracy of the tool execution results and the final answer to the question.

[0081] Effects of the First Embodiment According to the first embodiment, in the information processing device 1, the execution program 20, which identifies an answer to a question, exchanges information with the LLM 40, which instructs the execution program 20 to execute one of multiple tools, including a tool for acquiring file information based on information output from the execution program 20, a tool for creating a file summary, and a tool for extracting relevant portions of a file. The information processing device 1 then identifies an answer that includes a portion that may affect a file failure by repeatedly exchanging information with the LLM 40 until the LLM 40 causes the execution program 20 to execute the tool and outputs an answer using information obtained by the execution program 20. The information exchange process then involves causing a tool corresponding to the instruction to execute the instruction based on an instruction from the LLM 40, and outputting information indicating the result of the executed instruction to the LLM 40. As a result, the information processing device 1 can accurately identify a portion that may affect a source program failure by using multiple tools, including a tool for acquiring file information, a tool for creating a file summary, and a tool for extracting relevant portions of a file. That is, the information processing device 1 can gradually narrow down the information needed to obtain an answer by using a tool for acquiring file information, a tool for creating a file summary, and a tool for extracting relevant parts of the file. As a result, the information processing device 1 can accurately identify parts that affect the source program failure.

[0082] Furthermore, according to the first embodiment, in the information processing device 1, the process of executing an instruction is performed by exchanging information between an LLM that is separate and independent from the LLM 40 and a tool. In this way, by using an LLM that is separate and independent from the LLM 40 as a tool, the information processing device 1 can prevent noise information from being input to the LLM 40 that performs the main cause analysis, and prevent a decrease in the accuracy of the answer.

[0083] Furthermore, according to the first embodiment, the information processing device 1 executes instructions by including a persona in accordance with the content of the information exchanged in at least one of the information exchange with the LLM 40 and the information exchange with the LLMs 52 and 53. This enables the information processing device 1 to improve the accuracy of the tool execution results and the accuracy of the answers to questions.

[0084] The execution program 20 according to the first embodiment provides related information and answers the cause of a problem while having the LLM 40 consider the order in which to use the file information acquisition tool, file summary creation agent, and file related part extraction agent, and which files to focus on. In other words, the LLM 40 collects information while considering the order in which to collect the information necessary to obtain an answer, and then uses the collected information to generate and output an answer. However, the order in which the information is collected may affect the accuracy of the answer.

[0085] Therefore, in the following Example 2, an execution program 20 will be described that provides multiple steps, repeats the exchange of information for each of the multiple steps, and identifies an answer from the answer candidates output for each of the multiple steps.

[0086] FIG. 8 is a diagram illustrating an example of the functional configuration of an information processing device according to a second embodiment. Regarding the functional configuration of the information processing device 1 according to the second embodiment, the same components as those of the information processing device 1 illustrated in FIG. 2 are denoted by the same reference numerals, and descriptions of the overlapping components and operations will be omitted. The first embodiment differs from the second embodiment in that the LLM 40 is configured as a plurality of LLMs, LLM 40A and LLM 40B. The number of LLMs is not limited to two, and may be three or more. Furthermore, the LLMs 40A and 40B may be the same as the LLM 40, but are independent of the LLM 40. The first embodiment differs from the second embodiment in that the conversation processing unit 21 and the answer output unit 22 of the execution program 20 are replaced by a conversation processing unit 21A and an answer output unit 22A, respectively. The first embodiment differs from the second embodiment in that a result integration unit 23A is added to the execution program 20. Similarly to LLM 52 and LLM 53, LLM 40A and LLM 40B may be configured by other information devices (not shown) that are different from information processing device 1. Note that LLM 40A and LLM 40B may be referred to as LLM (A1) and LLM (A2), respectively.

[0087] The conversation processing unit 21A processes conversations with the LLM 40A and conversations with the LLM 40B.

[0088] For example, when the conversation processing unit 21A receives a question from a user, it generates multiple processing procedures based on the content of the question. Here, the procedure refers to, for example, the order of collecting information, such as "collect development documents 511 first, then collect source code 512," or "collect source code 512 first, then collect development documents 511." Another example refers to the order of executing tools, such as "execute the first tool first, then execute the second tool," or "execute the second tool first, then execute the first tool." The conversation processing unit 21A then outputs a request for an answer to the question to the LLM 40A, including information on multiple executable tools and one of the procedures. Additionally, the conversation processing unit 21A outputs a request for an answer to the question to the LLM 40B, including information on multiple executable tools and the other of the procedures.

[0089] The conversation processor 21A also receives an instruction to execute a tool from the LLM 40A. Based on the instruction from the LLM 40A, the conversation processor 21A causes the corresponding tool to execute the instruction. Additionally, the conversation processor 21A receives an instruction to execute a tool from the LLM 40B. Based on the instruction from the LLM 40B, the conversation processor 21A causes the corresponding tool to execute the instruction. Note that the conversation processor 21A may process conversations between the LLM 40A and the LLM 40B in series or in parallel.

[0090] The conversation processing unit 21A then repeats the exchange of information with the LLM 40A until the LLM 40A outputs a response. Additionally, the conversation processing unit 21A repeats the exchange of information with the LLM 40B until the LLM 40B outputs a response.

[0091] The result integration unit 23A integrates the answer results. For example, the result integration unit 23A receives the answer results from the LLM 40A. The result integration unit 23A receives the answer results from the LLM 40B. Then, the result integration unit 23A integrates the answer results.

[0092] The answer output unit 22A outputs the answer to the user. For example, the answer output unit 22A identifies the integrated answer result as the answer as is. Then, the answer output unit 22A outputs the identified answer to the user. Note that, although it has been described that the answer output unit 22A identifies the integrated answer result as the answer as is, this is not limited thereto, and when there are three or more LLMs, for example, any of the answer results may be identified as the answer by a method such as majority vote.

[0093] [Effects of Example 2] According to Example 2, the LLM 40A and the LLM 40B perform a plurality of procedures with the order of processing interchanged. In the information processing device 1, the process of identifying an answer involves repeatedly exchanging information for each of the plurality of procedures, and identifying an answer from the answer candidates output for each of the plurality of procedures. This allows the information processing device 1 to further improve the accuracy of the answer to the question sentence.

[0094] The illustrated tool group 30 is described as being provided within the execution program 20. However, the tool group 30 may be provided within the control unit 10 outside the execution program 20, as long as it is linked to the execution program 20. The tool group 30 is merely an example and may include various tools. The available tools may be changed for each question.

[0095] The illustrated LLM(A) 40, LLM(B) 52, and LLM(C) 53 may be existing LLMs or original LLMs. Although the LLM(B) 52 and LLM(C) 53 are described as being stored in the storage unit 50, they may also be LLMs on the web or on other information processing devices.

[0096] Furthermore, the components of the illustrated information processing device 1 do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of the information processing device 1 is not limited to that shown, and all or part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, the storage unit 50 may be connected to the information processing device 1 as an external device via a network.

[0097] Furthermore, the various processes described in the above embodiments can be realized by executing a pre-prepared program on a computer such as a personal computer or a workstation. Therefore, an example of a computer that executes a specific program that realizes the same functions as the information processing device 1 shown in Fig. 2 will be described below. Here, a specific program that realizes the same functions as the information processing device 1 will be described as an example. Fig. 9 is a diagram showing an example of a computer that executes a specific program.

[0098] 9 , computer 200 includes a CPU (Central Processing Unit) 203 that executes various types of arithmetic processing, an input device 215 that accepts data input from a user, and a display device 209. Computer 200 also includes a drive device 213 that reads programs and the like from a storage medium, and a communication I / F (Interface) 217 ​​that exchanges data with other computers via a network. Computer 200 also includes a memory 201 that temporarily stores various types of information, and an HDD (Hard Disk Drive) 205. Memory 201, CPU 203, HDD 205, display control unit 207, display device 209, drive device 213, input device 215, and communication I / F 217 are connected via a bus 219.

[0099] The drive device 213 is, for example, a device for the removable disk 211. The HDD 205 stores a specific program 205a and specific processing-related information 205b. The communication I / F 217 serves as an interface between the network and the inside of the device, and controls the input and output of data from other computers. The communication I / F 217 can be, for example, a modem or a LAN adapter.

[0100] The display device 209 displays a cursor, an icon, a tool box, and data such as documents, images, and function information. The display device 209 may be, for example, a liquid crystal display or an organic electroluminescence (EL) display.

[0101] The CPU 203 reads the specific program 205a, expands it in the memory 201, and executes it as a process. Such a process corresponds to each functional unit of the information processing device 1. The specific processing related information 205b includes, for example, the storage / DB 51. For example, the removable disk 211 stores each piece of information such as the specific program 205a.

[0102] It should be noted that the specific program 205a does not necessarily have to be stored in the HDD 205 from the beginning. For example, the program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD disk, magneto-optical disk, or IC card that is inserted into the computer 200. The computer 200 may then read and execute the specific program 205a from the medium.

[0103] REFERENCE SIGNS LIST 1 Information processing device 10 Control unit 20 Execution program 21 Conversation processing unit 22 Answer output unit 30 Tool group 31 Information acquisition unit 32 Summary creation unit 33 Related part extraction unit 40 LLM (A) 50 Memory unit 51 Storage / DB 52 LLM (B) 53 LLM (C) 511 Development document 512 Source code

Claims

1. A method for identifying an answer to a question, characterized in that the method includes a computer executing the following processes: a program for identifying an answer to a question exchanges information with a first learning model that instructs the program to execute one of a plurality of tools, including a tool for obtaining file information based on information output from the program, a tool for creating a file summary, and a tool for extracting relevant parts from within a file; and the program identifies the answer, including the part that may affect the file failure, by repeating the exchange of information with the first learning model until the first learning model outputs the answer using information obtained by having the program execute the tool; and the process of exchanging information includes: causing a tool corresponding to the instruction to execute the instruction based on the instruction of the first learning model; and outputting information indicating the result corresponding to the executed instruction to the first learning model.

2. The identification method described in claim 1, characterized in that the process of executing the instruction involves the tool exchanging information with a second learning model that is separate and independent from the first learning model to execute the instruction.

3. The method of identification described in claim 1, characterized in that the first learning model performs multiple steps with the order of processing changed, and the process of identifying the answer repeats the exchange of information for each of the multiple steps and identifies the answer from the answer candidates output for each of the multiple steps.

4. The method of claim 1, wherein the file includes software source code or documentation related to the source code.

5. The identification method described in claim 2, characterized in that the process of executing the instruction includes processing that includes a persona according to the content of the information exchanged in at least one of the information exchange with the first learning model and the information exchange with the second learning model.

6. The method of claim 5, wherein the persona includes the first learning model and / or the second learning model being a software development expert.

7. A program for identifying an answer to a question, the program exchanging information with a first learning model that instructs the program to execute one of a plurality of tools, including a tool for obtaining file information based on information output from the program, a tool for creating a file summary, and a tool for extracting relevant parts from a file, and identifying the answer including the part that will affect the file failure by repeating the exchange of information with the first learning model until the first learning model outputs the answer using information obtained by having the program execute the tool, the program exchanging information with the first learning model, the program exchanging information with the first learning model, the first learning model executing the tool corresponding to the instruction based on the instruction of the first learning model, and outputting information indicating the result corresponding to the executed instruction to the first learning model.

8. An information processing device comprising: a first learning model that instructs the program to execute one of a plurality of tools, including a tool for obtaining file information based on information output from a program that identifies an answer to a question, a tool for creating a file summary, and a tool for extracting relevant parts within a file; an input / output unit that exchanges information; and an identification unit that identifies the answer, including parts that will affect the file failure, by repeating information exchange with the first learning model until the first learning model outputs the answer using information obtained by having the program execute the tool, wherein the input / output unit causes a tool corresponding to the instruction to execute the instruction based on the instruction of the first learning model, and outputs information indicating the result corresponding to the executed instruction to the first learning model.

Images