Identification method, identification program, and information processing apparatus

Abstract

To improve the accuracy of an answer in ReAct.SOLUTION: An information processor 1 exchanges information with a LLM40 which instructs a tool to be executed by an executing program 20 among a plurality of tools including a tool for acquiring information of a file on the basis of information output from the executing program 20 for specifying an answer to a question sentence, a tool for creating an outline of the file, and a tool for extracting a related part in the file, and repeats the exchange of the information with the LLM40 until the LLM40 outputs the answer. The processing of exchanging the information includes specifying an answer including a part affecting the failure of the file, causes a tool corresponding to an instruction to execute the instruction based on the instruction of the LLM40, and outputs information indicating a result corresponding to the executed instruction to the LLM40. The processing of the information processor 1 may be applied to failure analysis of source programs using LLM40, for example.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a specification method and the like. [Background technology]

[0002] In recent years, large-scale language models known as LLMs (Large Language Models) have been attracting attention. A method has been proposed that combines this LLM with a program to collect necessary information and then generate an answer. One such method is called ReAct (see, for example, Non-Patent Document 1). ReAct is a broad technology that repeatedly uses external tools to collect information and generate an answer. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-086678 [Patent Document 2] Japanese Patent Publication No. 2023-018624 [Patent Document 3] US Patent Application Publication No. 2019 / 0250979 [Patent Document 4] US Patent Application Publication No. 2014 / 0278290 [Non-patent literature]

[0004] [Non-Patent Document 1] Shunyu Yao,et al.“ReAct:Synergizing Reasoning and Acting in Language Models” Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional ReAct is a method that repeatedly uses external tools to gather information and generate answers, and it is necessary to design appropriate tools to use them, otherwise the accuracy of answers will decrease.

[0006] In one aspect, the present invention aims to improve the accuracy of answers in ReAct. [Means for solving the problem]

[0007] In one aspect, in the identification method, a program that identifies an answer to a question exchanges information with a first learning model that instructs the program to execute a tool from among a plurality of tools, including a tool for acquiring file information based on information output from the program, a tool for creating a file summary, and a tool for extracting relevant portions of a file, and identifies the answer including a portion that may affect the failure of the file by repeating the exchange of information with the first learning model until the first learning model causes the program to execute the tool and outputs the answer using information obtained, The process of exchanging information is performed by a computer, which causes a tool corresponding to an instruction to execute the instruction based on the instruction of the first learning model, and outputs information indicating the result corresponding to the executed instruction to the first learning model. [Effects of the Invention]

[0008] According to one embodiment, the accuracy of answers can be improved in ReAct. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a flow of a specifying method according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a functional configuration of the information processing apparatus according to the first embodiment. [Figure 3]FIG. 3 is a diagram illustrating an example of a prompt used in the execution program according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the flow of the information acquisition process according to the first embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the flow of the outline creation process according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the flow of the related part extraction process according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a flowchart of an execution program according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a functional configuration of an information processing device according to the second embodiment. [Figure 9] FIG. 9 illustrates an example of a computer that executes a specific program. [Figure 10] FIG. 10 is a reference diagram for explaining ReAct. [Figure 11] FIG. 11 is a diagram showing a reference example of input / output between the ReAct execution program and the LLM. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the identification method, the identification program, and the information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments.

[0011] First, we will explain ReAct with reference to Figures 10 and 11. ReAct here is a technique that combines a program and LLM to collect necessary information and then generate an answer. ReAct is a broad technology that repeatedly uses external tools to collect information and generate an answer.

[0012] Figure 10 is a reference diagram explaining ReAct. In Figure 10, for example, the ReAct execution program identifies the answer to the question, "What is the cause of ____?" One example of ____ is when the output is different from what was expected when the source code was executed.

[0013] The ReAct execution program receives a question from the user, presents the question and available tools to the LLM, and asks the LLM to use the tools to come up with an answer to the question. In this example, the ReAct execution program presents the LLM with the question "What is the cause of ____?" and a set of available tools, including a file reading tool and a search tool. The target files could be, for example, source code or development documents.

[0014] The LLM outputs to the ReAct executor which tool to run in the next step, and the ReAct executor executes the output tool and returns the results to the LLM. The LLM then outputs the final answer to the question after using the tool several times. The ReAct executor then outputs the answer output by the LLM to the user.

[0015] Figure 11 is a diagram showing a reference example of input / output between the ReAct execution program and the LLM. As shown in Figure 11, input / output between the ReAct execution program and the LLM is performed using prompts. Here, the prompts representing input from the ReAct execution program to the LLM are indicated by symbols p1, p2, and p3. The responses representing the output of the LLM are indicated by symbols r1, r2, and r3.

[0016] First, the input and output shown in Figure 11 is a case where a calculation problem is used as a question. The first prompt p1 represents the input from the ReAct execution program to the LLM. This prompt p1 includes the available tools, an explanation of the rules of the conversation, and the problem to be solved. In this example, the available tools are "terminal" and "Calculator." The problem to be solved is "Calculate the square of the capacity (GB) of the storage mounted in / boot on this machine." The part indicated by the symbol a1 is implemented, for example, by the open source software (OSS) LangChain.

[0017] When LLM receives prompt p1, it interprets it and outputs which tool to run. The response output from LLM is shown as r1. In this example, response r1 contains "terminal" as the tool and "df -h / boot" as the argument.

[0018] The ReAct execution program, which receives response r1, executes the tool specified in response r1 with arguments and returns the execution result to the LLM. The prompt that the ReAct execution program gives to the LLM is shown in p2. Here, it shows " / dev / sda2 2.0G...".

[0019] Then, upon receiving prompt p2, the LLM interprets prompt p2 and outputs which tool to execute. The response output from the LLM is shown as r2. In this example, response r2 indicates "Calculator" as the tool and "2.0*2.0" as the argument.

[0020] Then, the ReAct execution program that receives response r2 executes the tool specified in response r2 with arguments and provides the execution results to the LLM. The prompt that the ReAct execution program provides to the LLM is indicated by p3, which contains "4.0".

[0021] Then, upon receiving prompt p3, the LLM outputs the final answer to the question. The response output from the LLM is indicated by r3. In this case, prompt p6 contains the entry "Final Answer: 4.0". When "Final Answer" is output, the ReAct execution program terminates. In other words, after using the tool several times, the LLM outputs the final answer to the question, and the ReAct execution program terminates.

[0022] Such a ReAct execution program repeatedly uses tools to gather information and generate answers, but it is necessary to design appropriate tools to use.

[0023] For example, suppose the question is to identify the parts of the source code of a program under development that affect a failure. In such a case, if a tool is prepared that simply reads and searches files, a single file containing the source code of the program under development may implement multiple functions, and may contain a lot of information that is not important to the failure that has occurred. Similarly, if a tool is prepared that simply reads and searches files, the source code of the program under development may span multiple files, and may contain a lot of information that is not important to the failure that has occurred. If a lot of unimportant information is included, the accuracy of the answer will decrease when identifying the parts that cause the failure.

[0024] Therefore, in the following embodiment, a method for accurately identifying locations that will be affected by a fault from the source code of a program under development will be described. [Example]

[0025] (Flow of identification method) FIG. 1 is a diagram illustrating an example of the flow of the identification method according to the first embodiment. As illustrated in FIG. 1, the execution program 20 presents a question and available tools to a cause analysis LLM (LLM(A)) and communicates with the LLM(A) about the question. In response to an instruction from the LLM(A), the execution program 20 acquires data from the storage / DB 51 and issues instructions to the associated Agent (LLM). Then, in response to the output from the LLM(A), the execution program finally identifies an answer corresponding to the question. Note that the identification method according to the first embodiment may be an extension of an existing ReAct, or may be an extension of an existing technology that combines the execution program 20 and the LLM(A) and further repeatedly uses tools to collect necessary information and then generate an answer.

[0026] In Figure 1, the question might be, for example, "What is the reason why the output is different from what is expected when the source code is executed?" The available tools are a file information acquisition tool, a file summary creation agent (LLM(B)), and a file-related part extraction agent (LLM(C)). LLM(B) and LLM(C) are independent LLMs separate from LLM(A), but they can also be the exact same LLM as LLM(A). By performing file summary creation and file-related part extraction in a sub-LLM, it is possible to prevent noise information from being input to LLM(A), which performs the main cause analysis, and thus prevent a decrease in the accuracy of the answers.

[0027] Under these circumstances, the execution program 20 converses with the cause analysis LLM (LLM(A)) about the question (S100). For example, the LLM(A) outputs an instruction to the execution program 20, such as "Show me a list of program files (source code)" (S110). As an example, the LLM(A) instructs the execution program 20 to use a file information acquisition tool to execute the instruction. The file information acquisition tool is used to grasp the entire program file (source code).

[0028] In response to the instruction from the LLM(A), the execution program 20 uses the file information acquisition tool to acquire data from the storage / DB 51 (S120). Then, the execution program 20 outputs the result of the instruction to the LLM(A) (S130).

[0029] Next, LLM(A), which has received the result of the instruction, outputs an instruction to the execution program 20, for example, "File xx seems to be related, please give me an overview" (S140). As an example, LLM(A) instructs the execution program 20 to use a file overview creation agent to execute the instruction. The file overview creation agent is an LLM(B) that is separate and independent from LLM(A), and is used to grasp the overview of parts that seem to be related.

[0030] In response to an instruction from LLM(A), the execution program 20 acquires a file summary created using the file summary creation agent (S150).The execution program 20 then outputs the result of the instruction to LLM(A) (S160).

[0031] Next, the LLM(A) that has received the result of the instruction further outputs an instruction to the execution program 20 saying, "Please also give me an overview of the file called XX" (S170). As an example, the LLM(A) instructs the execution program 20 to use a file overview creation agent that is used to execute the instruction. The file overview creation agent is used to grasp an overview of parts that may be related.

[0032] In response to an instruction from the LLM(A), the execution program 20 acquires a file summary created using the file summary creation agent (S180).The execution program 20 then outputs the result of the instruction to the LLM(A) (S190).

[0033] Next, the LLM(A) that has received the result of the instruction outputs an instruction to the execution program 20, for example, "Extract only the part that processes XX from △△" (S200). As an example, the LLM(A) instructs the execution program 20 to use the file related part extraction agent that is used to execute the instruction. The file related part extraction agent is an LLM(C) that is separate and independent from the LLM(A), and is used to grasp the details of only the parts that seem to be related.

[0034] In response to an instruction from LLM(A), the execution program 20 obtains information that extracts relevant parts using the file relevant part extraction agent (S210).The execution program 20 then outputs the results of the instruction to LLM(A) (S220).

[0035] Next, the LLM(A) that has received the result of the instruction outputs an instruction to the execution program 20, for example, "Extract the part where xx processing is performed at the call destination of xx" (S230). As an example, the LLM(A) instructs the execution program 20 to use a file-related part extraction agent that is used to execute the instruction. The file-related part extraction agent is used to grasp the details of only the parts that seem to be related.

[0036] In response to an instruction from LLM(A), the execution program 20 obtains information on extracted relevant parts using the file relevant part extraction agent (S240).The execution program 20 then outputs the result of the instruction to LLM(A) (S250).

[0037] Then, the LLM(A) generates an answer to the question using each piece of acquired information, and outputs the generated answer to the execution program 20 (S260).

[0038] As described above, the execution program 20 provides the relevant information and answers the cause of the failure while having the LLM(A) consider the order in which to focus on the file information acquisition tool, file summary creation agent, and file related part extraction agent, and the order in which to focus on which files.The execution program 20 then uses the tools and agents to provide the LLM(A) with an overall picture of the file from the file name and structure, the implementation status of surrounding parts from the summary, and detailed information on only parts that seem to be related, thereby reducing the amount of information according to importance and efficiently inputting information into the LLM(A).

[0039] As a result, by using the file information acquisition tool, file summary creation agent, and file related part extraction agent, the execution program 20 can efficiently collect information needed to obtain an answer in a step-by-step manner, thereby enabling the cause of the failure to be identified with high accuracy. Furthermore, by having an LLM that is separate from the LLM(A) execute the creation of a file summary and the extraction of related parts, the execution program 20 can prevent noise information from being input to the LLM(A) that performs the main cause analysis, thereby preventing a decrease in the accuracy of the answer. In other words, the execution program 20 can improve the accuracy of the answer to the question.

[0040] (Functional configuration of information processing device) 2 is a diagram illustrating an example of a functional configuration of an information processing device according to an embodiment. As illustrated in FIG. 2, the information processing device 1 has a conversation with the LLM 40 and identifies an answer to a question.

[0041] The LLM 40 is an artificial intelligence (AI) that analyzes causes. The LLM 40 corresponds to the cause analysis LLM (LLM(A)) in Figure 1. The LLM 40 receives a request for an answer to a question from the execution program 20, including information on multiple executable tools and an explanation of the rules of conversation with the execution program 20. The tool information includes how to use the tools. The LLM 40 also interprets the tool information and the rules of conversation to infer the order in which the tools should be used and which files to focus on. The LLM 40 then instructs the execution program 20 to execute a specific tool from among the multiple tools to obtain the information necessary to answer the question. The LLM 40 also generates an answer to the question using the execution results corresponding to the instructions output from the execution program 20 and outputs the generated answer to the execution program 20. The LLM 40 may be, for example, an existing ReAct LLM. The LLM 40 may also be referred to as LLM(A). The LLM 40 is an example of a first learning model.

[0042] The information processing device 1 has a control unit 10 and a storage unit 50. The storage unit 50 has a storage / DB (DataBase) 51, an LLM 52, and an LLM 53. Note that the LLM 52 and the LLM 53 may be configured by other information processing devices (not shown) different from the information processing device 1, in which case the other information processing devices and the information processing device 1 may communicate with each other as necessary to exchange information.

[0043] The storage / DB 51 stores, for example, development documents 511 and source code 512. Note that the objects stored in the storage / DB 51 differ depending on the question.

[0044] LLM52 is, for example, an AI that creates a summary of a file. LLM52 corresponds to the file summary creation agent (LLM(B)) in Figure 1. LLM53 is, for example, an AI that extracts relevant parts of a file. LLM53 corresponds to the file relevant part extraction agent (LLM(C)) in Figure 1. Note that LLM52 and LLM53 may be written as LLM(B) and LLM(C), respectively.

[0045] The control unit 10 has an execution program 20. The execution program 20 has a conversation processing unit 21, a response output unit 22, and a tool group 30. The conversation processing unit 21 is an example of an input / output unit. The response output unit 22 is an example of an identification unit.

[0046] The conversation processing unit 21 processes conversations with the LLM 40 .

[0047] For example, when the conversation processing unit 21 receives a question from a user, it outputs to the LLM 40 a request for an answer to the question, including information on a plurality of executable tools.

[0048] The conversation processing unit 21 also receives an instruction to execute a tool from the LLM 40. Then, based on the instruction from the LLM 40, the conversation processing unit 21 causes the tool corresponding to the instruction to execute the instruction. As an example, if the instruction from the LLM 40 is to acquire a list of target files (source code 512), the conversation processing unit 21 causes the information acquisition unit 31 (described later) to execute the instruction. As another example, if the instruction from the LLM 40 is to create an outline of the target file, the conversation processing unit 21 causes the outline creation unit 32 (described later) to execute the instruction. As another example, if the instruction from the LLM 40 is to extract relevant passages from the target file, the conversation processing unit 21 causes the relevant passage excerpting unit 33 (described later) to execute the instruction. Then, the conversation processing unit 21 outputs the results corresponding to the instructions executed by the tool to the LLM 40.

[0049] Then, the conversation processing unit 21 repeats the exchange of information with the LLM 40 until the LLM 40 outputs a response.

[0050] The answer output unit 22 outputs the answer to the user. For example, when the answer output unit 22 receives information from the LLM 40, it identifies an answer from the information. Then, the answer output unit 22 outputs the identified answer to the user.

[0051] The tool group 30 includes, for example, an information acquisition unit 31, an outline creation unit 32, and a relevant part extraction unit 33.

[0052] The information acquisition unit 31 acquires information. For example, the information acquisition unit 31 acquires a list of target files (source code 512) from the storage / DB 51 in response to an instruction from the conversation processing unit 21. Then, the information acquisition unit 31 returns the acquired list of target files (source code 512) to the conversation processing unit 21.

[0053] The summary creation unit 32 creates a summary. For example, in response to an instruction from the conversation processing unit 21, the summary creation unit 32 reads a target file from the storage / DB 51 and causes the LLM 52 to create a summary of the read target file. The summary creation unit 32 then returns the created summary of the target file to the conversation processing unit 21.

[0054] The relevant passage excerpting unit 33 excerpts relevant passages. For example, in response to an instruction from the conversation processing unit 21, the relevant passage excerpting unit 33 reads a target file from the storage / DB 51 and causes the LLM 53 to excerpt relevant passages from the read target file. The relevant passage excerpting unit 33 then returns the excerpted relevant passages to the conversation processing unit 21.

[0055] (prompt used in the executed program) Fig. 3 is a diagram showing an example of a prompt used in the execution program according to the embodiment 1. Input and output between the execution program 20 and the LLM 40 is performed using prompts, as shown in Fig. 3. Fig. 3 shows a prompt that indicates the first input from the execution program 20 to the LLM.

[0056] This prompt includes available tools t0, an explanation of the rules of the conversation, and a problem to be solved. In this example, available tools t0 include "show_program_file_list," "show_overview_of_program_file," and "get_related_code."

[0057] "show_program_file_list" is a file information acquisition tool and corresponds to the information acquisition unit 31. "show_overview_of_program_file" is a file overview creation agent and corresponds to the overview creation unit 32. "get_related_code" is a file related part excerpt agent and corresponds to the related part excerpt unit 33. Note that the available tools t0 are not limited to those shown in Figure 3, and may change for each question.

[0058] An explanation of the rules of the conversation and the problem to be solved are then input as explained in the reference example of input / output between the ReReAct execution program and the LLM in Fig. 11. After this prompt, the conversation continues between the execution program 20 and the LLM 40, as in the reference example of input / output between the ReReAct execution program and the LLM in Fig. 11.

[0059] The flow of the information acquisition process, summary creation process, and relevant part extraction process executed by the execution program 20 will now be described with reference to FIGS.

[0060] (Information acquisition process flow) 4 is a diagram illustrating an example of the flow of information acquisition processing according to the first embodiment. It is assumed that the LLM (A) receives a request for an answer to a question from the execution program 20. The question is assumed to mean "Please tell me why the output is different from the expected when the source code is executed." The prompt indicating the received request contains information on multiple executable tools, an explanation of the rules of the conversation, the question, and the like.

[0061] Then, the LLM(A) outputs an execution instruction to the execution program 20 to "show me the list of source code" (s1). In this case, the prompt indicating the output execution instruction contains information such as "show_program_file_list" as the tool and nothing as the argument.

[0062] Upon receiving the execution instruction, the execution program 20 obtains a list of the target source code 512 from the storage / DB 51. As an example, the path of the directory in which the files are saved is defined in advance in the storage unit 50. Therefore, the execution program 20 reads a list of files under the directory path of the target source code 512 from the storage / DB 51 (s2).

[0063] Then, the execution program 20 outputs a list of the source code 512 that has been read to the LLM(A).

[0064] (Outline creation process flow) 5 is a diagram illustrating an example of the flow of the outline creation process according to the embodiment 1. It is assumed that the LLM(A) receives a list of target source code 512 from the execution program 20.

[0065] Then, LLM(A) outputs an execution instruction to the execution program 20 to the effect that "the source code 'moduleA / main.py' seems to be related, so please give me an overview" (s3). In this case, the prompt indicating the output execution instruction describes "show_overview_of_program_file" as the tool and "moduleA / main.py" as the argument.

[0066] The execution program 20, which has received the execution instruction, reads the source code 512 called "moduleA / main.py" from the storage / DB 51 (s4). Then, the execution program 20 instructs the LLM (B) to execute the following for the read source code 512 (s5). The execution instruction here is "Please explain the classes and functions written in the source code 512 called "moduleA / main.py"."

[0067] Then, upon receiving the execution instruction, LLM(B) creates an overview of the classes and functions written in the source code 512 called "moduleA / main.py." LLM(B) then returns the created overview to the execution program 20. The execution program 20 then outputs the created overview to LLM(A).

[0068] (Flow of relevant section extraction process) 6 is a diagram illustrating an example of the flow of the relevant part extraction process according to the embodiment 1. It is assumed that the LLM(A) receives an outline of the source code 512 called “moduleA / main.py” from the execution program 20.

[0069] Then, the LLM(A) outputs to the execution program 20 an execution instruction meaning "Extract the part of the source code 'moduleA / main.py' that is executing the process of XX" (s6). Here, the prompt indicating the output execution instruction describes "get_related_code" as the tool and "'moduleA / main.py', 'the part that is executing the process of XX'" as the arguments.

[0070] Upon receiving the execution instruction, the execution program 20 reads the source code 512 called "moduleA / main.py" from the storage / DB 51 (s7). Then, the execution program 20 instructs the LLM (C) to execute the following for the read source code 512. The execution instruction here is "Extract the part of the Source Code that matches the contents written in the Information." The Source here is "moduleA / main.py." The Information here is "the part where the XX process is being executed."

[0071] Then, upon receiving the execution instruction, LLM(C) extracts "the part where XX processing is being executed" from "moduleA / main.py". Then, LLM(C) returns the extracted part to the execution program 20. Then, the execution program 20 outputs the extracted part to LLM(A).

[0072] (Flowchart of execution program) Here, a flowchart of the execution program 20 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a flowchart of the execution program according to the first embodiment.

[0073] As shown in FIG. 7, the execution program 20 determines whether or not a question has been received from a user (step S11). If it is determined that a question has not been received (step S11; No), the execution program 20 repeats the determination process until a question is received. On the other hand, if it is determined that a question has been received (step S11; Yes), the execution program 20 presents the question and available tools to the LLM 40 (step S12). Here, the question is assumed to be, for example, a question meaning "Please tell me the cause of the bug in the source code." Available tools are assumed to be, for example, a file information acquisition tool, a file summary creation agent, and a file-related part excerpt agent.

[0074] Then, the execution program 20 executes file information acquisition processing (step S13). For example, when the execution program 20 receives an instruction from the LLM 40 to acquire a list of target files (source code) using a file information acquisition tool, the execution program 20 acquires the list of target files (source code 512) from the storage / DB 51 using the file information acquisition tool. Then, the execution program 20 outputs the acquired list of target files (source code 512) to the LLM 40.

[0075] The execution program 20 then executes the file summary creation process (step S14). For example, when the execution program 20 receives an instruction from the LLM 40 to create a summary of the target file using the file summary creation agent, the execution program 20 reads the target file from the storage / DB 51 and causes the LLM 52 to create a summary of the read target file. The execution program 20 then outputs the created summary of the target file to the LLM 40.

[0076] The execution program 20 then executes the file related portion extraction process (step S15). For example, when the execution program 20 receives an instruction from the LLM 40 to extract related portions from a target file using the file related portion extraction agent, the execution program 20 reads the target file from the storage / DB 51 and causes the LLM 53 to extract the related portions from the read target file. The execution program 20 then outputs the extracted related portions to the LLM 40.

[0077] Then, the execution program 20 identifies the answer (step S16). For example, the execution program 20 receives the answer from the LLM 40 and identifies the location of the bug in the source code 512. Then, the execution program 20 ends.

[0078] In the first embodiment, the execution program 20 collects information about the source code 512 and identifies the location of a bug in the source code 512. However, the execution program 20 is not limited to this. It may also collect information about files other than the source code 512, such as the development document 511, and identify inconsistencies between the source code 512 and the development document 511. In such a case, for example, the question may be, "Please tell me the reason why the output when executing the source code is different from the expected one in relation to the development document." The execution program 20 then outputs a question including information about multiple tools to the LLM 40 and, based on an instruction from the LLM 40, causes the corresponding tool to execute the instruction. As an example, an existing search tool is added to the tool group 30. When the execution program 20 receives an instruction from the LLM 40 to search the target development document 511, it causes the search tool to read the target development document 511 and outputs the execution result to the LLM 40. Thereafter, as shown in FIGS. 4 to 6, the execution program 20 uses a tool to collect information about the source code 512 in response to instructions from the LLM 40, and outputs the collected results to the LLM 40. The execution program 20 then repeatedly exchanges information with the LLM 40 until the LLM 40 outputs an answer, and identifies an answer to the question. For example, the identified answer might be, "The XX document in the development documentation says XX, but the source code says △△."

[0079] Furthermore, each of the steps of the file information acquisition process (step S13), the file summary creation process (step S14), and the file relevant portion extraction process (step S15) may be performed multiple times in accordance with instructions from the LLM 40, and the order of each process may be changed. For example, after the file information acquisition process (step S13) is performed once, the file summary creation process (step S14) and the file relevant portion extraction process (step S15) may be performed for a certain file, and then the file summary creation process (step S14) and the file relevant portion extraction process (step S15) may be performed for another file, after which the answer is identified (step S16).

[0080] Furthermore, when presenting the question and available tools to the LLM 40 (step S12), when executing the file summary creation process (step S14), or when executing the file relevant portion excerpt process (step S15), the execution program 20 may include information about persona settings in the information exchanged with each LLM for the LLMs 40, 52, and 53 in FIG. 2. One example of the information about persona settings is, "You are a software development expert." Such personas may be the same for the LLMs 40, 52, and 53, or different personas may be set for each depending on the content of the question and the processing content of the available tools. This allows the LLMs 40, 52, and 53 to interpret the question and the execution content of the tools more specifically and professionally, thereby improving the accuracy of the tool execution results and the final answer to the question.

[0081] [Effects of Example 1] According to the first embodiment, in the information processing device 1, the execution program 20, which identifies an answer to a question, exchanges information with the LLM 40, which instructs the execution program 20 to execute one of multiple tools, including a tool for acquiring file information based on information output from the execution program 20, a tool for creating a file summary, and a tool for extracting relevant portions from a file. The information processing device 1 then identifies an answer, including a portion that may affect the file failure, by repeatedly exchanging information with the LLM 40 until the LLM 40 causes the execution program 20 to execute the tool and outputs an answer using information obtained by the execution program 20. The information exchange process then involves causing a tool corresponding to the instruction to execute the instruction based on an instruction from the LLM 40, and outputting information indicating the result of the executed instruction to the LLM 40. As a result, the information processing device 1 can accurately identify a portion that may affect the source program failure by using multiple tools, including a tool for acquiring file information, a tool for creating a file summary, and a tool for extracting relevant portions from a file. In other words, the information processing device 1 can gradually narrow down the information needed to obtain an answer by using the tool for acquiring file information, the tool for creating a file summary, and the tool for extracting relevant portions from a file. As a result, the information processing device 1 can accurately identify the part of the source program that is affected by the fault.

[0082] Furthermore, according to the first embodiment, in the information processing device 1, the process of executing an instruction is performed by exchanging information between an LLM that is separate and independent from the LLM 40 and the tool. In this way, by using an LLM that is separate and independent from the LLM 40 as a tool, the information processing device 1 can prevent noise information from being input to the LLM 40 that performs the main cause analysis, and prevent a decrease in the accuracy of the answer.

[0083] Furthermore, according to the first embodiment, the information processing device 1 executes instructions by including a persona in accordance with the content of the information exchanged in at least one of the exchange of information with the LLM 40 and the exchange of information with the LLM 52 and LLM 53. This enables the information processing device 1 to improve the accuracy of the tool execution results and the accuracy of the answers to questions. [Example]

[0084] The execution program 20 according to the first embodiment provides relevant information and answers the cause of a problem while having the LLM 40 consider the order in which the file information acquisition tool, file summary creation agent, and file related part extraction agent should be executed and which files to focus on. In other words, the LLM 40 collects information while considering the order in which to collect the information necessary to obtain the answer, and uses the collected information to generate and output the answer. However, the order in which the information is collected may affect the accuracy of the answer.

[0085] Therefore, in the following second embodiment, an execution program 20 will be described which provides a plurality of steps, repeats the exchange of information for each of the steps, and identifies an answer from answer candidates output for each of the steps.

[0086] FIG. 8 is a diagram illustrating an example of a functional configuration of an information processing device according to a second embodiment. Regarding the functional configuration of the information processing device 1 according to the second embodiment, the same components as those of the information processing device 1 illustrated in FIG. 2 are denoted by the same reference numerals, and redundant descriptions of the configuration and operation will be omitted. The first embodiment differs from the second embodiment in that the LLM 40 is configured as a plurality of LLMs, namely, LLM 40A and LLM 40B. The number of LLMs is not limited to two, and may be three or more. Although the LLMs 40A and 40B may be the same as the LLM 40, they are independent of the LLM 40. The first embodiment differs from the second embodiment in that the conversation processing unit 21 and the answer output unit 22 of the execution program 20 are changed to a conversation processing unit 21A and an answer output unit 22A, respectively. The first embodiment differs from the second embodiment in that a result integration unit 23A is added to the execution program 20. Furthermore, like LLM 52 and LLM 53, LLM 40A and LLM 40B may be configured by other information devices (not shown) that are different from information processing device 1. Note that LLM 40A and LLM 40B may also be written as LLM(A1) and LLM(A2), respectively.

[0087] The conversation processing unit 21A processes conversations with the LLM 40A and conversations with the LLM 40B.

[0088] For example, when the conversation processor 21A receives a question from a user, it generates multiple processing procedures based on the content of the question. The procedures here refer to, for example, the order in which information is collected, such as "collect development documents 511 first, then collect source code 512," or "collect source code 512 first, then collect development documents 511." Another example refers to the order in which tools are executed, such as "execute the first tool first, then execute the second tool," or "execute the second tool first, then execute the first tool." The conversation processor 21A then outputs a request for an answer to the question, including information on multiple executable tools and one of the procedures, to the LLM 40A. Additionally, the conversation processor 21A outputs a request for an answer to the question, including information on multiple executable tools and the other of the procedures, to the LLM 40B.

[0089] The conversation processor 21A also receives an instruction to execute a tool from the LLM 40A. Then, based on the instruction from the LLM 40A, the conversation processor 21A causes the tool corresponding to the instruction to execute the instruction. In addition, the conversation processor 21A receives an instruction to execute a tool from the LLM 40B. Then, based on the instruction from the LLM 40B, the conversation processor 21A causes the tool corresponding to the instruction to execute the instruction. Note that the conversation processor 21A may process conversations for the LLM 40A and the LLM 40B in series or in parallel.

[0090] The conversation processing unit 21A then repeats the exchange of information with the LLM 40A until the LLM 40A outputs a response. Additionally, the conversation processing unit 21A repeats the exchange of information with the LLM 40B until the LLM 40B outputs a response.

[0091] The result integration unit 23A integrates the answer results. For example, the result integration unit 23A receives the answer results from the LLM 40A. The result integration unit 23A receives the answer results from the LLM 40B. Then, the result integration unit 23A integrates the answer results.

[0092] The answer output unit 22A outputs the answer to the user. For example, the answer output unit 22A identifies the integrated answer result as the answer as is. Then, the answer output unit 22A outputs the identified answer to the user. Note that, although it has been described that the answer output unit 22A identifies the integrated answer result as the answer as is, this is not limited thereto, and when there are three or more LLMs, for example, any of the answer results may be identified as the answer by a method such as majority vote.

[0093] [Effects of Example 2] According to the second embodiment, the LLM 40A and the LLM 40B perform a plurality of procedures with the order of the processes interchanged. In the information processing device 1, the process of identifying an answer involves repeating the exchange of information for each of the plurality of procedures, and identifying an answer from the candidate answers output for each of the plurality of procedures. This allows the information processing device 1 to further improve the accuracy of the answer to the question sentence.

[0094] The illustrated tool group 30 is described as being provided within the execution program 20. However, the tool group 30 may be provided within the control unit 10 outside the execution program 20, as long as it is linked to the execution program 20. The tool group 30 is merely an example and may include various tools. The tools that can be used may be changed for each question.

[0095] Furthermore, the illustrated LLM(A) 40, LLM(B) 52, and LLM(C) 53 may be existing LLMs or original LLMs. Furthermore, although the case where LLM(B) 52 and LLM(C) 53 are stored in the storage unit 50 has been described, they may also be LLMs on the web or on other information processing devices.

[0096] Furthermore, the components of the illustrated information processing device 1 do not necessarily have to be physically configured as shown. That is, the specific manner in which the information processing device 1 is distributed and integrated is not limited to that shown, and all or part of the information processing device 1 can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, the storage unit 50 may be connected to the information processing device 1 as an external device via a network.

[0097] The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Therefore, an example of a computer that executes a specific program that realizes the same functions as the information processing device 1 shown in Fig. 2 will be described below. Here, the specific program that realizes the same functions as the information processing device 1 will be described as an example. Fig. 9 is a diagram showing an example of a computer that executes a specific program.

[0098] 9, computer 200 includes a CPU (Central Processing Unit) 203 that executes various types of arithmetic processing, an input device 215 that accepts data input from a user, and a display device 209. Computer 200 also includes a drive device 213 that reads programs and the like from a storage medium, and a communication I / F (Interface) 217 ​​that exchanges data with other computers via a network. Computer 200 also includes a memory 201 that temporarily stores various types of information, and an HDD (Hard Disk Drive) 205. Memory 201, CPU 203, HDD 205, display control unit 207, display device 209, drive device 213, input device 215, and communication I / F 217 are connected via a bus 219.

[0099] The drive device 213 is, for example, a device for the removable disk 211. The HDD 205 stores a specific program 205a and specific processing related information 205b. The communication I / F 217 manages the interface between the network and the inside of the device, and controls the input and output of data from other computers. The communication I / F 217 can be, for example, a modem or a LAN adapter.

[0100] The display device 209 is a display device that displays a cursor, an icon, a toolbox, and data such as documents, images, and function information. The display device 209 can be, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0101] The CPU 203 reads the specific program 205a, expands it in the memory 201, and executes it as a process. Such a process corresponds to each functional unit of the information processing device 1. The specific processing related information 205b includes, for example, the storage / DB 51. For example, the removable disk 211 stores each piece of information such as the specific program 205a.

[0102] It should be noted that the specific program 205a does not necessarily have to be stored in the HDD 205 from the beginning. For example, the program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD disk, magneto-optical disk, or IC card that is inserted into the computer 200. The computer 200 may then read and execute the specific program 205a from the medium. [Explanation of symbols]

[0103] 1. Information processing equipment 10 Control Unit 20 Executable Programs 21 Conversation processing unit 22 Answer output section 30 Tools 31 Information Acquisition Department 32 Overview Creation Department 33 Relevant excerpts 40 LLM(A) 50 Storage section 51 Storage / DB 52 LLM(B) 53 LLM(C) 511 Development Documents 512 Source Code

Claims

1. The program that identifies the answer to a question is exchanging information with a first learning model that instructs the program to execute a tool from among a plurality of tools, including a tool for acquiring file information based on information output from the program, a tool for creating a file summary, and a tool for extracting relevant portions of the file; and identifying the answer including a portion that affects the failure of the file by repeatedly exchanging information with the first learning model until the first learning model outputs the answer using information obtained by causing the program to execute the tool; The process of exchanging information includes: Based on an instruction of the first learning model, causing a tool corresponding to the instruction to execute the instruction; outputting information indicating a result corresponding to the executed instruction to the first learning model; A method for identifying a target object, characterized in that the processing is carried out by a computer.

2. The process of executing the instruction is performed by exchanging information between the tool and a second learning model that is separate and independent from the first learning model. The method according to claim 1 .

3. The first learning model performs a plurality of steps in which the order of processing is changed; The process of identifying the answer includes repeating exchange of information for each of the plurality of steps, and identifying the answer from the answer candidates output for each of the plurality of steps. The method according to claim 1 .

4. The file is Contains the source code of the software or documentation relating to said source code, The method according to claim 1 .

5. the process of executing the instruction includes processing that includes a persona according to the content of the information exchanged in at least one of the exchange of information with the first learning model and the exchange of information with the second learning model; The method of claim 2, wherein:

6. The persona includes the first learning model and / or the second learning model being a software development expert; The method according to claim 5 .

7. A program for identifying an answer to a question, exchanging information with a first learning model that instructs the program to execute a tool from among a plurality of tools, including a tool for acquiring file information based on information output from the program, a tool for creating a file summary, and a tool for extracting relevant portions of the file; and identifying the answer including a portion that affects the failure of the file by repeatedly exchanging information with the first learning model until the first learning model outputs the answer using information obtained by causing the program to execute the tool; The process of exchanging information includes: Based on an instruction of the first learning model, causing a tool corresponding to the instruction to execute the instruction; outputting information indicating a result corresponding to the executed instruction to the first learning model; A specific program that causes a computer to execute a process.

8. a first learning model that instructs the program to execute a tool from among a plurality of tools including a tool for acquiring file information based on information output from a program that identifies an answer to a question, a tool for creating a file summary, and a tool for extracting relevant portions of a file; and an input / output unit that exchanges information; an identification unit that identifies the answer including a part that affects the failure of the file by repeating information exchange with the first learning model until the first learning model outputs the answer using information obtained by causing the program to execute the tool, The input / output unit Based on an instruction of the first learning model, causing a tool corresponding to the instruction to execute the instruction; outputting information indicating a result corresponding to the executed instruction to the first learning model; 1. An information processing device comprising:

Images