Program, information processing method, and information processing apparatus

Abstract

To provide a program, and the like, configured to support system development appropriately.SOLUTION: A program causes a computer to execute the processes of: obtaining first deliverables in a first step of system development and second deliverables in a second step which is different from the first step; generating substitutes for the second deliverables by inputting the first deliverables and a prompt to generate the substitutes for the second deliverables from the first deliverables to a language model; determining whether a difference between the second deliverables and the substitutes falls within a predetermined range; correcting, when the difference does not fall within the predetermined range, the second deliverables or the substitutes using the language model; and repeating the correction of the second deliverables or the substitutes until the difference between the second deliverables and the substitutes falls within the predetermined range or a specified number of times is reached.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] There are technologies for supporting system development. For example, Patent Document 1 discloses a system development support device that detects, from attribute information of deliverables created in each process of system development, whether any of the deliverables has been changed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6482998 Summary of the Invention [Problem to be solved by the invention]

[0004] In one aspect, an object is to provide a program or the like that can favorably support system development. [Means for solving the problem]

[0005] In one aspect, the program causes a computer to execute a process of acquiring a first deliverable from a first stage of system development and a second deliverable from a second stage different from the first stage, generating the substitute by inputting the first deliverable and a prompt instructing the generation of a substitute for the second deliverable from the first deliverable into a language model, determining whether the difference between the second deliverable and the substitute is within a predetermined range, and if it is determined that the difference is not within the predetermined range, modifying the second deliverable or the substitute using the language model, and repeatedly modifying the second deliverable or the substitute until the difference between the second deliverable and the substitute falls within the predetermined range or a predetermined number of times is reached. [Effects of the Invention]

[0006] In one aspect, system development can be favorably supported. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a product generation evaluation system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of a server. [Figure 3] FIG. 1 is a diagram showing an overview of a first embodiment. [Figure 4] FIG. 1 illustrates a deliverable production task performed by an engineer agent. [Figure 5] FIG. 1 illustrates a deliverable evaluation task performed by a reviewer agent. [Figure 6] 10 is a flowchart illustrating an example of a processing procedure executed by a server. [Figure 7] 10 is a flowchart showing the procedure of a subroutine for generating a product. [Figure 8] 10 is a flowchart showing the procedure of a subroutine for product evaluation. [Figure 9] 10 is a flowchart showing the procedure of a subroutine for modifying a deliverable. [Figure 10] FIG. 10 is a diagram showing an outline of a modified example. [Figure 11] FIG. 1 illustrates a deliverable analysis task performed by a reverse engineer agent. [Figure 12] FIG. 10 is a diagram showing an overview of a second modified example. [Figure 13] FIG. 10 is a diagram showing an overview of a second embodiment. [Figure 14] FIG. 10 is a diagram showing an overview of a second embodiment. [Figure 15] 10 is a flowchart illustrating an example of a processing procedure executed by a server according to the second embodiment. [Figure 16] FIG. 10 is a diagram showing an overview of a third embodiment. [Figure 17] FIG. 10 is a diagram showing an overview of a third embodiment. [Figure 18]11 is a flowchart illustrating an example of a processing procedure executed by a server according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] The present invention will be described in detail below with reference to the drawings showing embodiments thereof. (Embodiment 1) Fig. 1 is an explanatory diagram showing an example of the configuration of a deliverable generation evaluation system. In this embodiment, a deliverable generation evaluation system that uses an LLM (Large Language Model) 50 to generate and evaluate deliverables in each process of system development will be described. The deliverable generation evaluation system includes an information processing device 1, a terminal 2, and a generation server 3. Each device is communicatively connected via a network N such as the Internet.

[0009] The information processing device 1 is an information processing device capable of various information processing and transmitting and receiving information, such as a server computer or a personal computer. In this embodiment, the information processing device 1 is assumed to be a server computer, and for simplicity, will be referred to as server 1 below. The server 1 uses the LLM 50 to generate a deliverable (second deliverable) in a downstream process (second process) from a deliverable (first deliverable) in an upstream process (first process) of system development, and also performs processing to evaluate the generated deliverable.

[0010] The terminal 2 is an information processing terminal used by a system developer, such as a personal computer. The server 1 receives a request from the terminal 2 (client) and generates and evaluates a deliverable.

[0011] The generation server 3 is a server computer that generates a response sentence to an input sentence using the LLM 50 in response to an external request. The LLM50 generates and evaluates the deliverables by requesting the generation server 3 to generate and evaluate the deliverables through the LLM50 Programmable Interface.

[0012] LLM50 is a machine learning model that has been trained on a huge amount of training data and generates a response sentence when an input sentence is received. Examples of LLM50 include GPT (Generative Pretrained Transformer) and BERT (Bidirectional Encoder Representations from Transformers), but the model is not particularly limited.

[0013] 2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit . The control unit 11 has one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc., and performs various information processing, control processing, etc. by reading and executing a program P stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside. The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores the program P (program product) and other data necessary for the control unit 11 to execute processing.

[0014] The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed by software.

[0015] Furthermore, in this embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input, a display unit that displays images, etc. Furthermore, the server 1 may be provided with a reading unit that reads a portable storage medium 1a such as a CD (Compact Disk)-ROM or a DVD (Digital Versatile Disc)-ROM, and may read and execute the program P from the portable storage medium 1a.

[0016] Fig. 3 is a diagram showing an overview of the first embodiment. Fig. 3 illustrates how, in a new development process of a system, an "engineer agent" generates a downstream process deliverable from an upstream process deliverable, a "reviewer agent" evaluates the generated downstream process deliverable, and feeds back the evaluation results to the "engineer agent." An overview of this embodiment will be described based on Fig. 3.

[0017] In this embodiment, the server 1 uses the LLM 50 to perform a process of generating and evaluating deliverables in each stage of system development in response to a request from the terminal 2 (client). To perform this process, the server 1 is provided with a software program hereinafter referred to as an "agent." In the following description, an "agent" that performs the task of generating a deliverable is referred to as an "engineer agent," and an "agent" that performs the task of evaluating a deliverable is referred to as a "reviewer agent."

[0018] The server 1 receives a request to generate a deliverable from the terminal 2. The request to generate a deliverable includes a deliverable in an upstream process from the deliverable to be generated. When the server 1 receives the request to generate a deliverable, it executes an engineer agent and executes a task to generate the deliverable in the downstream process.

[0019] Figure 4 shows the deliverable generation task executed by the engineer agent. When a deliverable generation request is received from a client (terminal 2), the engineer agent creates a prompt that instructs the LLM 50 to generate a deliverable in a downstream process. Specifically, the engineer agent creates the prompt by referencing a database (shown as a "prompt table" in Figure 4) that stores prompt templates. The prompt includes a standard phrase that instructs the generation of the deliverable, a template for the deliverable to be generated, and rules for generating the deliverable. The engineer agent selects a template from the database according to pre-registered settings and creates the prompt.

[0020] The engineer agent calls the API of the generation server 3 to generate the deliverable. In other words, the engineer agent sends the upstream process deliverable specified in the request and the prompt created above to the generation server 3, which inputs this data into the LLM 50 and generates the downstream process deliverable. For example, if the upstream process deliverable is a basic design document for a system, the downstream process deliverables will include a detailed design document and source code.

[0021] In this case, the engineer agent generates downstream process deliverables multiple times by inputting the same deliverables and prompts multiple times into the LLM 50 to evaluate the stability of the output from the LLM 50. The engineer agent then calculates an index value for evaluating the stability of the deliverable output based on the similarity between the multiple generated deliverables.

[0022] In the following description, for convenience, this index value will be referred to as the "first index value" to distinguish it from the "second index value" described below.

[0023] For example, when deliverables A, B, C, D, etc. are generated, the engineer agent randomly selects one of the deliverables and uses it as the comparison standard. For example, if deliverable A is selected as the standard, the engineer agent calculates the similarity (e.g., BLEU (Bilingual Evaluation Understudy) score) between that deliverable A and each of the other deliverables B, C, D, etc. The engineer agent then calculates the average value and standard deviation of the similarity as the first index value. For example, when calculating the BLEU score as similarity, the closer the average value is to 1, the more likely the LLM50 output is to be the same each time, and the closer the standard deviation is to 0, the more likely the LLM50 output is to be the same each time.

[0024] In this way, the engineer agent calculates a first index value for evaluating the stability of the output of the deliverable by the LLM 50. The engineer agent stores the generated deliverable and the first index value in a database (shown as a "feedback table" in FIG. 4). The engineer agent also returns the deliverable generated by the LLM 50 and the first index value to the client.

[0025] Returning to FIG. 3, the explanation continues. As described above, the server 1 generates a downstream process deliverable by inputting the upstream process deliverable and a prompt instructing the generation of the deliverable into the LLM 50. Next, the server 1 receives a deliverable evaluation request from the terminal 2. The evaluation request includes the deliverable to be evaluated. When the evaluation request is received, the server 1 executes a reviewer agent and performs a task of evaluating the deliverable.

[0026] FIG. 5 is a diagram showing a deliverable evaluation task performed by a reviewer agent. When a request to evaluate a deliverable is received, the reviewer agent creates a prompt that instructs the LLM 50 to evaluate the deliverable. Specifically, the reviewer agent creates the prompt by referencing a database that stores prompt templates. The prompt includes a standard phrase that instructs the evaluation of the product, a list of evaluation items that will be used as criteria for evaluating the deliverable, and the like. The reviewer agent selects a template from the database according to pre-registered settings and creates the prompt.

[0027] The reviewer agent calls the API of the generation server 3 to evaluate the deliverable. That is, the reviewer agent inputs the deliverable to be evaluated specified in the request and the prompt created above into the LLM 50, and obtains the evaluation results of the deliverable for each evaluation item. For example, the reviewer agent obtains, as the evaluation results, list data showing, for each evaluation item, a score (ranging from 1 to 10) indicating whether the deliverable is good or bad, and comments indicating items that need to be corrected (for example, if the evaluation item is a bug check, text explaining how to fix the bug).

[0028] In this case, the reviewer agent obtains the evaluation results of the deliverable multiple times by inputting the same deliverable and prompt multiple times into the LLM 50 in order to evaluate the stability of the evaluation result output by the LLM 50 and the quality of the deliverable. Then, the reviewer agent calculates an index value for evaluating the stability of the evaluation result output and the quality of the deliverable based on the multiple evaluation results obtained.

[0029] In the following description, for convenience, this index value will be referred to as the "second index value" to distinguish it from the above-mentioned "first index value."

[0030] For example, the reviewer agent attempts to evaluate a deliverable N times and calculates the average score, minimum score, standard deviation, etc. for each evaluation item as second index values. By using this second index value (for example, determining whether the average score and minimum score are above a threshold), it is possible to evaluate whether the quality of the deliverable is above a certain level. Furthermore, the stability of the output of the evaluation result can be evaluated depending on the level of the second index value (for example, standard deviation).

[0031] In this way, the reviewer agent calculates a second index value for evaluating the stability of the evaluation result output or the quality of the deliverable. The reviewer agent stores the evaluation result and the second index value obtained from the LLM 50 in a database. The reviewer agent also returns the evaluation result and the second index value obtained from the LLM 50 to the client.

[0032] As shown in FIG. 5, when the reviewer agent causes the LLM 50 to evaluate a deliverable, the reviewer agent may also generate a proposed revision for the deliverable at the same time. For example, the reviewer agent inputs a prompt to the LLM 50 that not only evaluates the deliverable but also instructs the LLM 50 to revise the deliverable based on the evaluation results. This allows the reviewer agent to obtain the deliverable revision proposal along with the deliverable evaluation results. In this case, the server 1 does not need to use an engineer agent to revise the deliverable (generate a proposed revision) as described below, and the reviewer agent alone can repeatedly evaluate the deliverable and generate a proposed revision.

[0033] Returning to FIG. 3, the explanation will continue. As described above, the server 1 evaluates a deliverable by inputting the deliverable to be evaluated (the deliverable of the downstream process) and a prompt instructing the evaluation of the deliverable into the LLM 50. After evaluating the deliverable, the server 1 determines whether or not a predetermined termination condition is met. If the termination condition is met, the server 1 ends the series of processes.

[0034] The termination conditions are whether or not the number of times deliverables have been generated and evaluated reaches a specified number of times, and whether or not the first index value and / or the second index value satisfy a predetermined termination condition (threshold condition). When the number of times deliverables have been generated and evaluated reaches the specified number of times, the server 1 terminates the series of processes. Also, when the first index value satisfies the termination condition (for example, when the average value of the similarities between deliverables is used as the first index value and the average value is equal to or greater than a threshold), the server 1 terminates the series of processes. Also, when the second index value satisfies the termination condition (for example, when the average value of the scores in the evaluation results is used as the second index value and the average value is equal to or greater than a threshold), the server 1 terminates the series of processes.

[0035] In this way, when the number of times the deliverables have been generated and evaluated reaches a specified number, or when sufficient stability or quality has been achieved, the server 1 terminates the processing and finally outputs the deliverables generated by the LLM 50 to the terminal 2 as the final deliverables.

[0036] On the other hand, if the termination condition is not met, the server 1 feeds back the evaluation result of the deliverable by the LLM 50 to the engineer agent, and corrects the deliverable.

[0037] Specifically, the engineer agent generates a proposed revision for the deliverable by inputting the deliverable to be evaluated and a prompt instructing the revision of the deliverable into the LLM 50. The prompt includes the evaluation result of the deliverable obtained from the LLM 50 in addition to a standard phrase instructing the revision of the deliverable.

[0038] As described above, the evaluation results output from the LLM 50 include, in addition to the score for each evaluation item, comments for each evaluation item indicating how to correct the item and what should be corrected. In other words, if the score for each evaluation item is low, a comment explaining how to correct the item can be obtained. The engineer agent can appropriately correct the deliverable by inputting a prompt including the evaluation result (comments explaining how to correct the item) into the LLM 50.

[0039] In this way, the engineer agent generates a revision proposal for the deliverable by inputting the deliverable and the prompt into the LLM 50. Note that, similar to the above, the engineer agent generates revision proposals multiple times by inputting the same deliverable and prompt into the LLM 50 multiple times, and calculates a first index value based on the similarity between the multiple generated deliverables.

[0040] When a proposed revision to the deliverable is generated, the server 1 executes the reviewer agent again to evaluate the proposed revision. That is, the reviewer agent obtains an evaluation result of the proposed revision by inputting the proposed revision to the deliverable and a prompt instructing the evaluation of the proposed revision into the LLM 50. Note that, similar to the above, the reviewer agent obtains multiple evaluation results by inputting the same proposed revision and prompt into the LLM 50 multiple times, and calculates a second index value for evaluating the stability of the output of the evaluation result and the quality of the proposed revision.

[0041] The server 1 repeatedly generates (modifies) and evaluates the deliverables until the termination condition is met. That is, the server 1 repeatedly generates the revision proposals until the number of deliverable generation and evaluations reaches a predetermined number, or the first index value and / or the second index value meets the termination condition. In this way, the server 1 generates the final deliverables.

[0042] As described above, according to this embodiment, the generation and evaluation of the deliverables are repeatedly executed, thereby making it possible to obtain a suitable deliverable semi-automatically.

[0043] 6 is a flowchart showing an example of a processing procedure executed by the server 1. The processing contents executed by the server 1 will be described with reference to FIG. When the control unit 11 of the server 1 receives a request to generate a deliverable from the terminal 2, the control unit 11 executes a subroutine to generate a deliverable (second deliverable) of a downstream process (second process) from a deliverable (first deliverable) of an upstream process (first process) (step S11). Specifically, the control unit 11 generates a deliverable of the downstream process by inputting the deliverable of the upstream process and a prompt instructing the generation of a deliverable of the downstream process from the deliverable to the LLM 50.

[0044] When the control unit 11 receives a request to evaluate a deliverable from the terminal 2, it executes the subroutine for evaluating the deliverable of the downstream process generated in step S11 (step S12). Specifically, the control unit 11 inputs the deliverable of the downstream process and a prompt instructing the evaluation of the deliverable into the LLM 50, thereby obtaining the evaluation result of the deliverable.

[0045] As described above, the control unit 11 may evaluate and modify the deliverable by inputting the deliverable and a prompt for evaluation and modification of the deliverable into the LLM 50. In this case, the process of step S12 may be repeatedly executed until the termination condition is satisfied, without executing the process of step S14 described below.

[0046] The control unit 11 determines whether a predetermined termination condition is satisfied (step S13). Specifically, in steps S11 and S12, the control unit 11 calculates an index value for evaluating the stability of the output of the deliverable or the quality of the deliverable, and determines whether the index value satisfies the termination condition. Alternatively, the control unit 11 determines whether the number of times the deliverable has been generated and evaluated has reached a specified number. If it is determined that the termination condition is satisfied (S13: YES), the control unit 11 terminates the series of processes.

[0047] If it is determined that the termination condition is not satisfied (S13: NO), the control unit 11 executes a subroutine to modify the deliverable generated in step S11 (step S14). Specifically, the control unit 11 generates a modification proposal for the deliverable by inputting the deliverable generated in step S11 and a prompt instructing modification of the deliverable, including the evaluation result acquired in step S12, into the LLM 50.

[0048] The control unit 11 returns the process to step S12. In this case, the control unit 11 evaluates the corrected deliverable again (step S12) and determines whether the termination condition is met (step S13). If it is determined that the termination condition is not met (S13: NO), the control unit 11 corrects the deliverable again (step S14). In this way, the control unit 11 repeatedly generates (corrects) and evaluates the deliverable.

[0049] 7 is a flowchart showing the steps of the subroutine for generating a product. The processing content of the subroutine in step S11 will be described with reference to FIG. The control unit 11 of the server 1 references a database that stores prompt templates and creates a prompt that instructs the LLM 50 to generate a deliverable (step S31). The prompt includes a fixed phrase that instructs the generation of a deliverable, a template for the deliverable to be generated, a generation rule for the deliverable, etc.

[0050] The control unit 11 generates a downstream process deliverable by inputting the upstream process deliverable and the prompt created in step S31 into the LLM 50 (step S32). Specifically, the control unit 11 generates multiple deliverables by inputting the same deliverable and prompt into the LLM 50 multiple times.

[0051] The control unit 11 calculates the similarities between the multiple generated deliverables, and calculates a first index value for evaluating the stability of the output of the deliverables by taking the average value, standard deviation, etc. of the calculated similarities (step S33). The control unit 11 outputs the deliverables generated in step S32, the first index value calculated in step S33, etc. to the terminal 2 (step S34), and returns the subroutine.

[0052] 8 is a flowchart showing the procedure of the subroutine for product evaluation. The processing content of the subroutine in step S12 will be described with reference to FIG. The control unit 11 of the server 1 refers to a database that stores prompt templates and creates a prompt that instructs the LLM 50 to evaluate the deliverables of the downstream process (step S51). The prompt includes a standard phrase that instructs the evaluation of the deliverables, a list of evaluation items that are used as criteria for evaluating the deliverables, etc.

[0053] The control unit 11 inputs the deliverable of the downstream process and the prompt created in step S51 into the LLM 50, thereby obtaining an evaluation result of the deliverable of the downstream process (step S52). Specifically, the control unit 11 inputs the same deliverable and prompt into the LLM 50 multiple times, thereby obtaining multiple evaluation results.

[0054] The control unit 11 calculates a second index value for evaluating the stability of the output of the deliverable and the quality of the deliverable by taking the average score, minimum score, standard deviation, etc. of each evaluation item in the multiple evaluation results obtained (step S53). The control unit 11 outputs the evaluation results obtained in step S52, the second index value calculated in step S53, etc. to the terminal 2 (step S54), and returns the subroutine.

[0055] 9 is a flowchart showing the steps of the subroutine for correcting the deliverables. The processing content of the subroutine in step S14 will be described with reference to FIG. The control unit 11 of the server 1 refers to a database that stores prompt templates and creates a prompt that instructs the LLM 50 to modify the deliverable (step S71). The prompt includes a standard phrase that instructs the LLM 50 to modify the deliverable, the evaluation result of the deliverable obtained in step S12, etc.

[0056] The control unit 11 generates a revision proposal for the deliverable by inputting the deliverable of the downstream process and the prompt created in step S71 into the LLM 50 (step S72). As in step S32, the control unit 11 generates multiple revision proposals by inputting the same deliverable and prompt into the LLM 50 multiple times.

[0057] The control unit 11 calculates the similarities between the generated revision proposals, and calculates the average value, standard deviation, etc. of the calculated similarities to calculate a first index value for evaluating the stability of the output of the deliverable (revision proposal) (step S73). The control unit 11 outputs the revision proposals generated in step S72, the first index value calculated in step S73, etc. to the terminal 2 (step S74), and returns the subroutine.

[0058] As described above, according to the first embodiment, system development can be favorably supported.

[0059] (Variation 1) In the first embodiment, a new system development process was described in which a downstream process deliverable is created from an upstream process deliverable. On the other hand, this embodiment may also be applied to reverse engineering, which is analyzing an upstream process deliverable from a downstream process deliverable. In this modified example, a form in which this system is applied to a reverse engineering process for analyzing an existing implementation will be described.

[0060] Fig. 10 is a diagram showing an overview of a modified example. Fig. 10 illustrates a process in which a "reverse engineer agent" generates a deliverable (second deliverable) of an upstream process (second process) from a deliverable (first deliverable) of a downstream process (first process), a reviewer agent evaluates the generated deliverable, and the evaluation result is fed back to the "reverse engineer agent." This modified example will be described with reference to Fig. 10.

[0061] In this modified example, the server 1 is provided with a "reverse engineer agent" as an agent that performs processing using the LLM 50. The reverse engineer agent is a program that executes the task of generating an upstream process deliverable from a downstream process deliverable, i.e., the task of analyzing the deliverable. When a deliverable analysis request is received from the terminal 2 (client), the server 1 executes the reverse engineer agent and executes the task of generating an upstream process deliverable from a downstream process deliverable.

[0062] 11 is a diagram showing a deliverable analysis task executed by a reverse engineer agent. The reverse engineer agent receives a deliverable analysis request from a client. The analysis request includes the deliverable of the downstream process to be analyzed.

[0063] When an analysis request is received, the reverse engineer agent creates a prompt that instructs the LLM50 to generate an upstream process deliverable from a downstream process deliverable. Specifically, the reverse engineer agent creates the prompt by referencing a database that stores prompt templates. The prompt includes a standard phrase that instructs the generation of an upstream process deliverable, a template for the downstream process deliverable to be generated, etc.

[0064] The reverse engineer agent generates an upstream process deliverable by inputting the downstream process deliverable to be analyzed and the prompt created above into the LLM50. To evaluate the stability of the deliverable output by the LLM50, the reverse engineer agent generates the deliverable multiple times by inputting the same deliverable and prompt into the LLM50 multiple times. The reverse engineer agent calculates the similarity between the multiple generated deliverables, and calculates a first index value for evaluating the stability of the deliverable output by taking the average value, standard deviation, etc. of the calculated similarity. The reverse engineer agent returns the generated upstream process deliverable and the first index value to the client.

[0065] Returning to Figure 10, the explanation will continue. As described above, the server 1 generates an upstream process deliverable from a downstream process deliverable using a reverse engineer agent. Next, the server 1 receives an evaluation request for the generated upstream process deliverable from the terminal 2. When the server 1 receives the evaluation request for the deliverable, it executes a reviewer agent and performs a task of evaluating the upstream process deliverable.

[0066] Specifically, as described in FIG. 5, the reviewer agent creates a prompt that instructs the LLM 50 to evaluate the deliverable, and obtains an evaluation result for the deliverable by inputting the upstream deliverable to be evaluated and the prompt into the LLM 50. As in the first embodiment, the reviewer agent obtains evaluation results multiple times by inputting the same deliverable and prompt into the LLM 50 multiple times. Based on the multiple obtained evaluation results, the reviewer agent calculates a second index value for evaluating the stability of the evaluation result output and the quality of the deliverable. The reviewer agent returns the evaluation result and the second index value to the client.

[0067] As in the first embodiment, the reviewer agent may correct the deliverable at the same time as evaluating the deliverable.

[0068] When the server 1 obtains the evaluation result of the deliverable, it feeds back the evaluation result to the reverse engineer agent and causes the LLM 50 to modify the deliverable. That is, similar to the engineer agent described in the first embodiment, the reverse engineer agent generates a modification proposal for the deliverable by inputting the deliverable and a prompt instructing the modification of the deliverable into the LLM 50. Note that in this case, as in the above, the reverse engineer agent inputs the same deliverable and prompt into the LLM 50 multiple times to obtain multiple modification proposals, and calculates a first index value from the similarity of each modification proposal.

[0069] When a revision proposal is generated, the server 1 evaluates the revision proposal using a reviewer agent. The server 1 repeatedly performs the above-mentioned generation (revision) and evaluation of the deliverable until the first index value and / or the second index value satisfy the termination condition or reach a predetermined number of times. As a result, the final deliverable of the upstream process is generated.

[0070] The server 1 may input the upstream process deliverable generated by the reverse engineer agent into the engineer agent, thereby transitioning from the reverse engineering process to a new development process aimed at refactoring, specification changes, performance improvements, etc. That is, the engineer agent may generate a downstream process deliverable by inputting the upstream process deliverable generated by the reverse engineer agent and a prompt instructing the generation of a downstream process deliverable from the upstream process deliverable into the LLM 50. In this case, the server 1 repeatedly generates (modifies) and evaluates the downstream process deliverable, as in the first embodiment, to generate the final downstream process deliverable.

[0071] As described above, according to this modification, the system can be applied to a reverse engineering process in which an upstream process product is generated (analyzed) from a downstream process product.

[0072] Since the other points are the same as those in the first embodiment, detailed explanations of the flowchart and other details will be omitted in this modification.

[0073] (Variation 2) In this modified example, when generating a downstream process product (second product) from an upstream process product (first product), a form is described in which the line number or block code of the upstream process product is assigned to the downstream process product.

[0074] 12 is a diagram showing an overview of Modification 2. As in the first embodiment, the server 1 generates a downstream process deliverable (e.g., source code) from an upstream process deliverable (e.g., a design document) using the LLM 50. In this case, in this modification, the server 1 performs a process of assigning line numbers or block codes of the upstream process deliverable to each line or code block of the downstream process deliverable.

[0075] Specifically, the server 1 instructs the creation of a downstream process deliverable from an upstream process deliverable, and creates a prompt that instructs the downstream process deliverable to be assigned a line number or block code corresponding to the upstream process deliverable. The server 1 inputs the prompt and the upstream process deliverable into the LLM 50, thereby creating a downstream process deliverable to which the line number or block code of the upstream process deliverable has been assigned.

[0076] As an example, Figure 12 illustrates a process in which line numbers from an upstream process deliverable are assigned to each line of a downstream process deliverable, and the downstream process deliverable is evaluated. Specifically, the upper part of Figure 12 illustrates the upstream process deliverable (design document), the middle part of Figure 12 illustrates the downstream process deliverable (source code) to which line numbers from the upstream process deliverable have been assigned, and the lower part of Figure 12 illustrates the evaluation results of the downstream process deliverable. As shown in the upper and middle parts of Figure 12, the server 1 inputs a prompt to the LLM 50 instructing it to assign line numbers from the upstream process deliverable to the downstream process deliverable, thereby generating a downstream deliverable in which the line numbers of the corresponding upstream process deliverable are assigned to each line of the downstream process deliverable.

[0077] By assigning the line number of the upstream process deliverable to each line of the downstream process deliverable, the LLM 50 can more appropriately evaluate the downstream process deliverable. The evaluation results of the downstream process deliverable are shown in the lower part of Figure 12. The server 1 creates a prompt to instruct the evaluation of the downstream process deliverable and inputs it into the LLM 50 along with the downstream process deliverable to which the line number of the upstream process deliverable has been assigned, thereby obtaining the evaluation results of the downstream process deliverable from the LLM 50. For example, as shown in the lower part of Figure 12, the evaluation results are displayed for each line of the downstream process deliverable, indicating whether or not they match the corresponding line of the upstream process deliverable. By providing the evaluation results to the LLM 50 and having it make corrections, it is clear which line needs to be corrected, allowing the LLM 50 to easily make corrections.

[0078] (Embodiment 2) In this embodiment, we will explain a form in which a suitable final product can be obtained by generating a substitute for the downstream process product (second product) and repeatedly modifying the downstream process product and / or the substitute until the difference between the substitute and the downstream process product falls within a specified range.

[0079] 13 and 14 are diagrams showing an outline of the second embodiment. The outline of this embodiment will be described with reference to FIGS.

[0080] In this embodiment, the deliverable of the upstream process is a detailed design document, and the deliverable of the downstream process is source code. In this embodiment, an operation test of the source code, which is the deliverable of the downstream process, is performed, and its quality is evaluated using LLM50.

[0081] To ensure the quality of source code, it is important to conduct sufficient operational testing in addition to the quality of detailed design documents. Currently, testing is conducted with limited human and time resources, but the use of LLM50 can effectively support the testing process. However, directly evaluating test results using LLM50 can result in increased computational costs as the number of test cases increases. In addition, because LLM50 behaves probabilistically, evaluation results can be unstable, and it is difficult to clearly demonstrate the basis for the evaluation results.

[0082] Therefore, in this embodiment, LLM50 generates alternative source code with the same specifications as the source code to be evaluated, runs test cases under the same conditions as the original source code on the alternative source code, and compares the results and behavior. This makes it less subject to resource constraints, has high explainability and reproducibility, and can detect specification deficiencies and ambiguity.

[0083] The explanation will be started based on Fig. 13. First, the server 1 acquires an upstream process deliverable (detailed design document) and a downstream process deliverable (source code). The downstream process deliverable may be generated by the process described in the first embodiment or may be created manually.

[0084] In the following description, the above downstream process deliverables will be referred to as "original deliverables" to distinguish them from the "alternatives" described below.

[0085] The server 1 generates a substitute (source code with the same specifications) for the original deliverable by inputting the deliverable of the upstream process and a prompt that instructs the generation of a substitute for the deliverable of the downstream process from the deliverable of the upstream process into the LLM 50. The server 1 also generates test data to be executed on the original deliverable and the substitute by inputting the deliverable of the upstream process and a prompt that instructs the generation of test data into the LLM 50.

[0086] The server 1 generates a source code file for testing based on the original deliverable, the alternative, and the test data, and then performs an operation test of the original deliverable and the alternative, and executes a comparison test to compare the test results.

[0087] 14, the explanation continues. The server 1 analyzes the test results. If the test results of the original deliverable and the substitute match, or if the predetermined number of times is reached, the server 1 ends the series of processes.

[0088] If an execution error occurs in the original deliverable and / or the substitute, the server 1 generates a prompt that outputs the cause of the error and instructs the user to correct the original deliverable and / or the substitute where the error occurred. The server 1 inputs the original deliverable and / or the substitute where the error occurred and the generated prompt into the LLM 50, thereby generating the results of an investigation into the cause of the error and a corrected version of the original deliverable and / or the substitute.

[0089] The server 1 analyzes the cause of the error and ends the series of processes if the error is environment-dependent. In this case, the server 1 changes the environment settings and then executes the series of processes again.

[0090] If the cause of the error is due to the implementation of the source code, the server 1 re-runs the operation test of the corrected original deliverable and / or the alternative to confirm its operation. If the operation test has been performed a specified number of times, the server 1 ends the series of processes. If the error is not resolved, the server 1 investigates the cause of the error and corrects the deliverable again. If the error is resolved, the server 1 re-runs the comparison test of the original deliverable and the alternative.

[0091] If the test results of the original deliverable and the substitute do not match, the server 1 generates a prompt that outputs the deviation factors and instructs the user to modify the original deliverable and / or the substitute. The server 1 inputs the prompt and the original deliverable and / or the substitute into the LLM 50, thereby generating the results of the investigation into the deviation factors and a modified version of the original deliverable and / or the substitute. Specifically, the server 1 inputs the original deliverable, the substitute, as well as upstream process deliverables, test data, and the outputs of the original deliverable and the substitute into the LLM 50, thereby generating the results of the investigation into the deviation factors and a modified version of the original deliverable and / or the substitute.

[0092] The server 1 analyzes the cause of the deviation in the test results based on the investigation results. If the cause of the deviation is a defect in the upstream process deliverable (design document), the server 1 temporarily terminates the processing, corrects the upstream process deliverable, and then executes the series of processes again. If the cause of the deviation is a defect in the original deliverable and / or substitute, the server 1 executes an operational test of the corrected original deliverable and / or substitute, and compares the test results again.

[0093] In this way, if the test results of the original product and the substitute do not match, i.e., if the difference between the original product and the substitute is not within a predetermined range, the server 1 repeatedly performs corrections on the original product and / or the substitute until the difference falls within the predetermined range (or until a predetermined number of corrections is reached), thereby obtaining a more suitable final product.

[0094] 15 is a flowchart showing an example of a processing procedure executed by the server 1 according to Embodiment 2. The processing executed by the server 1 in this embodiment will be described with reference to FIG. The control unit 11 of the server 1 acquires a product (first product) of an upstream process and an original product (second product) of a downstream process (step S201). The original product may be generated by the process described in the first embodiment or may be created manually.

[0095] The control unit 11 generates a substitute for the downstream process deliverable from the upstream process deliverable using the LLM 50, and generates test data to be used for executing an operation test of the original deliverable and the substitute (step S202). The control unit 11 executes an operation test on the original deliverable and the substitute using the same test data (step S203).

[0096] The control unit 11 determines whether an execution error has occurred in the original deliverable and / or the substitute (step S204). If it determines that an execution error has occurred (S204: YES), the control unit 11 generates a prompt that instructs output of the error cause of the original deliverable and / or the substitute and modification of the original deliverable and / or the substitute (step S205). The control unit 11 inputs the prompt and the original deliverable and / or the substitute in which the error occurred to the LLM 50, thereby generating the results of an investigation into the cause of the error and a modified version of the original deliverable and / or the substitute (step S206). The control unit 11 returns the process to step S203.

[0097] If it is determined that no execution error has occurred (S204: NO), the control unit 11 determines whether the test results of the original deliverable and the substitute match (step S207). If it is determined that the test results do not match (S207: NO), the control unit 11 generates a prompt that outputs the cause of deviation in the test results and instructs the user to modify the original deliverable and / or the substitute (step S208). The control unit 11 inputs the prompt and the original deliverable and / or the substitute into the LLM 50, thereby generating the investigation results of the cause of deviation and modified versions of the original deliverable and / or the substitute (step S209). The control unit 11 returns the process to step S203.

[0098] If it is determined that the test results match (S207: YES), the control unit 11 ends the series of processes.

[0099] As described above, according to the second embodiment, a more preferable final product (second product) can be obtained.

[0100] (Embodiment 3) In the second embodiment, a configuration has been described in which an operation test is performed on the original deliverable and a substitute in a downstream process to check whether the deliverable is suitable. However, depending on the deliverable, dynamic checks may not be possible and only static checks may be possible. In this embodiment, a configuration in which static checks of the deliverable are performed will be described.

[0101] 16 and 17 are diagrams showing an outline of embodiment 3. The outline of this embodiment will be described with reference to FIGS.

[0102] In this embodiment, we will assume that the deliverables of the upstream process are basic design documents and the deliverables of the downstream process are detailed design documents. In this case, it is obviously not possible to perform operational tests on the deliverables, so in this embodiment, we compare the original deliverable with the substitute using a method different from that in the second embodiment.

[0103] The explanation will begin with reference to Fig. 16. First, the server 1 acquires a deliverable from an upstream process and an original deliverable from a downstream process. The original deliverable may be one generated by the process described in the first embodiment or one created manually.

[0104] By executing the structural information assignment program, the server 1 breaks down the deliverables of the upstream process into predetermined structural units and assigns structural information to each structural unit. The structural units are, for example, line numbers, hierarchical structures as documents (e.g., chapters, articles, paragraphs, etc.), table numbers, etc., but are not limited thereto. For example, the server 1 assigns structural information by pasting the deliverables of the upstream process broken down into structural units into a prompt and inputting them into the LLM 50. Note that the assignment of structural information may also be performed using a method that does not rely on the LLM 50.

[0105] The server 1 generates a substitute for the original deliverable by inputting the upstream process deliverable with the structure information and a prompt instructing the generation of a substitute for the downstream process deliverable from the upstream process deliverable to the LLM 50. In this case, a template for the downstream process deliverable may be provided to the LLM 50.

[0106] The server 1 first compares the upstream process deliverable with the downstream process deliverable (original deliverable and substitute) and determines whether they are consistent. The server 1 evaluates the consistency between the upstream process deliverable and the original deliverable by inputting into the LLM 50 the upstream process deliverable with structural information and the original downstream process deliverable, along with a prompt instructing the LLM 50 to evaluate whether the original deliverable is consistent with the upstream process deliverable. Specifically, as in the method described in the first embodiment, predetermined evaluation items are provided to the LLM 50, and an evaluation is performed for each evaluation item. However, the evaluation items in this embodiment include whether the portions of the original deliverable that reflect the description content of the upstream process deliverable are clear, whether there are any undescribed portions in the upstream process deliverable, whether there are any deficiencies in the description content of the upstream process deliverable, and whether the content reflected in the original deliverable is consistent with the upstream process deliverable.

[0107] Similarly, for substitutes, the server 1 evaluates the consistency between the upstream process deliverable and the substitute by inputting into the LLM 50 the upstream process deliverable to which structural information has been assigned, the substitute for the downstream process deliverable, and a prompt instructing the evaluation of whether the substitute is consistent with the upstream process deliverable.

[0108] Based on the evaluation results output from the LLM 50, the server 1 determines whether the original deliverable and the substitute are consistent with the deliverable from the upstream process (whether there is any excess or deficiency). If it determines that there is a mismatch due to a defect in the deliverable from the upstream process, the server 1 temporarily terminates the process and presents instructions to the user to correct the deliverable from the upstream process. After correcting the deliverable from the upstream process, the server 1 executes the series of processes again.

[0109] If it is determined that the original deliverable and / or substitute are inconsistent due to a defect in the original deliverable and / or substitute, the server 1 corrects the original deliverable and / or substitute using the LLM 50. That is, the server 1 creates a prompt that instructs correction of the original deliverable and / or substitute that is inconsistent with the deliverable of the upstream process, and inputs this into the LLM 50 along with the original deliverable and / or substitute to generate a corrected version of the original deliverable and / or substitute. The server 1 repeatedly corrects the original deliverable and / or substitute until the deliverable of the upstream process and the original deliverable and substitute are consistent.

[0110] The explanation continues with reference to FIG. 17. When the upstream process deliverable is consistent with the original deliverable and the substitute, the server 1 compares the original deliverable and the substitute. The server 1 evaluates the original deliverable and the substitute by inputting the upstream process deliverable, the downstream process original deliverable, and the substitute, to which structural information has been assigned, and prompts instructing the LLM 50 to evaluate the original deliverable and the substitute. Specifically, similar to the method described in the first embodiment, the server 1 causes the LLM 50 to perform scoring for each predetermined evaluation item. Furthermore, in this embodiment, the server 1 causes the LLM 50 to determine whether the lower-scoring deliverable should be corrected based on the difference in scores between the original deliverable and the substitute, and to output the correction details. The server 1 causes the server 1 to perform evaluations for each structural information from lowest to highest hierarchy, and when evaluating higher-level structural information, the server 1 also causes the server 1 to perform a self-evaluation of the evaluation of the next lower hierarchy and the correction details.

[0111] The server 1 compares the evaluation results of the original deliverable and the substitute. If the comparison results in the two being equivalent (i.e., the difference in the scores for each evaluation item is within a predetermined range), the server 1 outputs the original deliverable (or the substitute) as the final deliverable.

[0112] On the other hand, if the evaluation of the original deliverable is low compared to the alternative, the server 1 modifies the original deliverable using the LLM 50. For example, the server 1 creates a prompt to instruct modification of the original deliverable, and inputs the prompt into the LLM 50 together with the original deliverable and its evaluation results (contents to be modified), thereby generating a modified version of the original deliverable.

[0113] Similarly, if the substitute has a lower rating compared to the original deliverable, the server 1 modifies the substitute using the LLM 50.

[0114] When the original deliverable or the substitute is modified, the server 1 evaluates the modified original deliverable and the substitute again and compares them. The server 1 repeatedly modifies the original deliverable and / or the substitute until the difference in the evaluation results falls within a predetermined range. The server 1 keeps a history of modifications and stops processing if the same modifications are repeated.

[0115] In this way, by determining whether the upstream process deliverables are consistent with the original deliverables and substitutes, and then repeating the process of corrections until the original deliverables and substitutes are equivalent, a more suitable final deliverable can be obtained.

[0116] 18 is a flowchart showing an example of a processing procedure executed by the server 1 according to Embodiment 3. The processing executed by the server 1 in this embodiment will be described with reference to FIG. The control unit 11 of the server 1 acquires a product (first product) of an upstream process and an original product (second product) of a downstream process (step S301). The original product may be generated by the process described in the first embodiment or may be created manually.

[0117] The control unit 11 assigns structural information to the deliverable of the upstream process (step S302). The control unit 11 generates a substitute for the deliverable of the downstream process by inputting the deliverable of the upstream process to which the structural information has been assigned and a prompt instructing the LLM 50 to generate a substitute for the deliverable of the downstream process from the deliverable of the upstream process (step S303).

[0118] The control unit 11 evaluates the consistency between the upstream process deliverable and each of the original deliverable and substitute by inputting the original deliverable and substitute, and a prompt instructing the LLM 50 to evaluate whether the original deliverable and substitute are consistent with the upstream process deliverable (step S304).

[0119] Based on the evaluation results of step S304, the control unit 11 determines whether the original deliverable and the substitute are consistent with the deliverable of the upstream process (step S305). If it is determined that the original deliverable and / or the substitute are not consistent with the deliverable of the upstream process (S305: NO), the control unit 11 creates a prompt instructing the correction of the original deliverable and / or the substitute that are not consistent with the deliverable of the upstream process (step S306). The control unit 11 corrects the original deliverable and / or the substitute by inputting the prompt and the original deliverable and / or the substitute that are not consistent with the deliverable of the upstream process into the LLM 50 (step S307). The control unit 11 returns the process to step S304.

[0120] If it is determined that the original deliverable and the substitute are consistent with the deliverable of the upstream process (S305: YES), the control unit 11 evaluates the original deliverable and the substitute by inputting the original deliverable and the substitute and a prompt instructing the evaluation of the original deliverable and the substitute into the LLM 50 (step S308).

[0121] Based on the evaluation results of the original deliverable and the substitute, the control unit 11 determines whether the difference between the original deliverable and the substitute is within a predetermined range (step S309). If it determines that the difference is not within the predetermined range (S309: NO), the control unit 11 creates a prompt instructing the user to modify the original deliverable or the substitute (step S310). The control unit 11 modifies the original deliverable or the substitute by inputting the prompt and the original deliverable or the substitute into the LLM 50 (step S311). The control unit 11 returns the process to step S308.

[0122] If it is determined that the difference between the original product and the substitute is within a predetermined range (S309: YES), the control unit 11 ends the series of processes.

[0123] As described above, according to the third embodiment, a more preferable final product (second product) can be obtained.

[0124] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0125] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any combination, regardless of the reference format. Furthermore, although the claims use a format in which a claim references two or more other claims (multi-claim format), this is not limited to this format. A format in which multiple claims (multi-multi-claims) that reference at least one other multiple claim may also be used. [Explanation of symbols]

[0126] 1. Server (information processing device) 11 Control section 12 Main memory 13 Communications Department 14 Auxiliary storage P Program 2. Devices 3 Generation Server 50 LLM (Language Modeling)

Claims

1. Acquire a first deliverable in a first stage of system development and a second deliverable in a second stage different from the first stage; generating an alternative to the second deliverable by inputting the first deliverable and a prompt to generate the alternative to the second deliverable from the first deliverable into a language model; determining whether a difference between the second deliverable and the substitute is within a predetermined range; if it is determined that the difference is not within a predetermined range, modifying the second deliverable or substitute using the language model; The second deliverable or the substitute is repeatedly modified until the difference between the second deliverable and the substitute falls within a predetermined range or until a predetermined number of modifications is performed. A program that causes a computer to perform a process.

2. performing an operational test on the second deliverable and the substitute using the same test data; Determining whether the test results of the second product and the substitute match, thereby determining whether the difference between the second product and the substitute is within a predetermined range. The program according to claim 1.

3. evaluating the second deliverable and the alternative by inputting the second deliverable and the alternative and a prompt indicating an evaluation of the second deliverable and the alternative, respectively, into the language model; By comparing the evaluation results of the second product and the substitute, it is determined whether the difference between the second product and the substitute is within a predetermined range. The program according to claim 1.

4. evaluating the consistency between the first deliverable and the second deliverable or substitute by inputting the first deliverable, the second deliverable or substitute, and a prompt that instructs the language model to evaluate whether the second deliverable or substitute is consistent with the first deliverable; Iteratively modifying the second deliverable or the alternative using the language model until an evaluation result indicating consistency is obtained. The program according to claim 1.

5. Acquire a first deliverable in a first stage of system development and a second deliverable in a second stage different from the first stage; generating an alternative to the second deliverable by inputting the first deliverable and a prompt to generate the alternative to the second deliverable from the first deliverable into a language model; determining whether a difference between the second deliverable and the substitute is within a predetermined range; if it is determined that the difference is not within a predetermined range, modifying the second deliverable or substitute using the language model; The second deliverable or the substitute is repeatedly modified until the difference between the second deliverable and the substitute falls within a predetermined range or until a predetermined number of modifications is performed. An information processing method in which processing is performed by a computer.

6. An information processing device including a control unit, The control unit Acquire a first deliverable in a first stage of system development and a second deliverable in a second stage different from the first stage; generating an alternative to the second deliverable by inputting the first deliverable and a prompt to generate the alternative to the second deliverable from the first deliverable into a language model; determining whether a difference between the second deliverable and the substitute is within a predetermined range; if it is determined that the difference is not within a predetermined range, modifying the second deliverable or substitute using the language model; The second deliverable or the substitute is repeatedly modified until the difference between the second deliverable and the substitute falls within a predetermined range or until a predetermined number of modifications is performed. Information processing device.

Images