Program generation device and program generation method

Abstract

The present invention suppresses the inclusion of bugs in a program when generating the program using a generative AI model.　A program generation device 10 comprises: a program acquisition unit 11 that causes a generative AI model 21 to generate a program, and acquires the program; a program execution unit 12 that executes the program acquired by the program acquisition unit 11; and a regeneration determination unit 13 that determines whether to regenerate the program according to the execution result obtained by executing the program using the program execution unit 12. The program acquisition unit 11 causes the generative AI model 21 to regenerate the program according to the determination made by the regeneration determination unit 13, and acquires the regenerated program.

Description

Program generation device and program generation method

[0001] The present invention relates to a program generation device and a program generation method for generating a program.

[0002] Patent Literature 1 describes an automatic program code generation device. The device extracts text data from a document, searches the extracted text data for semantic content, and extracts basic syntax for program code based on the searched semantic content. The search for semantic content from the text data and the extraction of basic syntax for program code based on the semantic content are each performed using a trained model.

[0003] Japanese Patent Application Laid-Open No. 2022-138568

[0004] It is conceivable to generate a program (program code) using a generative AI (artificial intelligence) model such as a large-scale language model (LLM). However, there is a risk that a program generated by a generative AI model may contain bugs.

[0005] One embodiment of the present invention has been made in consideration of the above, and aims to provide a program generation device and a program generation method that can prevent bugs from being included in a program when generating the program using a generative AI model.

[0006] In order to achieve the above-mentioned object, a program generation device according to one embodiment of the present invention comprises a program acquisition unit that causes a generation AI model to generate a program and acquires the program, a program execution unit that executes the program acquired by the program acquisition unit, and a regeneration judgment unit that judges whether to regenerate the program based on the execution result obtained by the program execution unit executing the program, and the program acquisition unit causes the generation AI model to regenerate the program based on the judgment made by the regeneration judgment unit, and acquires the regenerated program.

[0007] In a program generation device according to one embodiment of the present invention, a program is regenerated using a generative AI model according to the execution results of the program generated by the generative AI model. This regeneration can reduce bugs in the regenerated program. Therefore, according to the program generation device according to one embodiment of the present invention, when a program is generated using a generative AI model, it is possible to prevent bugs from being included in the program.

[0008] Incidentally, one embodiment of the present invention can be described not only as an invention of a program generation device as described above, but also as an invention of a program generation method as described below. These are essentially the same inventions, just in different categories, and have similar functions and effects.

[0009] That is, a program generation method according to one embodiment of the present invention includes a program acquisition step of having a generation AI model generate a program and acquire the program, a program execution step of executing the program acquired in the program acquisition step, and a regeneration judgment step of determining whether to regenerate the program depending on the execution result obtained by executing the program in the program execution step, wherein in the program acquisition step, the generation AI model regenerates the program depending on the judgment in the regeneration judgment step, and the regenerated program is acquired.

[0010] According to one embodiment of the present invention, when a program is generated using a generative AI model, it is possible to prevent the program from containing bugs.

[0011] FIG. 1 is a diagram showing the configuration of a program generation device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a prompt used to generate a program. FIG. 3 is a diagram showing an example of a program generated by a generative AI model. FIG. 4 is a diagram showing an example of information used to execute a program. FIG. 5 is a diagram showing an example of an error message obtained as a result of executing a program. FIG. 6 is a diagram showing an example of a prompt used to regenerate a program. FIG. 7 is a diagram showing an example of a program generated by a generative AI model. FIG. 8 is a diagram showing an example of a prompt used to regenerate a program. FIG. 9 is a diagram showing an example of a program regenerated by a generative AI model. FIG. 10 is a diagram showing an example of a prompt used to obtain a solution to an error. FIG. 11 is a diagram showing an example of information indicating a solution generated by a generative AI model. A flowchart showing a program generation method which is a process executed by a program generation device according to an embodiment of the present invention. FIG. 12 is a diagram showing the hardware configuration of a program generation device according to an embodiment of the present invention.

[0012] Hereinafter, embodiments of a program generating device and a program generating method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals and duplicated explanations will be omitted.

[0013] 1 shows a program generation device 10 according to this embodiment. The program generation device 10 is a device (system) used to generate a program. The generated program is, for example, program code (source code).

[0014] The program generation device 10 uses a generative AI model to generate a program. The generative AI model is a model generated by machine learning. The generative AI model is a model that can generate content in response to a prompt input, based on any one or a combination of the instructions, context, question, and output format indicated by the prompt, and return the content as response information. The prompt can also include input information, in which case the generative AI model generates response information targeted at the input information. The generative AI model may be, for example, an interactive AI model that includes an LLM and a user interface (UI) for interaction with the user, and enables text or voice chat with the user. Examples of such generative AI models include ChatGPT, GPT (registered trademark)-3.5, GPT-4V, PaLM2, etc.

[0015] A prompt is a set of instructions or information input to a generative AI model. The prompt may include initial information, parameters, questions, etc., for the generative AI model to execute a specific task. A prompt is information indicating an instruction or question entered by a user in an interactive system, such as a dialogue with a generative AI model or a command line interface (CLI). The prompt may express, through text, for example, the command to be executed by the interactive AI model, the task to be executed by the interactive AI model, the background / context (e.g., role, condition) that the interactive AI model should consider, the question to be answered by the interactive AI model, and the output format of the response information from the interactive AI model. The prompt may also include input information that is the target of the command / task executed by the interactive AI model. Examples of such input information include data files with file names containing a predetermined extension, such as text data, image data, application-related data, audio data, video data, and still image data. Application-related data is data such as document data, table data, and graph data that can be processed by a default application program.

[0016] In this embodiment, the generative AI model 21 is realized by an AI server (server device) 20. The generative AI model 21 may be stored in the AI ​​server 20, or may be stored in another device connected to the AI ​​server 20 via a network and made available by the AI ​​server 20. Also, although the above describes an example of an LLM, other AI models may also be used. Note that the generative AI model 21 and the AI ​​server 20 may be similar to conventional models.

[0017] A plurality of generative AI models 21 of different types may be used in the program generation device 10. In this case, the AI ​​server 20 may be provided for each generative AI model 21, or may be common to the plurality of generative AI models 21.

[0018] The program generation device 10 re-learns the generative AI model 21 in response to the generation of a program by the generative AI model 21. Re-learning the generative AI model 21 means performing machine learning again on the generative AI model 21 that was generated by machine learning. The re-learning of the generative AI model 21 is intended to enable more appropriate program generation.

[0019] The program generation device 10 is configured by a computer such as a PC (personal computer) or a server device. The program generation device 10 may be configured by multiple computers. The program generation device 10 can transmit and receive information to and from other devices, including the AI ​​server 20, via a network to obtain information necessary to realize functions. The program generation device 10 may be a Retrieval-Augmented Generation (RAG) system. In this case, the knowledge database searched by the RAG system may be implemented within the program generation device 10 or may be implemented in another device on a network (e.g., the cloud).

[0020] Next, the functions of the program generation device 10 according to this embodiment will be described. As shown in FIG. 1, the program generation device 10 includes a program acquisition unit 11, a program execution unit 12, a regeneration determination unit 13, and a relearning unit 14.

[0021] The program acquisition unit 11 is a functional unit that causes the generative AI model 21 to generate (create) a program and acquires the program generated by the generative AI model 21. The program acquisition unit 11 causes the generative AI model 21 to regenerate the program in accordance with a determination by the regeneration determination unit 13, which will be described later, and acquires the regenerated program. That is, the program generation device 10 performs processing related to recursive program generation. The program acquisition unit 11 may acquire information related to solutions to errors that occur during program execution, and use the acquired information related to the solutions to cause the generative AI model to regenerate the program.

[0022] The generated program is, for example, a function that receives input information (e.g., input values), performs calculations and judgments on the input values, and calculates output information (e.g., output values). The generated program may also be one other than the above.

[0023] A program generated by the generation AI may contain a bug. A bug contained in a program may, for example, cause an error when the program is executed, or cause the output from the program to be incorrect according to the input. As will be described later, if a bug is contained in the program acquired by the program acquisition unit 11, the program acquisition unit 11 acquires the program again. That is, the program acquisition unit 11 acquires multiple programs (multiple times) according to the program generated by the generation AI. The program acquisition unit 11 acquires the first program of the multiple programs, for example, as follows:

[0024] The program acquisition unit 11 generates a prompt that causes the generation AI model 21 to generate a program, and inputs the generated prompt to the generation AI model 21. Figure 2 shows an example of a prompt generated by the program acquisition unit 11. The program acquisition unit 11 receives information transmitted from the terminal of the user of the program generation device 10, and generates a prompt using the received information. Note that the program acquisition unit 11 may also acquire information for generating a prompt using a method other than the above.

[0025] The prompt includes information indicating program generation instructions and information indicating the type of program to be generated. The information indicating program generation instructions and the type of program to be generated may be, for example, a text description of the program code and the program's contents. However, this information may also be non-textual. In the example shown in Figure 2, the information is "Please provide a code implementation based on the following description: The input is an array named count, where each element is an integer in the range [0, 255]. Here, count[k] is the number of times k appears in the sample. Please create a function to calculate the 'mean' of this array, which is the sum of all elements divided by the total number of elements."

[0026] The prompt may include example inputs, which are examples of inputs (arguments) to the generated program when it is executed. The prompt may also include example return values, which are examples of output from the generated program when it is executed with the example inputs. In the example shown in FIG. 2, the information is "#Example function input sample_count=[0, 2, 3, 0, 1, 4, 0, 0, 2, 1] #Example return value 4.384615385."

[0027] Note that the above sample_count indicates, from the left side of the array, that when k = 0, count[k] is 0, i.e., there are 0 0s, when k = 1, count[k] is 2, i.e., there are 2 1s, when k = 2, count[k] is 3, i.e., there are 3 2s, when k = 3, count[k] is 0, i.e., there are 0 3s, etc. Therefore, the average (example return value) indicated in the program code description for the input example is (0 x 0 + 1 x 2 + 2 x 3 + 3 x 0 + 4 x 1 + 5 x 4 + 6 x 0 + 7 x 0 + 8 x 2 + 9 x 1) / (0 + 2 + 3 + 0 + 1 + 4 + 0 + 0 + 2 + 1) = 4.384615385.

[0028] The input example and the return value example may not be included in the prompt as described above, but may be generated as a separate file attached to the prompt.

[0029] The prompt may include information indicating the conditions under which the program is generated. The information indicating the conditions under which the program is generated may, for example, indicate the language of the program to be generated and the environment in which the generated program code will be executed (e.g., the version of the program language and the version of the OS (operating system) on which the program will be executed). The language of the program to be generated may, for example, be Python or SQL (Structured Query Language). However, the information indicating the conditions under which the program is generated may be information other than the above types. In the example shown in FIG. 2 , the information is "#Programming language of the code to be generated: Python #Environment in which the code to be generated is executed: Python 3.7 Ubuntu Server 23.10."

[0030] The prompt may include other information related to the generation of the program, and may be other than the above, as long as it allows the generation of the program.

[0031] The program acquisition unit 11 may generate a prompt by acquiring the prompt itself from the user's terminal. Alternatively, the program acquisition unit 11 may generate a prompt by embedding information acquired from the user's terminal at a predetermined position in a fixed phrase, which is pre-stored information. The program acquisition unit 11 may also generate a prompt by a method other than the above.

[0032] The program acquisition unit 11 transmits the generated prompt to the AI ​​server 20 and inputs it to the generation AI model 21. The generation AI model 21 accepts the input of the prompt and generates a program in accordance with the prompt. FIG. 3 shows an example of a program generated by the generation AI model 21 based on the prompt shown in FIG. 2. The generation AI model 21 transmits the generated program to the program generation device 10 in response to the transmitted prompt. The program acquisition unit 11 receives and acquires the program transmitted from the generation AI model 21. The program acquisition unit 11 outputs the acquired program to the program execution unit 12. Note that the acquisition of a program after regeneration will be described later.

[0033] The program execution unit 12 is a functional unit that executes the program acquired by the program acquisition unit 11. The program execution unit 12 may input a prepared input example, execute a program, and acquire a return value from the program.

[0034] The program execution unit 12 inputs a program from the program acquisition unit 11. The program execution unit 12 executes the input program using a previously prepared execution environment. The program execution unit 12 may execute the program in the same manner as in the past. The program execution environment is, for example, Docker. The program execution environment may also be a virtual environment. The program execution unit 12 may also store previously prepared input examples and execute the program by inputting the stored input examples.

[0035] For example, the program execution unit 12 stores in advance information for executing the program shown in Fig. 4 and executes the program using this information. In the example shown in Fig. 4, an input example (an array with sample_count) is prepared in advance. The program is also a function with calculate_mean(). The input example is input to the program, the program is executed, and a return value, result, is output.

[0036] The program execution unit 12 acquires information indicating the execution result of the program. When the program outputs output information that is the result of the program execution (for example, the result of calculation and judgment on an input example), the program execution unit 12 acquires the output information as information indicating the execution result of the program. When an error occurs during program execution, the program execution unit 12 acquires an error message as information indicating the execution result of the program.

[0037] An example of an obtained error message is shown in Fig. 5. The error message shown in Fig. 5 is an error message that is output when the program shown in Fig. 3 is executed. In the program shown in Fig. 3, the variable name "sum(coutn)" on the fourth line should be "count", but there is a spelling error as it is written as "coutn".

[0038] The program execution unit 12 may acquire the execution results of programs other than those described above. The program execution unit 12 outputs information indicating the execution results of the programs to the regeneration determination unit 13 and the relearning unit 14.

[0039] The regeneration determination unit 13 is a functional unit that determines whether to regenerate the program based on the execution result obtained by the program execution unit 12 executing the program. The regeneration determination unit 13 may determine whether to regenerate the program by comparing the return value acquired by the program execution unit 12 with example return values ​​prepared in advance corresponding to example inputs. The regeneration determination unit 13 may also determine whether to regenerate the program based on whether an error occurs as a result of executing the program.

[0040] The regeneration determination unit 13 determines whether to regenerate the program based on a predetermined determination criterion, for example, as follows: The regeneration determination unit 13 inputs information indicating the execution result of the program from the program execution unit 12. The criterion for determining whether to regenerate the program is that the program is regenerated if the information indicating the execution result of the program input from the program execution unit 12 indicates that the program contains a bug. If the information indicating the execution result of the program is an error message, that is, if an error occurs as a result of executing the program, the regeneration determination unit 13 determines to regenerate the program.

[0041] If the information indicating the execution result of the program is output information that is not an error message, the regeneration determination unit 13 compares the output information with comparison information that has been prepared and stored in advance to determine whether to regenerate the program. The comparison information is information that is expected to be output when the program is executed by the program execution unit 12. For example, the comparison information is an example of a return value that is output from a bug-free program when an example input is input to the program execution unit 12 and the program is executed.

[0042] If the output information input from the program execution unit 12 and the comparison information do not correspond to each other, for example, if the return value from the program input from the program execution unit 12 does not match a pre-stored example return value, the regeneration judgment unit 13 judges to regenerate the program. In this case, this is because the generated program contains a bug. If the output information input from the program execution unit 12 and the comparison information correspond to each other, for example, if the return value from the program input from the program execution unit 12 matches a pre-stored example return value, the regeneration judgment unit 13 judges not to regenerate the program.

[0043] The regeneration determination unit 13 may determine whether to regenerate the program by a method other than the above, as long as the method conforms to the above-mentioned determination criteria. If it determines not to regenerate the program, the regeneration determination unit 13 outputs the program acquired by the program acquisition unit 11 as the program generated by the program generation device 10. The output of the program by the regeneration determination unit 13 may be performed by, for example, any conventional method that makes the program available to the user.

[0044] When it is determined that the program should be regenerated, the regeneration determination unit 13 instructs the program acquisition unit 11 to regenerate (reacquire) the program. The regeneration determination unit 13 may also output information to be used for regenerating the program to the program acquisition unit 11.

[0045] The program acquisition unit 11 acquires the regenerated program, for example, as follows: The program acquisition unit 11 receives an instruction to regenerate the program from the regeneration determination unit 13. Upon receiving the instruction from the regeneration determination unit 13, the program acquisition unit 11 generates a prompt that causes the generation AI model 21 to regenerate the program, and inputs the generated prompt to the generation AI model 21. Program regeneration generates a program that eliminates bugs in a previously generated program.

[0046] The prompt for regenerating a program includes information indicating instructions to regenerate the program and information indicating how to regenerate the program, i.e., how to eliminate bugs. The information indicating instructions to regenerate the program and information indicating how to regenerate the program are, for example, instructions to regenerate the program and explanatory text indicating how to regenerate the program. However, this information may be information other than text. Furthermore, the prompt for regenerating a program may include the prompt (program generation instructions) used when the program was previously generated and the previously generated program (program code).

[0047] The program acquisition unit 11 may generate a prompt by embedding information at a predetermined position in a fixed phrase, which is pre-stored information. The program acquisition unit 11 acquires, as the information to be embedded, information obtained when the previous program was generated and information indicating the execution result of the program by the program execution unit 12. The program acquisition unit 11 may also generate a prompt by a method other than the above.

[0048] The program acquisition unit 11 transmits the generated prompt to the AI ​​server 20 and inputs it to the generation AI model 21. The generation AI model 21 accepts the input of the prompt and generates (regenerates) a program in accordance with the prompt. The generation AI model 21 transmits the generated program to the program generation device 10 in response to the transmitted prompt. The program acquisition unit 11 receives and acquires the program transmitted from the generation AI model 21. The program acquisition unit 11 outputs the acquired program to the program execution unit 12.

[0049] The regenerated program may be processed in the same manner as the initially acquired program. That is, the regenerated program may be executed by the program execution unit 12, and then the regeneration determination unit 13 may determine whether to regenerate the program (for the third or subsequent generation), and then the same processing as described above may be performed. Alternatively, the number of times (threshold) the program is regenerated may be set in advance, and when the set number of regenerations is reached, the regeneration of the program may be terminated.

[0050] Alternatively, the program acquisition unit 11 may output the regenerated program (without performing the same processing as for the initially acquired program). The regenerated program may be output, as with the output by the regeneration determination unit 13, by any conventional method that makes the program available to the user.

[0051] The prompt for regenerating the program may be based on the execution result of the program related to the determination made by the regeneration determination unit 13. For example, the prompt may be based on whether the information indicating the execution result of the program is an error message, or whether the return value from the program does not match a previously stored example return value.

[0052] 6 shows an example of a prompt for regenerating a program, generated by the program acquisition unit 11 when the information indicating the program execution result is an error message. In the example shown in Fig. 6, the information indicating an instruction to regenerate the program and the information indicating how to regenerate the program is the following part: "#Program regeneration instruction When the generated program was executed, the following error message was output, so please generate the program again."

[0053] If the information indicating the execution result of the program is an error message, the prompt for regenerating the program may include the error message. In the example shown in Figure 6, the error message is the part beginning with "#error message" that follows the information indicating an instruction to regenerate the program and the information indicating how to regenerate the program. The error message in the prompt in Figure 6 (the part beginning with "#error message") is the same as that shown in Figure 5.

[0054] In the example shown in Figure 6, the prompt (program generation instruction) when the program was previously generated and the previously generated program (program code) are the parts following the error message, starting with "#program generation instruction" and "#generated program." The prompt when the program was previously generated in the prompt in Figure 6 (the part starting with "#program generation instruction") is the one shown in Figure 2, and the previously generated program (the part starting with "#generated program") is the one shown in Figure 3.

[0055] In this case, the program acquisition unit 11 acquires information indicating the execution result of the program by the program execution unit 12 (information beginning with "#Error message") and information on the previous generation of the program (information beginning with "#Program generation instruction" and "#Generated program"), embeds the information at a predetermined position in the template, and generates a prompt for regenerating the program. In this case, the regenerated program will have the error resolved. For example, in the example program shown in Figure 3, the spelling error in the program, "coutn," is replaced with the correct spelling, "count."

[0056] FIG. 7 shows another example of a program generated by the generative AI model 21 based on the prompt shown in FIG. 2. When this program is executed, no errors occur, but the output is different from what is expected. As mentioned above, when the program is executed by entering the example input included in the prompt shown in FIG. 2, the return value, or output, should be 4.384615385. However, when the example input is entered and this program is executed, the return value is 5.7. This is because the line "mean = total_sum / len(count)" in the program shown in FIG. 7 mistakenly divides by len(count) instead of by total_count, as it should.

[0057] Thus, the program shown in Fig. 7 is an example of a case where the return value from the program does not match the example return value stored in advance. Fig. 8 shows an example of a prompt for program regeneration generated by the program acquisition unit 11 in this case. In the example shown in Fig. 8, the information indicating an instruction to regenerate the program and the information indicating how to regenerate the program is the following part: "#Program regeneration instruction When the generated program was executed, a return value different from what was expected was returned, so please generate the program again."

[0058] If the return value from the program does not match the stored example return value, the prompt to regenerate the program may include the different return value from the program than expected. In the example shown in Figure 8, the error message is "#different return value from expected 5.7" followed by information indicating instructions to regenerate the program and information on how to regenerate the program.

[0059] In the example shown in Figure 8, the prompt (program generation instruction) when the program was previously generated and the previously generated program (program code) are the parts following the error message, starting with "#program generation instruction" and "#generated program." The prompt when the program was previously generated in the prompt in Figure 8 (the part starting with "#program generation instruction") is the one shown in Figure 2, and the previously generated program (the part starting with "#generated program") is the one shown in Figure 7.

[0060] In this case, the program acquisition unit 11 acquires information indicating the execution result of the program by the program execution unit 12 (a return value different from the expected value from the program) and information on the previous generation of the program (information starting with "#Program generation instruction" and "#Generated program"), and embeds this information at a predetermined position in the template to generate a prompt for regenerating the program. In this case, the regenerated program will output the expected return value. Figure 9 shows an example of a regenerated program. When the example input is entered and the regenerated program is executed, the return value is the expected value of 4.384615385.

[0061] If the information indicating the result of program execution is an error message, the program acquisition unit 11 may acquire information related to a solution to the error that occurred during program execution, and use the acquired information related to the solution to have the generation AI model 21 regenerate the program.

[0062] In this case, the program acquisition unit 11 acquires an error message, which is information indicating the execution result of the program by the program execution unit 12. The program acquisition unit 11 performs a search using the error message to acquire information related to a solution to the error. The information related to a solution to the error is, for example, information on the Internet. Specifically, for example, the program acquisition unit 11 performs a search using the error message on a preset site that provides information about programs (for example, Stack Overflow), and acquires, as information related to a solution to the error, text within a page (for example, a best answer or a high-score article). The acquisition of information from the page may be performed, for example, based on html (HyperText Markup Language) tags.

[0063] Alternatively, a database storing error messages in association with information indicating solutions may be prepared in advance in the program generation device 10, and the program acquisition unit 11 may acquire information indicating solutions corresponding to error messages from the database using SQL or the like. Acquisition of information from a prepared database may be performed when there is an error due to a file name error such as "file not found" that cannot be resolved by searching on the Internet.

[0064] A search using an error message may be performed by a conventional method. For example, a search using an error message may be performed by using the entire error message as a keyword, or by using keywords extracted from the error message. Furthermore, information related to solutions based on an error message may be obtained by a method other than a search.

[0065] The program acquisition unit 11 may also acquire information indicating a solution using the generative AI model 21. In this case, the program acquisition unit 11 may acquire information indicating a solution with organized content from information indicating a solution obtained from the Internet or the like as described above using the generative AI model 21. The program acquisition unit 11 generates a prompt for acquiring the information. The prompt includes information indicating an instruction to provide a solution and information indicating an error that is the target of the solution. The prompt may also include information indicating a solution obtained from the Internet or the like as described above.

[0066] FIG. 10 shows an example of a prompt generated by the program acquisition unit 11 in this case for acquiring information indicating a solution. The information indicating an instruction to provide a solution is, for example, a text (sentence) indicating the instruction. In the example shown in FIG. 10, the information is the following part: "#Instruction content: Please tell me how to solve the problem if a program error like the one below occurs. An article containing the solution is listed in the prompt."

[0067] The information indicating the error for which a solution is provided is, for example, an error message. In the example shown in Fig. 10, this is the part beginning with "#Error content" following the information indicating the instruction to provide a solution. In the example shown in Fig. 10, the information indicating the solution obtained from the Internet or the like is the part beginning with "#Article containing the solution."

[0068] The program acquisition unit 11 transmits the generated prompt to the AI ​​server 20 and inputs it to the generative AI model 21. The generative AI model 21 receives the input of the prompt and generates information indicating a solution in accordance with the prompt. An example of information indicating a solution generated by the generative AI model 21 based on the prompt shown in FIG. 10 is shown in FIG. 11.

[0069] As shown in FIG. 11 , the solution generated by the generative AI model 21 includes, for example, an explanation of the solution and key points from a reference article (information indicating the solution obtained from the Internet, etc.). In the example shown in FIG. 11 , the explanation of the solution is as follows: "We will explain the solution by comparing the content of the error with the article. The details of the error are as follows: Copy Python code NameError: name 'coutn' is not defined This is an error that the name "coutn" is not defined. In other words, the cause is a spelling mistake in the variable name. "coutn" was mistakenly entered where the correct word should be "count". The solution is to check the error message and correct the spelling mistake."

[0070] In the example shown in Figure 11, the key points from the referenced article are: "Key points from the referenced article: - A NameError is an error that occurs when an undefined variable name is used. - Read the error message and confirm that the variable name is undefined. - You can resolve the error by correcting the spelling mistake. For details on how to read error messages and how to resolve them, please check the following points: 1. Type of error (NameError) 2. Cause of the error (use of an undefined variable name) 3. Relevant location (line number and variable name shown in the error message) You can resolve the error by following these steps."

[0071] By using information indicating solutions with the contents organized as described above to regenerate the program, the generation AI model 21 becomes more likely to resolve errors.

[0072] The program acquisition unit 11 includes information related to the acquired solution and information indicating instructions to refer to the information when regenerating the program in a prompt for regenerating the program. For example, the program acquisition unit 11 includes the information in a prompt for regenerating the program when the information indicating the program execution result is an error message (for example, the prompt shown in FIG. 6 ). The program acquisition unit 11 uses the prompt to have the generation AI model 21 regenerate the program in the same manner as described above, and acquires the regenerated program.

[0073] The relearning unit 14 is a functional unit that re-learns the generative AI model 21 in accordance with the execution results obtained by the program execution unit 12 executing the program based on preset learning standards. That is, the relearning unit 14 automatically updates the generative AI model 21 that generates the program. The relearning unit 14 may re-learn the generative AI model 21 by comparing the return value acquired by the program execution unit 12 with example return values ​​prepared in advance corresponding to example inputs. The relearning unit 14 may re-learn the generative AI model 21 in accordance with whether or not an error occurs as a result of executing the program. The relearning unit 14 may also re-learn the generative AI model 21 in accordance with the code length of the program. The relearning unit 14 may also perform static analysis of the program and re-learn the generative AI model 21 in accordance with the results of the static analysis.

[0074] The re-learning of the generative AI model 21 by the re-learning unit 14 is intended to enable the generative AI model 21 to more appropriately generate a program, for example, to prevent the program generated by the generative AI model 21 from containing bugs.

[0075] The relearning unit 14 re-learns the generative AI model 21, for example, as follows. The relearning unit 14 inputs information indicating the execution result of the program from the program execution unit 12. The relearning unit 14 calculates a score, which is a reward, for the execution result obtained by executing the program based on preset criteria, and performs reinforcement learning of the generative AI model 21 based on the calculated score. For example, the score is set higher the more the execution result of the program is expected (appropriate). Reinforcement learning is performed so as to increase the score. For example, the score is set for each type of execution result of the program. That is, the relearning unit 14 re-learns the generative AI model 21 according to the type of execution result obtained by executing the program.

[0076] For example, the relearning unit 14 calculates a score depending on whether an error occurs as a result of executing the program, i.e., whether the information indicating the execution result of the program is an error message. Furthermore, if the information indicating the execution result of the program is output information that is not an error message, the relearning unit 14 compares the output information with pre-prepared and stored comparison information and calculates a score based on the comparison. This comparison is similar to the comparison between the output information and the comparison information by the regeneration determination unit 13 described above.

[0077] For example, if the information indicating the program execution result is an error message, the relearning unit 14 assigns the lowest score of 1.0. If the return value from the program does not match the pre-stored return value example, the relearning unit 14 assigns the next lowest score of 2.0. If the return value from the program matches the pre-stored return value example, the relearning unit 14 assigns the highest score of 3.0. Note that the above scores of 1.0 and 2.0 may be reversed.

[0078] The relearning unit 14 may also set the score according to the code length of the program. Typically, for programs that achieve the same function, it is desirable for the code to have a short code length (a small amount of code). Therefore, the shorter the code length, the higher the score the relearning unit 14 assigns. In this case, the relearning unit 14 acquires the code of the program acquired by the program acquisition unit 11. The relearning unit 14 calculates the number of tokens corresponding to the code length of the program. The relearning unit 14 may set the final score (the score used for relearning the generative AI model 21) as the product of the score calculated as above (a score of 1.0, 2.0, or 3.0) and the reciprocal of the calculated number of tokens.

[0079] When code length is used, the program acquisition unit 11 may cause multiple generative AI models 21 of different types to generate programs, and these multiple programs may be used for relearning the generative AI model 21. In this case, the multiple programs are programs for the same function, and for example, were generated using the same prompt. In this case, the relearning unit 14 may use the product of the calculated score and the inverse of the ratio of the calculated number of tokens to the maximum number of tokens in the codes of these multiple programs (maximum number of tokens / calculated number of tokens) as the final score (the score used for relearning the generative AI model 21).

[0080] The relearning unit 14 may also perform static analysis of the program and set the score according to the results of the static analysis of the program. In this case, the relearning unit 14 acquires the program code acquired by the program acquisition unit 11. Static analysis of a program is analysis (verification) performed without executing the program. Static analysis of a program identifies areas that violate predetermined coding patterns (coding rules). For Python, an example of a coding rule is PEP8. Coding rules include, for example, the number of characters per line, how to write indents, how to insert spaces, how to import libraries, and naming conventions (function names, class names, module names, package names, etc.). Static analysis can be performed using, for example, conventional tools.

[0081] For example, the more issues raised by the static analysis, the lower the score the re-learning unit 14. In this way, by reflecting the results of static analysis in addition to the results of executing the program (dynamic analysis) in the score, it is possible to incorporate perspectives other than simple normal and abnormal, and to allow the generative AI model 21 to generate a program with high readability and maintainability.

[0082] The re-learning unit 14 may calculate the above score using the trained reward model. The reward model itself may be the same as in the past. Furthermore, training of the reward model and calculation of the score using the reward model may be performed in the same manner as in the past.

[0083] The re-learning unit 14 uses the calculated score to re-learn the generative AI model 21 by a conventional reinforcement learning method. As an update algorithm, which is a conventional reinforcement learning method, for example, Proximal Policy Optimization is used. The re-learning of the generative AI model 21 by the re-learning unit 14 is reflected in the generation of a program for the generative AI model 21 after the re-learning. The above are the functions of the program generation device 10 according to this embodiment.

[0084] Next, a program generation method, which is a process executed by the program generation device 10 according to this embodiment (a method of operation performed by the program generation device 10), will be described using the flowchart of Figure 12. In this process, first, the program acquisition unit 11 generates a prompt to cause the generation AI model to generate a program (S01, program acquisition step). Next, the program acquisition unit 11 inputs the prompt to the generation AI model 21, and acquires a program from the generation AI model 21 (S02, program acquisition step). Next, the program execution unit 12 executes the program acquired by the program acquisition unit 11 (S03, program execution step).

[0085] Next, the regeneration judgment unit 13 judges whether to regenerate the program based on the execution result obtained by the program execution unit 12 executing the program (S04, regeneration judgment step). If it is judged that the program should be regenerated (YES in S04), the program acquisition unit 11 generates a prompt to cause the generation AI model to regenerate the program (S05, program acquisition step). Next, the program acquisition unit 11 inputs the prompt for program regeneration to the generation AI model 21, and the program is acquired from the generation AI model 21 (S02, program acquisition step). Thereafter, the processing from S03 onwards is repeated for the regenerated program.

[0086] In determining whether to regenerate the program (S04), if it is determined that the program will not be regenerated (NO in S04), the regeneration determination unit 13 outputs the latest program acquired by the program acquisition unit 11 as the program generated by the program generation device 10 (S06).

[0087] Furthermore, after the program execution unit 12 executes the program (S03), the re-learning unit 14 re-learns the generative AI model 21 based on a preset learning standard in accordance with the execution result obtained by executing the program (S07, re-learning step). The above is the program generation method, which is the processing executed by the program generation device 10 according to this embodiment.

[0088] In this embodiment, the generative AI model 21 regenerates a program according to the execution results of the program generated by the generative AI model 21. This regeneration can reduce bugs in the regenerated program. For example, as described above, the program is regenerated so as to eliminate bugs that become apparent as a result of the program execution. Therefore, according to this embodiment, when a program is generated using the generative AI model 21, it is possible to prevent bugs from being included in the program.

[0089] As in the present embodiment, the return value acquired by executing the program may be compared with a return value example prepared in advance corresponding to an input example to determine whether to regenerate the program. With this configuration, if a bug in the program is a mismatch between the return value acquired by executing the program and the return value example, the bug can be resolved more appropriately.

[0090] As in the present embodiment, whether to regenerate a program may be determined depending on whether an error occurs as a result of executing the program. With this configuration, if a bug in the program is an error, the bug can be resolved more appropriately.

[0091] In this case, information related to a solution to an error that occurs during program execution may be acquired, and the acquired information related to the solution may be used to have the generative AI model 21 regenerate the program. This configuration allows program bugs to be resolved more appropriately based on the solution. However, it is not necessary to use information related to a solution to the error when regenerating the program. Furthermore, the decision on whether to regenerate the program may be made by a method other than the above, as long as it is made based on the execution results obtained by executing the program.

[0092] As in the present embodiment, the generative AI model 21 may be retrained. According to this configuration, the generative AI model 21 is retrained according to the execution results of the program generated by the generative AI model 21. This retraining can reduce bugs in the program generated by the generative AI model after the retraining. For example, as described above, the retraining of the generative AI model 21 is performed so that bugs that become apparent as a result of program execution do not occur in the program generated by the generative AI model 21 after the retraining. Therefore, according to the present embodiment, when a program is generated using the generative AI model 21, it is possible to suppress the inclusion of bugs in the program.

[0093] As in the present embodiment, the generative AI model may be retrained by comparing the return value acquired by executing the program with example return values ​​prepared in advance corresponding to example inputs. With this configuration, if a bug in the program is a mismatch between the return value acquired by executing the program and the example return values, it is possible to more appropriately reduce bugs in the program generated by the generative AI model after retraining.

[0094] As in the present embodiment, the generative AI model may be retrained depending on whether an error occurs as a result of executing the program. With this configuration, if a bug contained in the program is an error, it is possible to more appropriately reduce bugs in the program generated by the generative AI model after retraining. However, the retraining of the generative AI model may be performed by a method other than the above, as long as it is performed depending on the execution results obtained by executing the program.

[0095] As in this embodiment, the generative AI model may be retrained according to the code length of the program. With this configuration, the code length of the program generated by the generative AI model after retraining can be made appropriate. However, the retraining of the generative AI model does not have to be performed according to the code length of the program.

[0096] As in this embodiment, a static analysis of the program may be performed, and the generative AI model may be retrained in accordance with the results of the static analysis. With this configuration, the program generated by the generative AI model after the retraining can be made appropriate in accordance with the static analysis of the program. However, the retraining of the generative AI model does not necessarily have to be performed in accordance with the results of the static analysis. Furthermore, the retraining of the generative AI model 21 does not necessarily have to be performed.

[0097] In the above embodiment, the program generation device 10 is a PC or a server device, etc., separate from the terminal of the user of the program generation device 10, but is not limited to this. For example, the program generation device 10 may be part of the terminal. As an example, a RAG system configured by the functional units of the program generation device 10 may be implemented in the terminal. In this case, the knowledge database searched by the RAG system may be implemented inside the terminal, or may be implemented in another device on a network (e.g., the cloud).

[0098] Furthermore, the program generation device 10 may have the functions of the AI ​​server 20. In this case, the configuration can be realized by installing an application that executes the functions of the AI ​​server 20 in the program generation device 10. In this way, the generated AI model 21 may be implemented on a network (e.g., the cloud) other than the AI ​​server 20.

[0099] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. The functional block may also be realized by combining software with the single device or multiple devices.

[0100] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0101] For example, the program generation device 10 according to an embodiment of the present disclosure may function as a computer that performs information processing according to the present disclosure. Fig. 13 is a diagram illustrating an example of a hardware configuration of the program generation device 10 according to an embodiment of the present disclosure. The above-described program generation device 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like. The hardware configuration of the AI ​​server 20 may also be as described here.

[0102] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the program generation apparatus 10 may be configured to include one or more of the apparatuses shown in the drawings, or may be configured to exclude some of the apparatuses.

[0103] Each function in the program generation device 10 is realized by loading specified software (programs) onto hardware such as a processor 1001 and a memory 1002, causing the processor 1001 to perform calculations, control communication via a communication device 1004, and control at least one of reading and writing data in the memory 1002 and the storage 1003.

[0104] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, each function of the program generation device 10 described above may be realized by the processor 1001.

[0105] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, each function of the program generation device 10 may be implemented by a control program stored in the memory 1002 and running on the processor 1001. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0106] The memory 1002 is a computer-readable recording medium and may be configured, for example, by at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for performing information processing according to an embodiment of the present disclosure.

[0107] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be called an auxiliary storage device. The storage medium provided in program generation device 10 may be, for example, a database, a server, or other appropriate medium including at least one of memory 1002 and storage 1003.

[0108] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0109] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0110] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0111] Furthermore, the program generation device 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0112] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0113] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0114] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0115] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0116] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0117] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0118] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0119] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0120] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0121] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0122] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0123] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0124] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0125] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0126] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0127] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0128] The program generation device and program generation method disclosed herein have the following configuration. [1] A program generation device comprising: a program acquisition unit that causes a generative AI model to generate a program and acquires the program; a program execution unit that executes the program acquired by the program acquisition unit; and a regeneration determination unit that determines whether to regenerate the program based on an execution result obtained by execution of the program by the program execution unit, wherein the program acquisition unit causes the generative AI model to regenerate the program based on the determination made by the regeneration determination unit, and acquires the regenerated program. [2] The program generation device described in [1], wherein the program execution unit inputs a prepared input example, executes the program, and acquires a return value from the program, and the regeneration determination unit compares the return value acquired by the program execution unit with a prepared return value example corresponding to the input example, and determines whether to regenerate the program. [3] The program generation device described in [1] or [2], wherein the regeneration determination unit determines whether to regenerate the program based on whether an error occurs as a result of executing the program. [4] The program generation device according to [3], further comprising a re-learning unit that re-learns the generative AI model in accordance with an execution result obtained by the program execution unit executing the program based on a preset learning standard.[6] A program generation method including: a program acquisition step of having a generative AI model generate a program and acquire the program; a program execution step of executing the program acquired in the program acquisition step; and a regeneration determination step of determining whether to regenerate the program depending on an execution result obtained by executing the program in the program execution step, wherein in the program acquisition step, the generative AI model regenerates the program depending on the determination in the regeneration determination step, and the regenerated program is acquired.

[0129] 10...program generation device, 11...program acquisition unit, 12...program execution unit, 13...regeneration judgment unit, 14...relearning unit, 20...AI server, 21...generated AI model, 1001...processor, 1002...memory, 1003...storage, 1004...communication device, 1005...input device, 1006...output device, 1007...bus.

Claims

1. A program generation device comprising: a program acquisition unit that causes a generative AI model to generate a program and acquires the program; a program execution unit that executes the program acquired by the program acquisition unit; and a regeneration judgment unit that judges whether to regenerate the program depending on the execution result obtained by the program execution unit's execution of the program, wherein the program acquisition unit causes the generative AI model to regenerate the program depending on the judgment of the regeneration judgment unit, and acquires the regenerated program.

2. The program generation device of claim 1, wherein the program execution unit inputs a pre-prepared input example, executes the program, and obtains a return value from the program, and the regeneration judgment unit compares the return value obtained by the program execution unit with a pre-prepared return value example corresponding to the input example, and determines whether to regenerate the program.

3. The program generation device according to claim 1, wherein the regeneration decision unit decides whether to regenerate the program depending on whether an error occurs as a result of executing the program.

4. The program generation device according to claim 3, wherein the program acquisition unit acquires information relating to solutions to errors that occur during execution of the program, and causes the generative AI model to regenerate the program using the acquired information relating to the solutions.

5. The program generation device according to claim 1, further comprising a re-learning unit that re-learns the generated AI model in accordance with the execution results obtained by the program execution unit executing the program based on a preset learning standard.

6. A program generation method comprising: a program acquisition step of having a generative AI model generate a program and acquire the program; a program execution step of executing the program acquired in the program acquisition step; and a regeneration decision step of deciding whether to regenerate the program depending on the execution result obtained by executing the program in the program execution step, wherein in the program acquisition step, the generative AI model regenerates the program depending on the decision made in the regeneration decision step, and the regenerated program is acquired.

Images