Generative model utilized data processing system and data processing method

Abstract

To reduce burdens of verifying and / or correcting output data of a generative model.SOLUTION: A data processing system inputs verification result information, which is information regarding results of verifying output data in response to input of a first prompt or a second prompt to a generative model, and determines whether or not the verification result information satisfies conditions for generating the second prompt. If the verification result information satisfies the conditions, the data processing system uses the verification result information to generate the second prompt for the generative model.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates generally to data processing using generative models. [Background technology]

[0002] An example of a generative model is a text generation model, typically a large language model (LLM). An example of data processing using an LLM is programming language conversion. A technology related to programming language conversion is known, for example, as disclosed in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-191738 Summary of the Invention [Problem to be solved by the invention]

[0004] Programming language conversion is generally performed by a rule-based program (language translator). However, the rules for programming language conversion must comply with requirements depending on at least one of the following: the provider of the original program (e.g., the vendor), the recipient of the converted program (e.g., the customer), and the project involved in the programming language conversion. Preparing such rules is a burden. These issues specific to rule-based programming can be resolved by using LLM for programming language conversion.

[0005] However, because LLM is a probabilistic model, even if the same prompt is input to the LLM, the output data of the LLM may vary probabilistically. Such phenomena can become bugs in the program after conversion to a programming language. While it is expected that the accuracy of LLMs can be improved by preparing a large amount of training data and training the LLM, preparing a large amount of training data and training using such a large amount of training data is a heavy burden. Furthermore, even if the accuracy of LLMs improves, because LLMs are probabilistic models, it is virtually impossible to reduce the probability of the occurrence of the above-mentioned phenomena to zero.

[0006] Similar challenges may also arise when processing data using generative models other than LLM. [Means for solving the problem]

[0007] The data processing system receives verification result information, which is information regarding the verification results of output data for the input of the first prompt or the second prompt to the generative model, and determines whether the verification result information satisfies the conditions for generating a second prompt. If the verification result information satisfies the conditions, the data processing system generates a second prompt for the generative model using the verification result information. [Effects of the Invention]

[0008] According to one aspect of the present invention, it is possible to reduce the burden of verifying and / or correcting output data of a generative model. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of a system including a programming language conversion device according to an embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of a flow of programming language conversion according to the embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of source code after conversion. [Figure 4] FIG. 10 is a diagram illustrating an example of test code. [Figure 5] FIG. 10 is a diagram illustrating an example of an error message. [Figure 6] FIG. 10 is a diagram illustrating an example of source code after conversion. [Figure 7] FIG. 10 is a diagram illustrating an example of a Lint rule. [Figure 8] FIG. 10 is a diagram illustrating an example of an error message. [Figure 9] FIG. 10 is a diagram illustrating an example of source code before conversion. [Figure 10] FIG. 10 is a diagram illustrating an example of source code after conversion. [Figure 11] FIG. 10 is a diagram illustrating an example of an error message. [Figure 12] FIG. 10 is a diagram illustrating an example of a prompt template. [Figure 13] FIG. 10 is a diagram illustrating an example of a prompt. [Figure 14] FIG. 10 is a diagram illustrating an example of a prompt template. [Figure 15] FIG. 10 is a diagram illustrating an example of a prompt template. [Figure 16] FIG. 10 is a diagram illustrating an example of a prompt template. [Figure 17] FIG. 10 is a diagram illustrating an example of a prompt template. [Figure 18] FIG. 10 is a diagram illustrating an example of a trace. [Figure 19] FIG. 1 illustrates an example of a compressed trace. [Figure 20] FIG. 10 is a diagram illustrating an example of an error message. [Figure 21] FIG. 10 is a diagram illustrating an example of a prompt template. [Figure 22] FIG. 10 is a diagram illustrating an example of a flow of generating a prompt, inputting the prompt, inputting verification result information, and repeating the first and second determinations. DETAILED DESCRIPTION OF THE INVENTION

[0010] In the following description, an "interface apparatus" may refer to one or more interface devices, which may be at least one of the following: One or more I / O (Input / Output) interface devices. The I / O (Input / Output) interface devices are interface devices for at least one of an I / O device and a remote display computer. The I / O interface device for the display computer may be a communications interface device. The at least one I / O device may be a user interface device, for example, either an input device such as a keyboard and a pointing device, or an output device such as a display device. One or more communication interface devices. The one or more communication interface devices may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., a NIC and an HBA (Host Bus Adapter)).

[0011] In the following description, "memory" refers to one or more memory devices, which are an example of one or more storage devices, and may typically be a primary storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0012] In the following description, a "persistent storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), and specifically may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a non-volatile memory express (NVME) drive, or a storage class memory (SCM).

[0013] In the following description, the term "storage device" may refer to at least one of memory and persistent storage device.

[0014] Furthermore, in the following description, a "processor" may refer to one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a circuit that is a collection of gate arrays written in a hardware description language that performs some or all of the processing (for example, an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0015] In the following description, functions are sometimes described using the expression "yyy unit." However, the functions may be realized by one or more computer programs executed by a processor, by one or more hardware circuits (e.g., FPGAs or ASICs), or by a combination thereof. When a function is realized by a program executed by a processor, the specified processing is performed using a storage device and / or an interface device, etc., as appropriate, and therefore the function may be considered to be at least a part of the processor. Processing described using a function as the subject may be processing performed by a processor or a device having the processor. A program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable storage medium (e.g., a non-transitory storage medium). The description of each function is merely an example; multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0016] Hereinafter, an embodiment will be described with reference to the drawings. In the embodiment, an LLM is adopted as a generative model, and a programming language conversion is adopted as data processing using the LLM. For convenience, a program before programming language conversion is referred to as a "source program," and a program after programming language conversion is referred to as a "target program."

[0017] FIG. 1 is a diagram showing an example of the overall configuration of a system including a programming language conversion device according to an embodiment.

[0018] In this embodiment, the programming language conversion device (hereinafter referred to as the conversion device) 100 is a physical computer system (one or more computers), but it may alternatively be a logical computer system based on a physical computer system (for example, a system as a cloud computer service based on a cloud infrastructure).

[0019] The conversion device 100 is configured to perform the following (a) and (b). (a) It is checked whether or not information on the verification result of the output data after the programming language conversion by the LLM 151 to which the prompt is input satisfies the prompt generation conditions. (b) If the prompt generation condition is met in (a), a prompt is automatically generated to obtain different output data, and the process returns to (a).

[0020] That is, the conversion device 100 repeats (a) and (b) until the prompt generation condition is no longer satisfied.

[0021] The conversion device 100 includes an interface device 110, a storage device 120, and a processor 130 connected thereto.

[0022] The interface device 110 communicates with the LLM execution device 150 and the user interface device 190. Communication with at least the LLM execution device 150 may be performed via a communication network (e.g., the Internet or a Wide Area Network (WAN)). The LLM execution device 150 may be a physical computer system or a logical computer system. It receives a prompt from a sender, inputs the received prompt into the LLM 151, and returns data output from the LLM 151 to the sender of the prompt. The LLM 151 may be located outside the conversion device 100 as illustrated in FIG. 1, or may be stored in the storage device 120 of the conversion device 100 and the LLM within the conversion device 100 may be used. The user interface device 190 may be an input / output device such as a keyboard, a pointing device, and a display device, or may be a remote information processing terminal (e.g., a personal computer or a smartphone) acting as a client when the conversion device 100 is a server. The user can use the user interface device 190 to send desired instructions to the conversion device 100 and to view information sent from and displayed by the conversion device 100 .

[0023] The storage device 120 stores data and programs. The storage device 120 stores, for example, a source program source file 121, a target program source file 122, and one or more check tools 123. The source program source file 121 is a source file of a source program, and the language written in the source file is a first programming language. The target program source file 122 is a source file of a target program, and the language written in the source file is a second programming language converted from the first programming language. The check tool 123 is a tool used for checking, which is a determination of all or part of error determination. The one or more check tools 123 may include at least one of a compiler, a test tool, and a lint tool, and / or may include other tools. The storage device may also store one or more prompt templates 124. The storage device 120 also stores a program executed by the processor 130.

[0024] When the processor 130 executes the program, functions such as an LLM interface unit 132, an input unit 137, a processing unit 20, and an output unit 135 are realized. The processing unit 20 includes, for example, a prompt generation unit 131, a determination unit 134, and a user interface unit 136. The prompt generation unit 131 generates a prompt for the LLM 151. The LLM interface unit 132 inputs a prompt to the LLM 151 and receives output data from the LLM 151 in response to the input of the prompt. The input unit 137 inputs verification result information regarding the verification result of output data output as a result of inputting a prompt to the LLM 151, and stores the verification result information in the storage device 120 (e.g., memory). The determination unit 134 determines whether the verification result information satisfies a condition for generating a prompt 250Y (described below). If the result of the determination is true, the prompt generation unit 131 generates a prompt 250Y for the LLM 151 using the verification result information (e.g., the error information). Specifically, for example, the determination unit 134 performs a first determination as to whether the verification result information includes error information. If the result of the first determination is true, the determination unit 134 performs a second determination as to whether a condition for generating a prompt 250Y (hereinafter, a continuation condition) is satisfied based on the verification result information (e.g., the error information). If the determination result of the processing unit 20 is false, for example, if the verification result information includes information indicating no error has been detected (i.e., if the result of the first determination is false), the output unit 135 outputs (e.g., stores) the target program source file 122 containing the target program source code as the output data. The user interface unit 136 communicates with the user interface device 190, for example, receives requests from the user interface device 190, and transmits display information or other information to the user interface device 190. Note that an example of the verification result of the output data output as a result of a prompt being input to the LLM 151 may be the result of error determination by the error determination unit 133 (see FIG. 2). Some functions such as the error determination unit 133 (for example, the LLM interface unit 132) may be provided in the conversion device 100, but may also be provided in a device external to the conversion device 100 (for example, the LLM execution device 150 or another device).That is, information regarding the error determination result may be received from a device external to the conversion device 100. Alternatively, first software for realizing some of the functions 131, 132, and 134 to 137 other than the error determination unit 133 and second software for realizing the error determination unit 133 may be executed in the conversion device 100, the vendor of the first software may be different from the vendor of the second software, and the second software may receive information regarding the error determination result from the first software.

[0025] Of the above-described functions 20 and 131 to 137, some functions may be omitted, two or more functions may be combined into one function, or one function may be divided into two or more functions. Furthermore, functions other than the above-described functions 20 and 131 to 137 may also be implemented. For example, the function of the LLM interface unit 132 may be included in at least one of the prompt generation unit 131, the error determination unit 133, and the input unit 137, and the LLM interface unit 132 may not be present. Furthermore, when the error determination unit 133 is provided in the conversion device 100, the error determination unit 133 and the input unit 137 may be integrated.

[0026] FIG. 2 is a diagram illustrating an example of a flow of programming language conversion according to the embodiment.

[0027] 2 may be started, for example, as follows: The user interface unit 136 may receive a request for programming language conversion from the user interface device 190. The request may be associated with a designation of a source program and a designation of a programming language to be converted. The user interface unit 136 identifies the source program to be converted and the converted programming language from the request, and designates the identified source program and converted programming language to the prompt generation unit 131. As a method for designating the source program and the converted programming language to the prompt generation unit 131, instead of designating them via the user interface unit 136, another method may be adopted, such as registering a programming language conversion schedule in advance and designating them according to the schedule.

[0028] The prompt generation unit 131 reads the source program source file 121 of the specified source program from the storage device 120 and generates a prompt 250X (S201). The prompt 250X generated in S201 is a prompt for converting source code in a first programming language written in the source program source file 121 into source code in a second programming language. The prompt 250X may include all or part of the source code in the first programming language contained in the source program source file 121. If the prompt 250X includes part of the source code, the prompt 250X may also be generated for the remaining source code contained in the source program source file 121 in parallel or sequentially (for example, each time an error determination result, including no error detection, is obtained).

[0029] The LLM interface unit 132 inputs the prompt 250X generated in S201 to the LLM 151 (S202), and receives output data 260 in response to the input of the prompt 250X from the LLM 151 (S203).

[0030] The error determination unit 133 performs an error determination on the output data 260, specifically, an error determination including a determination as to whether or not an error occurs in the output data 260. The error determination is performed using the check tool 123. If an error occurs in the output data 260 during the error determination using the check tool 123, the check tool 123 outputs an error message related to the detected error. That is, the verification result information as information related to the result of the error determination includes error information related to the error. The input unit 137 inputs the verification result information and stores it in the storage device 120 (S204). The determination unit 134 performs a first determination as to whether or not the verification result information includes error information (S205).

[0031] If the first judgment result is false (if the verification result information does not contain error information), the output unit 135 outputs the target program source file 122 in which the source code in the second programming language in the output data 260 is written (S206).

[0032] If the first judgment result is true (if the verification result information includes error information), that is, if an error is detected from the output data 260, the judgment unit 134 makes a second judgment as to whether the continuation condition is satisfied (whether to generate a prompt 250Y for error correction) (S207).

[0033] If the second determination result is false, a predetermined process for stopping the programming language conversion is performed. For example, the user interface unit 136 outputs information notifying the user interface device 190 that the programming language conversion has stopped (S208). The user can learn that the programming language conversion has stopped by viewing the information. The information may include information indicating the reason for the stopping of the programming language conversion (e.g., an error has not been corrected even after a predetermined number of prompt generation and input attempts).

[0034] If the second determination result is true, the prompt generation unit 131 automatically generates a prompt 250Y that is expected to obtain source code in the second programming language in which the detected error has been corrected (S209). Then, S202 and subsequent steps (i.e., input of the generated prompt 250Y to the LLM 151 and subsequent steps) are performed again. That is, the sequence of (S202) → (S203) → (S204) → (S205) → (S207) → (S209) is repeated until the error determination result of the output data 260 of the LLM 151 indicates that no error has been detected (see FIG. 22).

[0035] A detailed example applicable to this embodiment will be described below. Note that "Java" is a registered trademark.

[0036] (1) First, an example of a check tool 123 other than a compiler will be described.

[0037] The checking tool 123 other than a compiler can be at least one of the following: (1a) Testing tools. (1b) Lint tool.

[0038] The tool (1a), ie, the test tool, is, for example, as follows:

[0039] The test tool has test code in which expected values ​​are described, and is a tool for checking whether values ​​that match the expected values ​​described in the test code are obtained. Specifically, the test tool is a tool that targets executable code after compiling source code generated by LLM151, executes the executable code, and checks whether the values ​​obtained by the execution match the expected values ​​described in the test code.

[0040] For example, to test 2+2 and check whether the result is 4, use the following:

[0041] An example of the converted source code is shown below (see Figure 3). public class Translated{ public int add(int a, int b) { return a + b; } }

[0042] An example of test code is as follows (see Figure 4): Test public void testAdd() { / / Create an instance of the class to be tested Translated target = new Translated(); / / Call the method under test and check the result int result = target.add(2, 2); / / Check that the expected result matches the actual result assertEquals(4, result); }

[0043] An example of an error message that may appear when an error is detected during a check using a test tool is shown below (see Figure 5). org.junit.ComparisonFailure: expected: <4> but was: <5> at your.package.name.Translated.testAdd(Translated.java:15)

[0044] For tool (1b), i.e., the Lint tool, for example, it is as follows:

[0045] The Lint tool is a tool for determining whether the source code generated by LLM151 conforms to predetermined rules (hereinafter referred to as Lint rules). These Lint rules may include definitions of poor writing that is likely to lead to bugs, or may include coding conventions (for example, indentation should be expressed with four spaces). Lint rules may be prepared for each organization, such as a team.

[0046] For example, an example of a check using the Lint tool is to check whether coding rules regarding indentation are being followed. An example of such a check is as follows:

[0047] An example of the converted source code is as follows (see Figure 6): public class Translated{ public int add(int a, int b) { return a + b; } }

[0048] An example of a Lint rule is as follows (see Figure 7): <module name=""TreeWalker”"> <!-- Indent setting --> <module name=""Indentation”"> <property name=""basicOffset”" value=""4” / "> <property name=""caseIndent”" value=""4” / "> <property name=""throwsIndent”" value=""4” / "> < / property> < / property> < / property> < / module> < / module>

[0049] An example of an error message that is displayed when an error is detected during a check using the Lint tool is as follows (see Figure 8): [ERROR] / path / to / Translated.java:10: Indentation

[0050] Multiple check tools 123, for example, two or more of a compiler, test tool, and lint tool, may be used. The multiple check tools 123 may be executed sequentially, or two or more check tools 123 may be executed in parallel. For example, because testing can be applied to compiled code, the execution order of the check tools 123 may be compiler → test tool → lint tool. Note that the lint tool may be executed before or in parallel with the compiler or test tool; however, in this embodiment, it is executed after the compiler and test tool are executed. This is because both the compiler and test tool determine the correctness of the converted program (e.g., whether the desired result is obtained), while the lint tool determines aspects of the program that have a lower priority (e.g., readability) than correctness. For example, assuming that a functional check (compiler: high priority) and an appearance check (lint tool: low priority) are performed, the following is expected. If either the functional check result or the visual check result is NG, a prompt will be automatically generated. However, if the functional check and visual check are performed only once each, the processing time and cost will not change significantly. However, in cases where functional and visual checks are repeatedly performed, if the functional check results are NG but the visual check results are OK, and code is re-entered, there is a high possibility that the visual appearance will be damaged in order to correct the functionality, so the advantage of repeatedly performing visual checks to correct the functionality, then performing functional checks and correcting the functionality is not high. Furthermore, if neither the functionality nor the appearance is modified, the final output is code that passes the functional check, which means the conversion is essentially successful. The difficulty of modifying the appearance is low and the amount of work required is small. On the other hand, code that passes the visual check is essentially conversion failure. Therefore, the difficulty of modifying the functionality is high and the amount of work required is large. Therefore, by executing in order of priority, it is expected that unnecessary correction efforts (especially the cost of manual correction when automatic correction is not possible) can be reduced.

[0051] (2) Next, an example of automatic generation of prompt 250Y based on the result of error determination will be described.

[0052] The method for automatically generating the prompt 250Y may be, for example, at least one of the following methods. (2a) Question generation to elicit context when querying LLM151. (2b) Interpretation of error messages obtained during error detection (e.g., information organization). (2c) A combination of (2a) and (2b).

[0053] Method (2a) is a method of generating a prompt using a prompt template 124. A prompt template 124 may be prepared for each check tool 123. Depending on which check tool 123 outputs an error message during error determination, the prompt generation unit 131 selects a prompt template 124 from one or more prompt templates 124, and generates a prompt 250Y using the selected prompt template 124 and the output error message.

[0054] An example in which the checking tool 123 is a compiler is as follows.

[0055] An example of source code before conversion is as follows (see Figure 9): IDENTIFICATION DIVISION. PROGRAM-ID. MainProgram. DATA DIVISION. WORKING-STORAGE SECTION. 01 NUM1 PIC 9(3) VALUE 0. 01 NUM2 PIC 9(3) VALUE 0. 01 RESULT PIC 9(4) VALUE 0. PROCEDURE DIVISION. DISPLAY 'COBOL Addition Program'. ACCEPT NUM1. ACCEPT NUM2. ADD NUM1 TO NUM2 GIVING RESULT. STOP RUN.

[0056] An example of source code after conversion is shown below. The example below contains an error in which a semicolon is missing at the end of a statement (see Figure 10). public class Translated{ public int add(int a, int b) { return a + b } }

[0057] An example of the error message output by the compiler is as follows (see Figure 11): Translated.java:4: error: ';' expected return a + b ^ 1 error

[0058] An example of one prompt template 124 prepared for the compiler is as follows (see FIG. 12): (The relevant code or message is set between "<<" and ">>". The same applies to the other prompt templates 124. The set containing "<<", ">>", and the description between them is an example of text that specifies the type of information to be set and the location where that type of information should be set.) When I compile the following code, <<Program Insertion Section>> The following error occurred: <<Error message insertion part>> Please revise your code to avoid this error.

[0059] Based on this prompt template 124, prompt generation unit 131 generates prompt 250Y that includes a description of the source code before conversion and the output error message set in the relevant section (i.e., between "<<" and ">>") of prompt template 124. An example of prompt 250Y is as follows (see FIG. 13): When I compile the following code, public class Translated{ public int add(int a, int b) { return a + b } } The following error occurred: Translated.java:4: error: ';' expected return a + b ^ 1 error Please revise your code to avoid this error.

[0060] An example of the prompt template 124 when the check tool 123 is a test tool is as follows (see FIG. 14): The code below, <<Program Insertion Section>> I tested it with the following test code: <<Test code insertion part>> The following failure occurred: <<Test result insertion section>> Please rewrite your program to avoid this failure.

[0061] An example of the prompt template 124 when the check tool 123 is a Lint tool is as follows (see FIG. 15): The code below, <<Program Insertion Section>> When checked with the following Lint rules, <<Lint Definition Insertion Section>> The following failures occurred. <<Lint Result Insertion Section>> Please modify the program to avoid this failure.

[0062] Also, for example, for at least one check tool 123, a prompt template 124 including prompt engineering Tips for obtaining better results by specifying the programming language after conversion may be prepared, and a prompt 250Y may be generated using such a prompt template 124. An example of such a prompt template 124 is as follows (see Figure 16). Note that "prompt engineering" is to design and / or optimize the query statement in order to obtain a more desirable output from the LLM. "Prompt engineering Tips" are Tips for prompt engineering, and from the results of such engineering, know-how on what kind of prerequisite information, vocabulary, and query syntax should be given for each purpose to obtain better results is summarized. An example of know-how for giving prerequisite information that leads to better results during program language conversion is "You are ~~~"). You are a <<Programming Language>> developer. When the following code was compiled, <<Program Insertion Section>> The following errors occurred. <<Error Display Insertion Section>> Please modify the code to avoid this error.

[0063] Also, for example, for at least one check tool 123, a prompt template 124 may be prepared to maintain the context of the source code before conversion. An example of such a prompt template 124 is as follows (see Figure 17). You are a <<new programming language>> developer. Convert the following <<old programming language>> programs, <<Old program insertion part>> The following <<new programming language>> programs I had it generated. <<Program Insertion Section>> When I compile this code, The following error occurred: <<Error message insertion part>> Please revise your code to avoid this error.

[0064] Method (2b) is a method of simplifying a redundant error message (for example, removing noise) and using the simplified error message to generate prompt 250Y. A specific example will be described below.

[0065] For example, suppose that when the above test (2+2) is executed, the following trace is output (see Figure 18). However, let's assume that the only information needed is the first and last sentences below. org.junit.ComparisonFailure: expected: <4> but was: <null> at org.junit.Assert.assertEquals(Assert.java:115) at org.junit.Assert.assertEquals(Assert.java:144) at your.package.name.CalculatorTest.testAddWithNull(CalculatorTest.java:15) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:89) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:541) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:763) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:463) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209)

[0066] The prompt generator 131 interprets the trace and compresses it as follows (see FIG. 19): org.junit.ComparisonFailure: expected: <4> but was: <null> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209)

[0067] Furthermore, when a test tool is executed for error detection, not only the test result (success or failure of the test) but also the following runtime error message may be generated. An example of a runtime error message is as follows (see Figure 20): java.lang.NullPointerException at org.junit.Assert.assertEquals(Assert.java:115) (Omitted) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209)

[0068] In this case, in order to select an appropriate prompt template 124, the prompt generation unit 131 may, in addition to simplifying the trace, interpret that "a runtime error has occurred" and analyze auxiliary information for selecting an appropriate prompt template 124. This combination of auxiliary information is method (2c). If a runtime error occurs during true / false determination by the test tool, the prompt generation unit 131 selects the following prompt template 124 (see FIG. 21) and generates prompt 250Y using that prompt template 124. You are a <<programming language>> developer. The code below, <<Program Insertion Section>> I tested it with the following test code: <<Test code insertion section>> The following runtime error was output: <<Reduction result insertion part>> Please rewrite your program to avoid this failure.

[0069] (3) A text generation model such as an LLM (hereinafter, a prompt generation model) may also be used to generate prompt 250Y (and prompt 250X). In this case, a preprocessing template may be used. A "preprocessing template" is a prompt template used to generate a prompt for a prompt generation model to generate prompt 250 for LLM 151. A preprocessing template may include at least one of the following: Context (e.g., text like, "You will use generative AI to come up with prompts to modify a given program.") Text written in the location to be set in the preprocessing template, which indicates what description (for example, test code, Lint rule, or error message) should be set.

[0070] Furthermore, the preprocessing template may be associated with at least one of the following: For example, a preprocessing template may be prepared for each type of check tool. The type of checking tool 123 (e.g., compiler, test tool, or lint tool). - Criteria for determining what results are required to determine whether the error result is true. A history of prompts previously generated using the prompt template 124 (which may include the results of error detection for each prompt).

[0071] When an error message is output by any of the check tools 123, the prompt generation unit 131 selects a pre-processing template associated with the check tool 123 that output the error message. The prompt generation unit 131 sets an error message and / or other descriptions in the pre-processing template, and inputs a prompt having the pre-processing template that includes a context and in which a description has been set to the prompt generation model, thereby acquiring a prompt 250Y for the LLM 151 as output data of the prompt generation model. The prompt generation unit 131 generates the acquired prompt 250Y. The generated prompt 250Y is input to the LLM 151 by the LLM interface unit 132.

[0072] (4) When the result of the error determination of the data 260 output in response to the input of the prompt 250Y for correction includes a re-detection of an error (for example, the same or a different error message is output as previously), the processing to be performed is, for example, at least one of the following processing (4a) and (4b). (4a) Processing common to multiple check tools 123. (4b) For at least one of the one or more check tools 123, processing according to the check tool 123.

[0073] (4a) The processes common to multiple check tools 123 are, for example, as follows:

[0074] The error determination unit 133 or the input unit 137 is configured to accumulate the error message output in the error determination as a history in the storage device 120 (for example, a memory) each time an error determination is made. Each time an error determination is made, the following (a1) and (a2) are performed. (a1) The error determination unit 133 or the input unit 137 determines whether or not the error message output in the current error determination is the same as the error message output in the immediately preceding error determination. If the determination result of (a2)(a1) is true, the error determination unit 133 or the input unit 137 outputs information indicating that the same error message as the immediately preceding one has been obtained. Based on this information, the prompt generation unit 131 selects a re-generation prompt template 124 and generates a prompt 250Y using the re-generation prompt template 124. The prompt 250Y is input to the LLM 151 via the LLM interface unit 132.

[0075] A "prompt template for regeneration" is a prompt template 124 used to generate a prompt 250Y for correcting an error that was not corrected the previous time. The prompt template 124 for regeneration includes a sentence that causes the LLM 151 to generate a prompt using a different approach from the one generated in response to the previous prompt 250Y, such as, for example, "The previous correction proposal resulted in a similar error message. Please correct it using a different approach." When a prompt 250Y generated using a prompt template for regeneration 124 containing such a sentence is input to the LLM 151, output data 260 in which the error was corrected the previous time is expected. A "previous error" is an error detected in the previous error judgment. A "previous error that was not corrected" is an error that was detected again in the previous error judgment even though a prompt 250Y for correcting the previous error was input.

[0076] Hereinafter, the generation of prompt 250Y for correcting the previous error can be referred to as "regeneration" for convenience.

[0077] The determination unit 134 performs the second determination based on at least one of the following (X) and (Y). (X) The error determination unit 133 or the input unit 137 counts the number of program matches, which is the number of times source code with the same character string continues to be generated after regeneration, for the program to be corrected (a program that conforms to the source code described in the output data 260 and contains an error). The determination unit 134 determines whether the number of program matches is equal to or less than a first threshold. If the result of this determination is false, that is, if the number of program matches exceeds the first threshold, the determination result is "stop." Note that the number of program matches is reset by the error determination unit 133 (for example, the number is set to "0") if source code with a different character string is detected after regeneration. (Y) The error determination unit 133 or the input unit 137 counts the number of error message matches, which is the number of times the same error message as the immediately preceding error message is output for the program to be modified. The determination unit 134 determines whether the number of error message matches is equal to or less than a second threshold. If the result of this determination is false, that is, if the number of error message matches exceeds the second threshold, the determination result is "stop." Note that the number of error message matches is reset by the error determination unit 133 (for example, the number is set to "0") when an error message different from the immediately preceding error message is output. The second threshold may be the same value as the first threshold or a different value.

[0078] In this way, the determination unit 134 monitors the program match count and / or the error message match count, and when either count exceeds the threshold value (or when all counts exceed the respective threshold values), it determines that the source program cannot be converted into a program that can run correctly. In this case, the user interface unit 136 notifies the user (e.g., the developer) via the user interface device 190 of notification information including information indicating that the programming language conversion cannot be performed. Note that all or part of the output data 260 containing the source code of the program to be corrected may be stored in the storage device 120 by the error determination unit 133 or the input unit 137. Furthermore, when an error message is output, a set including the output error message and information indicating the type of the check tool 123 that output the error message may be stored in the storage device 120 by the error determination unit 133 or the input unit 137. The notification information output by the user interface unit 136 may include, in addition to information indicating that the programming language conversion cannot be performed, the source code obtained until it is determined that the programming language conversion cannot be performed (until the determination result "stop" is obtained), an error message, and information indicating the type of check tool 123 that output the error message. This allows the user to estimate what corrections should be made in manual work even if error-free source code in the second programming language cannot be obtained, and it is expected that a target program that conforms to the error-free source code in the second programming language will be created with a small amount of work based on the error-containing source code in the second programming language obtained by the LLM 151.

[0079] In the explanation of (4a), the "last time" is used in the error determination, but the "previous time" may be used instead of the "last time". This is because even if the current (most recent) error determination detects that the error was corrected the last time (even if the error message of the last time was not output), the same error as the error detected in the time before the last time may be detected, or the same error message as the error message output in the time before the last time may be output. When the "last time" is used instead of the "last time", the number of program matches and the number of error message matches may each be a cumulative value.

[0080] (4b) For at least one check tool 123, the processing according to the check tool 123 includes generating a prompt 250Y according to the difference between the current check result and the previous check result by the check tool 123.

[0081] For example, suppose that the check tool 123 is a test tool that runs test cases 1 to 5 (examples of multiple different test cases). Each time the test tool performs a check, it records the success or failure of each test case in, for example, the storage device 120. The prompt generation unit 131 identifies differences between the current check result (e.g., results including which test cases were successful and which test cases were failed) and the previous check result of the test tool, and generates prompt 250Y based on the identified differences. The identified differences may include, for example, one or more test cases that were successful the previous time but failed this time, and / or one or more test cases that failed the previous time but succeed this time. Prompt 250Y may be generated based on whether there were more successful test cases in the previous or current check and / or the test cases in the difference. For example, if the identified difference is that test cases 1, 2, and 5 were successful in the previous check, but only test case 1 was successful in the current check (an example of a situation in which more test cases were successful in the previous check), the prompt generator 131 may generate prompt 250Y including a sentence such as, "Since test cases 2 and 5 have failed in this correction, discard this correction and try again using the previous test code correction proposal." Such a prompt 250Y is an example of a prompt 250Y that causes the LLM 151 to generate source code based on the source code contained in the output data 260 generated in response to the previous prompt 250Y, rather than the source code contained in the output data 260 generated in response to the current prompt 250Y.

[0082] Also, for example, assume that the check tool 123 is a Lint tool. A Lint tool typically performs a check on a sentence-by-sentence basis. The prompt generation unit 131 determines whether the source code in the previous or current output data 260 should be used for correction based on the number of failures (the number of sentences for which the success / failure result was false) in the previous check (which may not be limited to the previous check) or the number of failures in the current check, and generates a prompt 250Y according to the determination. For example, if the number of failures in the current check is greater than the number of failures in the previous check, the prompt generation unit 131 discards the source code in the current output data 260 and generates a prompt 250Y that causes the LLM 151 to generate a corrected version of the source code based on the source code in the previous output data 260.

[0083] Although one embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. The present invention can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment is included within the scope and spirit of the invention, and is also included in the invention described in the claims and their equivalents.

[0084] For example, the above description can be summarized as follows. Note that the following summary may include supplementary explanations and explanations of variations of the above description. For example, in the following description, "error" may be an "error" in the narrow sense of an abnormality, or an "error" in the broad sense of something to be corrected.

[0085] The above-described conversion device 100 is an example of a device to which a data processing system according to an embodiment of the present invention is applied. The data processing system includes an input unit (e.g., input unit 137) and a processing unit (e.g., processing unit 20). The input unit inputs verification result information, which is information regarding the verification result of output data in response to the input of a first prompt (e.g., prompt 250X) or a second prompt (e.g., prompt 250Y) to a generative model. The processing unit determines whether the verification result information satisfies a condition for generating a second prompt, and if the verification result information satisfies the condition, generates a second prompt for the generative model using the verification result information. This "condition" may include, for example, that the verification result information includes error information and that a continuation condition is satisfied. The generated second prompt is input to the generative model. This reduces the burden of verifying and / or correcting the output result of the generative model. The generative model is typically a probabilistic model and may be referred to as a generative AI (Artificial Intelligence) model. The generative model may be a model that has been trained to obtain target data. The output data may be verified by a verification unit, and an example of the verification unit may be the error determination unit 133. Furthermore, at least a portion of the first prompt may be generated by a prompt generation unit, or may be created or edited by a user. A portion of the second prompt may be created or edited by a user. Furthermore, the data processing system may include an output unit (e.g., the output unit 135) that outputs output data of the generative model when the verification result information does not satisfy the above conditions.

[0086] The condition may include that the verification result information includes error information. The error information may include at least one of the following (a) to (d). Since the second prompt is generated based on such error information, an improvement in the success rate of error correction is expected. (a) An error message indicating an error contained in the validation result. (b) Data to be corrected, which is all or part of the output data of the generative model and is data related to errors contained in the verification results. (c) The difference between the current error information and the previous error information. (d) Verification results history.

[0087] Each verification (e.g., error determination) may be associated with (1) the verification determination serial number (e.g., which verification has been performed), (2) the result of the verification, and (3) the prompt entered immediately before the verification, and these pieces of information (1) to (3) may be stored. The processing unit may generate a second prompt based on the stored information.

[0088] An example of a generative model may be LLM151 (an example of a text generation model). An example of a first prompt may be prompt 250X for causing LLM151 to convert program code in a first programming language into program code in a second programming language. An example of a second prompt may be prompt 250Y for causing LLM151 to correct an error detected in error determination of output data from LLM151. The data to be corrected may include program code in the second programming language contained in the output data. This allows for the use of LLM151 to obtain a program whose programming language has been converted with little effort. The program code may be source code or machine code. LLM151 may be a trained model for programming language conversion, for example, a model that has learned a probabilistic correspondence between a first programming language and a second programming language.

[0089] An example of a processing unit and verification in a data processing system may be the processing unit 20 and error determination. The processing unit 20 may generate prompt 250Y using a prompt template. The prompt template may include at least one of the following first to fifth texts in addition to text instructing error correction. This makes it possible to efficiently generate prompt 250Y, which is expected to improve the success rate of error correction in programming language conversion. The first text includes text that specifies a first position and that the text set at the first position is program code related to at least the error in the output data. An example of the first text is text that includes the above-mentioned "<<program insertion portion>>". The second text includes a second position and text that specifies that the text set at the second position represents the content of the error determination. An example of the second text is text that includes the above-mentioned "<<Test code insertion portion>>". The third text includes a third position and text specifying that the text set at the third position is an error message obtained in the error determination. An example of the second text may be text including the above-mentioned "<<Test result insertion section>>". The fourth text may include a fourth position and text specifying that the text set at the fourth position is the program code before the modification. An example of the fourth text may be text including the above-mentioned "<<old program insertion portion>>". The fifth text may include a fifth position and text specifying that the text set at the fifth position is a simplified error message. An example of the fifth text may be text including the above-mentioned "<<Simplified result insertion portion>>".

[0090] The verification may include a check using one or more check tools. The error detection may be an error being detected in a check using at least one of the one or more check tools. Since checks are performed from one or more perspectives in this manner, improved verification accuracy is expected. Two or more of the multiple check tools may be executed sequentially or in parallel. The verification may end when an error is detected in a check using any of the check tools. An example of the one or more check tools may be one or more check tools 123.

[0091] Both the first programming language and the second programming language may be high-level languages. One or more check tools may include a compiler. Verification may include compiling the program code in the output data 160 into program code in a low-level language using a compiler. Verification result information (e.g., error information) may include an error message for a compilation error. By using a compiler as the check tool in this way, verification can be achieved without creating a new check tool. Note that the error determination unit 133 may be a compiler.

[0092] The one or more check tools may include a test tool. The verification may include checking, by the test tool, whether the test code is satisfied for each of one or more test cases for the output data. The verification result information (e.g., error information) may include an error message output when the test code is not satisfied for any of the test cases. Since each of the one or more test cases is checked in this manner, improvement in the accuracy of the verification is expected.

[0093] The one or more check tools may include a Lint tool. The verification unit 133 may use the Lint tool to check whether output data satisfies Lint rules. The verification result information (e.g., error information) may include an error message output when a Lint rule is not satisfied. This allows for a check separate from that performed by a compiler or test tool (e.g., a check that is more detailed than that performed by a compiler), which is expected to improve the accuracy of verification.

[0094] The multiple checking tools may include a compiler, a test tool, and a lint tool. The verification may include checking with the compiler followed by checking with the test tool, and checking with the lint tool followed by checking with the test tool. In this manner, efficient verification can be achieved. For example, if an error is detected in the check with the compiler or the test tool, the verification may end without checking with the lint tool.

[0095] The generative model may be a generative model other than a text generation model, such as an image generation model, a video generation model, or an audio generation model. For any type of generative model, at least a test tool may be provided as a checking tool. For example, if the generative model is an image generation model, verification may check whether pixel values ​​of each image region of the image as output data are expected pixel values. If an error is detected in the verification, indicating that the pixel values ​​do not match, a second prompt may be generated to correct the pixel values. Furthermore, for example, if the generative model is an audio generation model, verification may check whether the amount of noise in each time period of the audio as output data exceeds a certain amount. If an error is detected in the verification, indicating that the amount of noise exceeds a certain amount, a second prompt may be generated to reduce the amount of noise.

[0096] If the previous verification result information (e.g., error information) is better than the current verification result information (e.g., error information), the processing unit may generate a second prompt for correcting errors that were not included in the previous verification result but are included in the current verification result, based on the output data from which the previous verification result was obtained. This is expected to result in efficient error correction. Note that "the previous verification result information is better than the current verification result information" may mean that the number of errors detected in the previous verification is fewer than the number of errors detected in the current verification. Furthermore, the second prompt generated in this case may include text that represents a context in which the errors detected in the current verification are corrected and errors that were not detected in the previous verification are expected to be prevented from occurring again.

[0097] The verification may include checks using multiple check tools. For each of the multiple check tools, a prompt template corresponding to the check tool may be included. For each of the multiple check tools, if the result of the previous check is better than the result of the current check, the processing unit may generate a second prompt using the prompt template associated with the check tool to correct errors that were not detected in the previous check but were detected in the current check, based on the output data that was the subject of the previous check. Because the second prompt is generated using the prompt template corresponding to the check tool that detected the error, it is possible to generate a second prompt that is expected to improve the success rate of error correction.

[0098] If the result of the second determination is negative, the output unit may output the current output data of the generative model. In other words, if the result of the second determination is false, the second prompt may not be generated. This makes it possible to avoid a loop of automatic generation of the second prompt and input and verification from continuing unnecessarily. The continuation condition may include at least one of the following: The number of times that the same output data as the previous output data is obtained (for example, the number of program matches described above) is less than the first threshold value. The number of times the same error as that detected in the previous verification was detected (for example, the number of times the error message matches described above) is less than the second threshold.

[0099] The first prompt and / or the second prompt may be generated rule-based, or may be generated by a prompt generation model, which is a text generation model. Specifically, the processing unit may input a prompt generated using a preprocessing template into the prompt generation model, thereby obtaining the second prompt as output data of the prompt generation model. The preprocessing template may be a prompt template used to generate a prompt for the prompt generation model. The preprocessing template may include text representing a context as at least one of the purpose of the prompt as output data of the prompt generation model and the role of the prompt generation model based on the purpose. This is expected to enable efficient generation of the second prompt. [Explanation of symbols]

[0100] 100: Programming language conversion device, 131: Prompt generation unit, 132: LLM interface unit, 133: Error determination unit, 134: Determination unit, 135: Output unit, 136: User interface unit, 137: Input unit< / null> < / null>

Claims

1. an input unit that inputs verification result information, which is information about a verification result of output data for an input to a generative model of the first prompt or the second prompt; a processing unit that determines whether the verification result information satisfies a condition for generating a second prompt, and generates a second prompt for the generative model using the verification result information if the verification result information satisfies the condition; A data processing system comprising:

2. the condition includes that the verification result information includes error information; The error information includes at least one of the following (a) to (d): (a) an error message describing the error; (b) correction target data, which is all or part of the output data and is data related to the error; (c) the difference between the current error information and the previous error information, and (d) a history of verification results; 10. The data processing system of claim 1.

3. the generative model is a text generation model; the first prompt instructs the text generation model to convert program code in a first programming language into program code in a second programming language; The correction target data, which is all or a part of the output data, includes program code in a second programming language contained in the output data.

10. The data processing system of claim 1.

4. the prompt generator generates the second prompt using a prompt template; the prompt template includes at least one of first text, second text, third text, fourth text, and fifth text in addition to the text instructing correction of the error; the first text includes text that defines a first position and that the text set at the first position is program code related to at least an error in the output data; the second text includes a second position and text specifying that the text set at the second position is text that represents the content of the verification; the third text includes text specifying a third position and that the text set in the third position is an error message obtained in the verification; the fourth text includes text specifying a fourth position and that the text set at the fourth position is the program code before modification; the fifth text includes text specifying a fifth position and that the text set at the fifth position is an abbreviated error message; 4. The data processing system of claim 3.

5. the verifying includes checking with each of one or more checking tools; the verification result includes an error detected in a check using at least one of the one or more checking tools; 4. The data processing system of claim 3.

6. both the first programming language and the second programming language are high-level languages; the one or more checking tools include a compiler; the verifying includes compiling, by the compiler, the program code included in the output data into program code in a low-level language; The verification result information includes an error message for a compilation error.

6. The data processing system of claim 5.

7. the one or more checking tools include a test tool; the verifying includes checking, by the test tool, whether test code is satisfied for each of one or more test cases for the output data; The verification result information includes an error message output when a test code is not satisfied for any of the test cases.

6. The data processing system of claim 5.

8. the one or more checking tools include a Lint tool; the verification includes checking, by a Lint tool, whether the output data satisfies a Lint rule; The verification result information includes an error message output due to a Lint rule not being satisfied.

6. The data processing system of claim 5.

9. the plurality of checking tools include a compiler, a test tool, and a Lint tool; the verification includes performing a check by the compiler followed by a check by the test tool, and performing a check by the lint tool followed by a check by the test tool; 6. The data processing system of claim 5.

10. When the previous verification result information is better than the current verification result information, the prompt generation unit generates a second prompt for correcting an error that was not included in the previous verification result but was included in the current verification result, based on the output data from which the previous verification result was obtained.

10. The data processing system of claim 1.

11. The previous verification result information being better than the current verification result information means that the number of errors detected in the previous verification is smaller than the number of errors detected in the current verification.

11. The data processing system of claim 10.

12. The verification from which the verification result was obtained includes checks using a plurality of check tools; for each of the plurality of checking tools, a prompt template corresponding to the checking tool; For each of the plurality of check tools, if the result of the previous check is better than the result of the current check, the prompt generation unit generates a second prompt using a prompt template associated with the check tool to correct an error that was not detected in the previous check but was detected in the current check, based on the output data that was the target of the previous check.

11. The data processing system of claim 10.

13. The conditions include at least one of the following: The number of times that the same output data as the previous output data is obtained is less than a first threshold value; The number of times the same error as that detected in the previous verification is detected is less than a second threshold value; 10. The data processing system of claim 1.

14. the prompt generation unit inputs a prompt generated using a preprocessing template into a prompt generation model, which is a text generation model, to obtain the second prompt as output data of the prompt generation model; the preprocessing template is a prompt template used to generate a prompt for the prompt generation model; the preprocessing template includes text that represents a context as at least one of a purpose of a prompt as output data of the prompt generation model and a role of the prompt generation model based on the purpose; 10. The data processing system of claim 1.

15. an output unit that outputs output data of the generative model when the verification result information does not satisfy the condition; 10. The data processing system of claim 1, further comprising:

16. (a) inputting verification result information, which is information regarding the verification result of output data for the input to the generative model of the first prompt or the second prompt; (b) determining whether the verification result information satisfies a condition for generating a second prompt; (c) if the verification result information satisfies the condition, generating a second prompt for the generative model using the verification result information, and performing (a). A data processing method performed by a computer.

17. (a) inputting verification result information, which is information regarding the verification result of output data for the input to the generative model of the first prompt or the second prompt; (b) determining whether the verification result information satisfies a condition for generating a second prompt; (c) if the verification result information satisfies the condition, generating a second prompt for the generative model using the verification result information, and performing (a). A computer program that causes a computer to do something.

Images