Code verification device and code verification method

Abstract

Verify the validity of code reviews. [Solution] The code verification device 10 has a defect introduction unit 23 that generates contaminated code 13 by introducing a defect into generated code 11, a code review unit 24 that obtains review code 14 that has been corrected by reviewing the contaminated code 13, a review judgment unit 25 that calculates the detection rate of defects introduced into the contaminated code 13 based on the difference between the review code 14 and the contaminated code 13, and generates information that verifies the validity of the code review based on the defect detection rate, and an output unit 27 that outputs the information that verifies the validity of the code review.

Description

[Technical Field]

[0001] The present disclosure relates to a code verification device and a code verification method. [Background technology]

[0002] Generative AI (Artificial Intelligence) is applied to various fields. For example, Patent Document 1 is a document relating to a specific information processing device and a control method thereof. Patent Document 1 discloses a technology that aims to improve operability by presenting appropriate information and functions according to the user's operation history and situation. In particular, it is characterized by its ability to infer the user's intentions and efficiently provide necessary information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2024-117447 Summary of the Invention [Problem to be solved by the invention]

[0004] The output data automatically generated by generation AI such as that in Patent Document 1 is often still not of sufficient quality to be used as a deliverable product or the like. Therefore, we consider conducting code reviews on automatically generated code and correcting the code as appropriate before using it. The corrected code is expected to be improved to a higher quality. However, if the validity of the code review is low, the quality of the corrected code cannot be sufficiently improved. The validity of a code review can also be said to be the certainty or reliability of the code review. For example, a review that fixes many of the defects contained in the code is a highly valid review.

[0005] Therefore, the main purpose of this disclosure is to verify the validity of code reviews. [Means for solving the problem]

[0006] The code verification device of the present disclosure includes: a defect injection unit that generates an injected code by injecting a defect into a target code; a code review unit that acquires a review code corrected by reviewing the mixed code; a review determination unit that calculates a detection rate of defects mixed into the mixed code based on a difference between the review code and the mixed code, and generates information verifying the validity of the code review based on the detection rate of the defects; and an output unit that outputs information verifying the validity of the code review. Other means will be described later. [Effects of the Invention]

[0007] According to the present disclosure, the validity of a code review can be verified. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of a code verification system according to an embodiment of the present invention. [Figure 2] FIG. 10 is an explanatory diagram showing an example of an output code according to the present embodiment. [Figure 3] 3 is an explanatory diagram showing an example of a mixing code that is the source of generating the output code of FIG. 2 according to this embodiment. [Figure 4] 4 is a flowchart illustrating an operation of the code verification device according to the present embodiment. [Figure 5] 10 is an output screen in the case where the test is judged as passed according to this embodiment. [Figure 6] 10 is an output screen when a failure determination is made in this embodiment. [Figure 7] FIG. 2 is a hardware configuration diagram of each device of the code verification system according to the present embodiment. [Figure 8]FIG. 2 is a sequence diagram showing the operation of the code verification system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the present embodiment will be described with reference to the drawings.

[0010] FIG. 1 is a diagram showing the configuration of a code verification system 100. The code verification system 100 includes a code verification device 10 and an operation terminal 30. The code verification device 10 and the operation terminal 30 may be separate devices connected to a network as shown in Fig. 1, or may be housed in the same housing. The operation terminal 30 is operated by a user to input instructions to the code verification device 10 and receive results from the code verification device 10. The code verification device 10 has, as processing units, a code generation unit 21, a defect generation unit 22, a defect introduction unit 23, a code review unit 24, a review determination unit 25, a defect resolution unit 26, and an output unit 27. The code verification device 10 also has a storage unit that stores, as data to be processed by the processing units, generated code 11, defect information 12, introduced code 13, and review code 14.

[0011] The code (program, etc.) handled by the code verification device 10 will be explained below by state. The generated code 11 is generated by a generation AI or a human developer. This generated code 11 may contain accidental defects. The mixed code 13 is code in which an intentional defect is mechanically mixed into the generated code 11 based on the defect information 12 prepared in advance. Review code 14 is the modified code in which some defects have been removed from the contaminated code 13 through code review by the generation AI or a human reviewer. The output code 15 is a code obtained by mechanically removing intentional defects from the review code 14, and is output to the operation terminal 30 as a deliverable.

[0012] For example, the code generation unit 21 inputs the following design document as a prompt to the generation AI, and the generation AI outputs the generated code 11 in accordance with the design document. -Create a code that outputs the sales price for the sales year when you enter the price excluding tax and the sales year. The sales price will be the tax-exclusive price plus the consumption tax for the year of sale. The price excluding tax is stored in cell A1, the sales year in cell A2, and the sales price in cell A3.

[0013] FIG. 2 is an explanatory diagram showing an example of the output code 15. As shown in FIG. Output code 15 is VBA (Visual Basic for Applications) code for Microsoft Excel (registered trademark). Code 210 calculates the sales price based on the consumption tax rate for the sales year from the tax-excluded price entered in cell A1 and the sales year entered in cell A2, and outputs it to cell A3. To do this, code 210 uses four variables (taxExPrice, salesYear, salesPrice, taxRate).

[0014] FIG. 3 is an explanatory diagram showing an example of the mixed code 13 from which the output code 15 of FIG. 2 is generated. The differences between the output code 15 in FIG. 2 and the mixed code 13 in FIG. 3 will be described below. As an example of an intentional defect mixed into the mixed code 13, the conditional branch instruction "Select" was replaced with the misspelling "SeRect" at points 221A and 221B. This defect was mixed in by referring to the "misspelling" in the defect information 12.

[0015] As an example of an intentional defect mixed into the mixed code 13, in point 223, the output cell A3 for the sales price was mistaken for the input cell A2. This defect was mixed in by referring to the "incorrect variable selection" in defect information 12. As an example of an accidental defect that has been introduced into generated code 11, in section 222, the logic for applying the revised tax rate (0.10) from fiscal year 2019 is not described. This defect was introduced because the generator of generated code 11 was unaware of the tax rate revision from fiscal year 2019.

[0016] Returning to FIG. 1, the main part of the code verification device 10 has the following features. The defect-injecting unit 23 generates an injected code 13 by injecting a defect into the generated code 11. The code review unit 24 acquires the review code 14 corrected by reviewing the mixed code 13. The review determination unit 25 calculates the detection rate of defects mixed into the mixed code 13 based on the difference between the review code 14 and the mixed code 13, and generates information verifying the validity of the code review based on the defect detection rate. The output unit 27 outputs information verifying the validity of the code review. Each processing unit in FIG. 1 will be explained below with reference to the flowchart in FIG.

[0017] FIG. 4 is a flowchart showing the operation of the code verifying device 10. The code generation unit 21 causes a human (developer) or a generation AI to generate generated code 11, which is code to be reviewed (S101). The defect-injecting unit 23 generates the mixed code 13 by injecting a defect into the generated code 11 (S102). For this purpose, the defect-injecting unit 23 inputs a prompt based on the following defect information 12 to the generation AI, for example, so that the generation AI outputs the mixed code 13 in accordance with the defect information 12. Spelling mistakes (such as misspelling a statement in code) - Variable selection error (e.g., selecting the wrong variable as the variable referenced as an argument in a statement)

[0018] As a preliminary step, the defect generation unit 22 generates defect information 12 based on the databased history of past code reviews (defect training data), for example, by inputting the following prompt into the generation AI. - Extract the combination of before and after fixes as defects from the history of past code reviews. - Classify the extracted defects into categories such as spelling mistakes, and output the classification of defects that occur frequently as Defect Information 12.

[0019] The code review unit 24 obtains corrected review code 14 by having a human (reviewer) or a generation AI review the contaminated code 13 (S103). The generation AI with a code review function is implemented as, for example, GitHub Copilot.

[0020] Expert reviewer A conducted the following review of contamination code 13 in Figure 3. -An error in variable selection (point 223) was pointed out and corrected (Cell A2 → Cell A3). -Spelling mistakes (points 221A and 221B) were pointed out and corrected (correction of SeRect → Select). -Insufficient logic for applying the revised tax rate (0.10) from fiscal year 2019 was pointed out, and the correct content was added (added processing for Case 2019 To 9999). As a result, the mixed code 13 in FIG. 3 was corrected to the output code 15 in FIG. 2 (still the review code 14 at the step S103).

[0021] Novice reviewer B performed the following review on mixed code 13 in Figure 3. · The error in variable selection (point 223) was not noticed and not pointed out. -Spelling mistakes (points 221A and 221B) were pointed out and corrected (correction of SeRect → Select). The company did not notice or point out the lack of logic in applying the revised tax rate (0.10) from fiscal year 2019.

[0022] The review determination unit 25 verifies the review result (validity of the code review) based on the review code 14 and the review content (S104). Specifically, the review determination unit 25 calculates the defect detection rate as the percentage of defects that were correctly pointed out as defects (code correction) using the review code 14 and its review content among the defects introduced by the defect introduction unit 23 in S102. For example, an experienced reviewer A correctly points out two defects (an incorrect variable selection and a spelling error) introduced by the defect-introducing unit 23. Therefore, the review judgment unit 25 calculates the defect detection rate as "2 / 2 = 100%." On the other hand, novice reviewer B correctly points out only one of the two defects (a spelling error) as a defect. Therefore, review determination unit 25 calculates the defect detection rate as "1 / 2=50%."

[0023] The review judgment unit 25 judges whether the review result is pass or fail depending on whether the defect detection rate calculated in S104 is equal to or higher than a predetermined pass / fail judgment line (for example, 80%) (S105). The review judgment unit 25 judges that the expert reviewer A has passed because the defect detection rate is 100%, and that the novice reviewer B has failed because the defect detection rate is 50%.

[0024] If the review result is passed (Yes in S105), the output unit 27 outputs the review code 14 to the operation terminal 30. Alternatively, the output unit 27 may output the output code 15 instead of outputting the review code 14. In this case, the defect resolution unit 26 acquires the output code 15 obtained by removing (resolving) the defect introduced by the defect introduction unit 23 in S102 from the defects included in the review code 14, and outputs the output code 15 to the operation terminal 30 (S106). Therefore, the defect resolution unit 26 causes the output code 15 to be output by, for example, inputting the following prompt to the generation AI: Last time, we asked you to generate mixed code 13 by mixing the defect in defect information 12 into generated code 11. This time, please remove the defect in defect information 12 that was mixed into mixed code 13 for review code 14.

[0025] As a result, the output code 15 is expected to be a high-quality code with defects eliminated, as follows: Any accidental defects that occur during the process of the code generation unit 21 generating the generated code 11 are likely to be resolved by the expert reviewer A. This is because the expert reviewer A is expected to provide a reliable code review, since the defect detection rate is the acceptance standard. Any intentional defects that occur in the process of generating the mixed code 13 by the defect introduction unit 23 are reliably eliminated by the defect elimination unit 26. For example, a defect such as an incorrect variable selection that was not pointed out by novice reviewer B is automatically eliminated.

[0026] If the review result is unsuccessful (No in S105), the review determination unit 25 returns the process to S103 and causes the code review unit 24 to review the mixed code 13 again. Here, the output unit 27 displays the verification details of the current review result (information on detected defects and information on undetected defects) on a screen (FIG. 6) and provides feedback to the reviewer, thereby providing clues for re-review. The output unit 27 may also display the verification details of successful review results on a screen (FIG. 5) and provide feedback to the reviewer.

[0027] The code verification device 10 has been described above based on the operation in Fig. 4. As exemplified below, an individual generation AI (a generation AI with individually different teacher data) is selected in each process. A generation AI (first generation AI) suitable for code generation is selected as the generation AI used when the code generation unit 21 generates the generation code 11 by inputting the design document for the generation code 11 (S101). The generation AI (second generation AI) used in common when the defect introduction unit 23 introduces a defect into the code by inputting the generated code 11 and the defect information 12 (S102) and when the defect resolution unit 26 removes a defect from the code by inputting the review code 14 (S106) is a generation AI that is suitable for code editing. The same generation AI may be used in multiple processes, and different prompts may be input to the same generation AI in each process.

[0028] FIG. 5 shows the output screen when the test is judged as passed (Yes in S105). The output unit 27 outputs an output screen including the following information to the operation terminal 30. A pass / fail result such as "Pass! Outputs code with the defect removed." -Defect detection rate = 90% Number of detected defects by type (spelling error: 1, variable selection error: 3) This allows the expert reviewer A to confirm that the content of his / her review is appropriate. The defect detection rate and the pass / fail result based on the defect detection rate are examples of information that indicate the appropriateness of the code review.

[0029] FIG. 6 shows the output screen when the test is judged as unsuccessful (No in S105). The output unit 27 outputs an output screen including the following information to the operation terminal 30. · Pass / fail results such as "Failed. Please try to find other defects." -Defect detection rate = 30% Number of detected defects by type (spelling error: 1) That is, if the validity of the code review is not acceptable, the output unit 27 outputs information verifying the validity of the code review, including the defect detection rate. Furthermore, the output unit 27 outputs the number of defects corrected by review for each type of defect as information verifying the validity of the code review. This allows novice reviewer B to identify areas of oversight (weaknesses) in the content of his or her reviews.

[0030] 5 and 6, the output screen may also include information on the number of undetected defects by type. Furthermore, the output unit 27 may create a database of review content (correction differences between the mixed code 13 and the review code 14) and review content verification results (information included in the output screens of FIGS. 5 and 6) as a review database (not shown). This review database can be used to build a generative AI for reviews or to train reviewers. Furthermore, by cloud-based deployment of this review database and increasing the number of users, the amount of learning will increase, and it is expected that the accuracy of static code analysis by the generative AI will improve.

[0031] FIG. 7 is a diagram showing the hardware configuration of each device in the code verification system 100. Each device (code verification device 10, operation terminal 30) of the code verification system 100 is configured as a computer 900 having a CPU 901, a RAM 902, a ROM 903, a HDD 904, a communication I / F 905, an input / output I / F 906, and a media I / F 907. The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from a recording medium 917. Furthermore, the CPU 901 controls each processing unit by executing a program (also called an application or an app for short) loaded into the RAM 902. This program can also be distributed via a communication line or recorded on a recording medium 917 such as a CD-ROM and distributed.

[0032] FIG. 8 is a sequence diagram showing the operation of the code verification system 100. In order to request the code generation unit 21 to generate code (S201), the operation terminal 30 transmits a design document (prompt) to be referenced for code generation to the code generation unit 21 (S202). The code generating unit 21 generates the generated code 11 in accordance with the design document (S203), and returns the generated code 11 to the operation terminal 30 (S204).

[0033] The operation terminal 30 transmits the generated code 11 to the defect introducing unit 23 (S212) in order to request the defect introducing unit 23 to intentionally introduce a defect into the generated code 11 (S211). The defect introducing unit 23 generates the mixed code 13 by mixing the defect into the generated code 11 based on the defect information 12 (S213), and returns the mixed code 13 to the operation terminal 30 (S214).

[0034] The operation terminal 30 causes a human (reviewer) or a machine (generating AI) to review the mixed code 13, and acquires the result as the review code 14 (S221). The operation terminal 30 transmits the review code 14 and the review content thereof to the review determination unit 25 (S223) to request the mixed code 13 to determine the validity of the review code 14 (S222).

[0035] The review judgment unit 25 measures the defect detection rate of the review code 14 by comparing the difference between the mixed code 13 (and its defect information 12) used in S213 and the review code 14 (and its review content), and makes a pass / fail judgment by comparing the defect detection rate with the pass / fail judgment line (e.g., 80%) (S224). The review determination unit 25 returns the defect detection rate calculated in S224 and the pass / fail result to the operation terminal 30 (S225).

[0036] The operation terminal 30 branches depending on whether the test is passed or not (S226) in accordance with the pass / fail result returned in S225. If the answer is Yes in S226 (if the test is passed), the operation terminal 30 proceeds to S231, and if the answer is No in S226 (if the test is failed), the operation terminal 30 returns to S221. The operation terminal 30 transmits the review code 14 to the defect resolution unit 26 (S232) to request deletion of the defect mixed in S213 from the mixed code 13 (S231).

[0037] The defect resolution unit 26 generates an output code 15 from the review code 14 with the defect deleted by reversing the process in which the defect was introduced by the defect introduction unit 23 in S213 (S233), and returns the output code 15 to the operation terminal 30 (S234). The operation terminal 30 outputs the output code 15 from which the defect has been deleted from the output unit 27 by displaying it on the screen or outputting it as a file.

[0038] The code verification device 10 of this embodiment described above reviews the contaminated code 13, which is generated by the generation AI and into which defects have been automatically introduced. The review determination unit 25 then verifies the validity of the review by calculating the defect detection rate from the review code 14 after reviewing the contaminated code 13. The defect resolution unit 26 then generates the output code 15, in which the automatically introduced defects have been automatically resolved from the review code 14. This makes it possible to verify the validity of the code review and provide high-quality output code 15 with few defects. Furthermore, the output unit 27 can train novice reviewer B by displaying the validity of the code review on the screen.

[0039] The code review unit 24 may request a human or a generating AI to review the contaminated code 13. In either case, verifying the validity of the code review is important for improving the quality of the review and outputting high-quality code. The validity of this code review may be used, for example, to decide whether to accept or reject the review content, or to select which review content from multiple reviewers (which may include the generation AI) to adopt.

[0040] Furthermore, the present disclosure is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit and scope of the present disclosure as defined in the claims. For example, the above-described embodiments provide detailed and specific descriptions of the configuration of the code verification system 100 in order to clearly explain the present disclosure, and the system is not necessarily limited to systems that include all of the components described. Furthermore, it is possible to replace part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. It is also possible to add, replace, or delete other components from part of the configuration of each embodiment.

[0041] Furthermore, the above-described configurations, functions, processing units, etc. may be partially or entirely realized in hardware, for example, by designing them as integrated circuits, etc. As the hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used. Furthermore, each component of the code verification system 100 according to the above-described embodiment may be implemented in any hardware as long as the respective hardware can transmit and receive information to and from each other via a network. Furthermore, the processing performed by a certain processing unit may be realized by a single piece of hardware, or may be realized by distributed processing using multiple pieces of hardware. [Explanation of symbols]

[0042] 10 Code Verification Device 11 Generated code (target code) 12. Bug Information 13 Mixed Code 14 Review Code 15 Output Code 21 Code Generation Unit 22 Fault Generation Unit 23 Defective part 24 Code Review Department 25 Review Judgment Department 26 Troubleshooting Department 27 Output section 30 Operation terminal 100 Code Verification System

Claims

1. a defect injection unit that generates an injected code by injecting a defect into the target code; a code review unit that acquires a review code corrected by reviewing the mixed code; a review determination unit that calculates a detection rate of defects mixed into the mixed code based on a difference between the review code and the mixed code, and generates information verifying the validity of the code review based on the detection rate of the defects; and an output unit that outputs information verifying the validity of the code review. Code verification device.

2. The output unit outputs the review code when the validity of the code review is acceptable. The code verification device according to claim 1 .

3. the code verification device further includes a defect resolution unit, the defect resolution unit generates an output code by removing the defect introduced by the defect introduction unit from the review code; The output unit outputs the output code instead of the review code. The code verification device according to claim 1 .

4. the code verification device further includes a code generation unit, the code generation unit generates the target code by inputting a design document of the target code into a first generation AI; the defect-injecting unit inputs the target code and defect information into a second generation AI to generate the injected code into which a defect based on the defect information is injected; The defect resolution unit generates the output code by inputting the review code to the second generation AI. The code verification device according to claim 3 .

5. When the validity of the code review is judged to be unsatisfactory, the output unit outputs information verifying the validity of the code review, including a defect detection rate. The code verification device according to claim 1 .

6. The output unit outputs the number of defects corrected by the review for each type of defect as information verifying the validity of the code review. The code verification device according to claim 5 .

7. The code verification device includes a defect injection unit, a code review unit, a review determination unit, and an output unit, the defect-injecting unit generates a defect-injected code by injecting a defect into the target code; the code review unit acquires a review code corrected by reviewing the mixed code; the review determination unit calculates a detection rate of defects mixed into the mixed code based on a difference between the review code and the mixed code, and generates information verifying the validity of the code review based on the detection rate of the defects; The output unit outputs information verifying the validity of the code review. Code verification method.

Images