Code error correction method and system based on large language model

Abstract

The invention discloses a code error correction method and system based on a large language model, and the method comprises the steps: constructing and training a cue word optimization model which is used for optimizing an input initial code error correction cue word, and outputting the optimized code error correction cue word; a code error correction model is constructed and trained, and the code error correction model is used for correcting the input error code and outputting a corrected code; obtaining an error code input by a user and description content of an error, generating an initial code error correction cue word, and then inputting the initial code error correction cue word into the cue word optimization model, and inputting the optimized code error correction cue word output by the cue word optimization model, an error code provided by a user and description content of an error into a code error correction model so as to obtain a correction code output by the code error correction model. The invention relates to the technical field of computers, can realize an automatic code error correction function based on a large language model technology, effectively reduces the model training cost, and improves the error correction accuracy.

Description

Technical Field

[0001] The present invention relates to a code error correction method and system based on a large language model, and relates to the field of computer technology. Background Art

[0002] In the process of programming teaching, teachers and teaching assistants often need to spend a lot of time helping students solve programming problems, which is laborious and tiring. In this process, students often feel frustrated, which is not conducive to the development of teaching. Since the advent of the big model era, improving the quality of programming education based on big model technology has always been an important topic. High-quality completion of automated code error correction tasks will improve the quality of programming education and teaching experience.

[0003] Due to the transparency problem of large language models, technicians in this field generally agree that its internal mechanism is not yet clear, but black-box verification of its performance is completely feasible and convincing. At present, in the adjacent field of code generation, there are already evaluation standards generally recognized by technicians such as HumanEva l, but in the field of code error correction, there is no common evaluation mechanism for the time being. There are currently a certain number of immature evaluation mechanisms in this field, but they are usually based on engineering code and focus on solving problems encountered in engineering based on project code. They cannot be applied to other scenarios such as programming learning, and they are not universal, and cannot fully test the model's code error correction level.

[0004] For code error correction questions and answers under the framework of large models, there are generally two ways: retraining and no retraining. Solutions based on retraining generally have problems such as huge training costs and high initial operating costs; solutions based on non-retraining generally have problems such as difficult to measure the effect and advantage and high reliance on the basic model.

[0005] Therefore, how to realize the automatic code error correction function based on large language model technology, effectively reduce the model training cost and improve the error correction accuracy has become a technical issue that technical personnel have focused on. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a code correction method and system based on a large language model, which can realize the automatic code correction function based on the large language model technology, and effectively reduce the model training cost and improve the correction accuracy.

[0007] In order to achieve the above object, the present invention provides a code error correction method based on a large language model, comprising:

[0008] Step 1: Build and train a prompt word optimization model, which is used to optimize the input initial code error correction prompt words and output the optimized code error correction prompt words;

[0009] Step 2: Build and train a code error correction model, which is used to correct the input error code and output the corrected code, i.e., the corrected code;

[0010] Step three: obtain the error code and the description of the error input by the user, generate an initial code correction prompt word, and then input the initial code correction prompt word into the prompt word optimization model, and then input the optimized code correction prompt word output by the prompt word optimization model, the error code provided by the user, and the description of the error into the code correction model, so as to obtain the correction code output by the code correction model.

[0011] In order to achieve the above object, the present invention also provides a code error correction system based on a large language model, comprising:

[0012] A prompt word optimization model device is used to construct and train a prompt word optimization model, the prompt word optimization model is used to optimize the input initial code error correction prompt word, and output the optimized code error correction prompt word;

[0013] A code error correction model device is used to construct and train a code error correction model, the code error correction model is used to correct the input error code and output the corrected correct code, that is, the corrected code;

[0014] The code correction interactive device is used to obtain the error code and the description of the error input by the user, generate an initial code correction prompt word, and then input the initial code correction prompt word into a prompt word optimization model, and then input the optimized code correction prompt word output by the prompt word optimization model, the error code provided by the user and the description of the error into the code correction model, so as to obtain the correction code output by the code correction model.

[0015] In order to achieve the above object, the present invention further provides a computing device, comprising:

[0016] Memory and processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the code error correction method based on the large language model are implemented.

[0018] In order to achieve the above-mentioned object, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the code error correction method based on the large language model are implemented.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention first obtains optimized code correction prompt words through a prompt word optimization model, and then obtains the corrected code through the code correction model, which can effectively reduce the model training cost and improve the accuracy of automatic code correction; the present invention also optimizes the code correction prompt words by calculating the evaluation score, so as to have higher accuracy in answering questions. At the same time, the evaluation score also quantitatively evaluates the ability of the code correction model to perform code correction tasks in the context of a programming course; the present invention can also further provide users with intelligent resource recommendation functions on the basis of code correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of a code error correction method based on a large language model shown in an exemplary embodiment of the present invention.

[0021] Figure 2 An exemplary embodiment of the present invention is shown Figure 1 In step 1, the processing flow chart of the prompt word optimization model.

[0022] Figure 3 It is a structural diagram of a code error correction system based on a large language model shown in an exemplary embodiment of the present invention.

[0023] Figure 4 It is a schematic diagram of the structure of a computer device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, the present invention provides a code error correction method based on a large language model, comprising:

[0026] Step 1: Build and train a prompt word optimization model, which is used to optimize the input initial code error correction prompt words and output the optimized code error correction prompt words;

[0027] Step 2: Build and train a code error correction model, which is used to correct the input error code and output the corrected code, i.e., the corrected code;

[0028] Step three: obtain the error code and the description of the error input by the user, generate an initial code correction prompt word, and then input the initial code correction prompt word into the prompt word optimization model, and then input the optimized code correction prompt word output by the prompt word optimization model, the error code provided by the user, and the description of the error into the code correction model, so as to obtain the correction code output by the code correction model.

[0029] The prompt word optimization model and the code error correction model can be implemented based on the large language model.

[0030] As an optional implementation, Figure 2 As shown, in step 1, the processing of the prompt word optimization model may further include:

[0031] Step 11: Calculate the evaluation score of the input initial code error correction prompt word, and then input the initial code error correction prompt word and its evaluation score into the large language model, and the large language model outputs the optimized code error correction prompt word;

[0032] Exemplarily, the initial code error correction prompt words may include "building a scene of a programming competition", "playing the role of a programming teacher who can always correct errors correctly", "ensuring that the code can be compiled", "required to keep the original format as much as possible", and performing chain of thought (CoT) processing, that is, "required to decompose the problem and conceive a solution to this specific problem together to ensure that the correct answer is obtained", etc., which are not specifically limited in the embodiments of the present invention;

[0033] Step 12: Calculate the evaluation score of the optimized code correction prompt word output by the large language model, and determine whether the evaluation score is greater than a certain threshold. If so, it means that the optimized code correction prompt word output by the large language model is the optimal code correction prompt word, and the prompt word optimization model outputs the optimal code correction prompt word, and this process ends; if not, save the optimized code correction prompt word of the large language model and its evaluation score as a historical optimization record, and then extract multiple historical optimization records (including multiple optimized code correction prompt words and their evaluation scores) to continue to input the large language model, so as to obtain the optimized code correction prompt word that the large language model continues to output, and finally go to step 12.

[0034] As an optional implementation, the present invention may also form a test data set, and calculate the evaluation score of the code error correction prompt word according to the test samples in the test data set, including:

[0035] Get some homework data, including the description of the topic, submission status, the combination of correct code and incorrect code actually submitted by the user for this topic, each group of test input and output use cases corresponding to the topic, etc. Then group and clean the homework data according to the programming language, remove the interfering or polluted data, and randomly select some data as test samples to form a test data set. Here, the homework data obtained can belong to any programming language, such as C, C++, Java, C#, Python, JS, etc., can be from the homework assigned and uniformly submitted by various programming courses, can also come from online question banks and assessment systems, and programming homework topics can be in Chinese, English or other languages.

[0036] Thus, calculating the evaluation score of the code error correction prompt word may further include:

[0037] Step A1: select several test samples from the test data set, and then input the code correction prompt words, the erroneous code contained in each test sample and the target function description into the large language model one by one, and the large language model outputs the correction code corresponding to each test sample;

[0038] The target function is a piece of text that describes the specific function that the program is expected to perform without errors, which represents the correct behavior expected by the program. The incorrect code is a version of the correct code that contains one or more errors, such as syntax errors, logic errors, or other problems that hinder the function of the code.

[0039] It is worth noting that because users need to obtain both correction codes and correction explanations in real environments, when evaluating the quality of correction codes, the large language model is required to complete the generation of correction codes and correction explanations at the same time. Then, regular expressions are used to filter out the code part in the MarkDown code identifier (```.?[a-zA-Z]+), i.e., the correction code, from the output of the large language model.

[0040] Step A2: Pair each group of test input data with the correction code and the correct code of each test sample one by one and input them into the verification system to obtain the output results generated after the correction code and the correct code are executed respectively. The verification system can be an online program evaluation system (OJ) or an equivalent automatic code compilation and running tool, such as gcc;

[0041] Step A3: Compare the output results of the correction code and the correct code of each test sample to obtain the evaluation score of each test sample: if the output results are completely consistent, the evaluation score is 100 points; otherwise, the evaluation score is 0 points, and then calculate the evaluation score of the code correction prompt word, that is, the average evaluation score of all test samples.

[0042] As a further optional implementation, when constructing the test data set, erroneous codes can be generated by actively creating errors, thereby expanding the scope of the test data set and increasing the effectiveness of the test set, including: inputting correct codes into a large language model, and having the large language model create erroneous codes based on the correct codes, and then using the erroneous codes output by the large language model as test samples and adding them to the test data set.

[0043] It can be understood that the code correction prompt words obtained after optimization by the present invention have higher answering accuracy and educational guidance significance than the large language model itself, the large model system written for application development, or the code correction prompt words manually written by ordinary technicians in this field; at the same time, the evaluation score quantitatively evaluates the ability of the code correction model to perform code correction tasks in the context of programming courses.

[0044] The code error correction model may select a large language model with text generation capability, which may be a general large language model or a specific model pre-trained for a code scenario, and the embodiment of the present invention does not specifically limit this.

[0045] It is worth noting that in the era of rapid development of big models, the base big model version switching is very frequent. Therefore, as an optional implementation method, this method separates the part related to the big model version from other processing logic to facilitate version iteration of the system based on this method.

[0046] As an optional implementation, in step 3, after obtaining the correction code output by the code error correction model, the following may also be included:

[0047] Step B1, displaying the error code and the correction code input by the user to the user through a comparison view;

[0048] Step B2: When the user selects several code entries in the comparison view, the optimized code correction prompt word output by the prompt word optimization model, the error code and the description of the error input by the user, and the several code entries selected by the user are further input into the code correction model, thereby obtaining the correction code output by the code correction model. The present invention can continue to correct errors with the several code entries selected by the user as the focus.

[0049] As an optional implementation, step B1 may further include:

[0050] The error code and correction code input by the user are formatted so that the error code and the correction code are in the same format, and then the error code and the correction code after the code formatting are diffed to obtain the comparison data of the error code and the correction code, so as to intuitively display the specific modification content to the user. Among them, the specific implementation method of formatting the error code and the correction code input by the user can be: obtaining the AST representation of the error code and the correction code, traversing and collecting the AST, and using the textwrap open source tool to process the blank elements, and then using the YAPF method (Python), or calling Asty le (C, C++, Java, C#, JS) for formatting.

[0051] As an optional implementation, the present invention can also provide users with intelligent resource recommendation functions based on the correction code output by the code error correction model and the specific description of the error by the user, and also includes:

[0052] Step C1: Obtain business parameters and configuration information from the configuration information layer, complete the construction of the search service base, and start it;

[0053] Based on I Tunnel, TCP connection is established with Red is used as the base to establish an IP pool. For the accumulated SERP requests, the search engine request is first made with its own IP within 30 seconds, and the successful data is returned in sequence. When failures begin to occur, the IP pool is enabled. Preferably, a maximum of 10 IPs are obtained within 30 seconds, and the number of exemption requests supported by the tunnel is 5.

[0054] Step C2: connect to a headless browser based on Chromedp technology, obtain the results returned by the search engine, obtain the search results by regular expression matching, flatten the text, and use it as the return status of the search service base. Preferably, return the data to other user-side services through grpc;

[0055] Step C3, constructing a keyword extraction prompt: inputting the user's description of the error and the correction code output by the code error correction model into the large language model, thereby obtaining a number of keywords output by the large language model;

[0056] Keyword extraction tips may include key descriptions such as "Please extract key information from the content I provide and tell me which keywords to enter in the search box of the search engine to obtain the most relevant and eye-catching search results", "The answer will be directly written into the search box of the search engine, so please only give all the search keywords in '[]'"; where '[]' is a separator for the optimization of domestic large model characteristics;

[0057] Step C4: input a number of keywords output by the large language model into a search service base, and then divide the search results output by the search service base into graphic results and video results, and then push them to the user;

[0058] Step C5, record the user's click data on each search result. Specifically, parse the title and link of each search result into multi-dimensional features (the analyzed subject, title, link domain name, link key information, and other link features), use a vector database to record the links clicked by users, and then count the number of clicks on each search result by all users, and sort the image, text, and video results accordingly, so as to make recommendations to users.

[0059] like Figure 3 As shown, the present invention provides a code error correction system based on a large language model, comprising:

[0060] A prompt word optimization model device is used to construct and train a prompt word optimization model, the prompt word optimization model is used to optimize the input initial code error correction prompt word, and output the optimized code error correction prompt word;

[0061] A code error correction model device is used to construct and train a code error correction model, the code error correction model is used to correct the input error code and output the corrected correct code, that is, the corrected code;

[0062] The code correction interactive device is used to obtain the error code and the description of the error input by the user, generate an initial code correction prompt word, and then input the initial code correction prompt word into a prompt word optimization model, and then input the optimized code correction prompt word output by the prompt word optimization model, the error code provided by the user and the description of the error into the code correction model, so as to obtain the correction code output by the code correction model.

[0063] See also Figure 4 , Figure 4 4 is a block diagram of a computing device 400 shown in an exemplary embodiment of this specification. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0064] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0065] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0066] The computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 400 may also be a mobile or stationary server or a cloud server, etc.

[0067] The processor 420 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned code error correction method based on the large language model.

[0068] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the code error correction method based on the large language model belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the code error correction method based on the large language model.

[0069] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned code error correction method based on a large language model.

[0070] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the above-mentioned code error correction method based on a large language model belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned code error correction method or system based on a large language model.

[0071] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned code error correction method based on a large language model.

[0072] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the code error correction method based on the large language model belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the code error correction method or system based on the large language model.

[0073] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0074] The computer instructions include computer program codes, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0075] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A code error correction method based on a large language model, characterized in that: Included are: Step 1: Build and train a prompt word optimization model, which is used to optimize the input initial code error correction prompt words and output the optimized code error correction prompt words; Step 2: Build and train a code error correction model, which is used to correct the input error code and output the corrected code, i.e., the corrected code; Step three: obtain the error code and the description of the error input by the user, generate an initial code correction prompt word, and then input the initial code correction prompt word into the prompt word optimization model, and then input the optimized code correction prompt word output by the prompt word optimization model, the error code provided by the user, and the description of the error into the code correction model, so as to obtain the correction code output by the code correction model.

2. The method according to claim 1, characterized in that In step 1, the processing of the prompt word optimization model further includes: Step 11: Calculate the evaluation score of the input initial code error correction prompt word, and then input the initial code error correction prompt word and its evaluation score into the large language model, and the large language model outputs the optimized code error correction prompt word; Step 12: Calculate the evaluation score of the optimized code correction prompt word output by the large language model, and determine whether the evaluation score is greater than a certain threshold. If so, it means that the optimized code correction prompt word output by the large language model is the optimal code correction prompt word, and the prompt word optimization model outputs the optimal code correction prompt word, and this process ends; if not, save the optimized code correction prompt word of the large language model and its evaluation score as a historical optimization record, and then extract multiple historical optimization records to continue to input the large language model, so as to obtain the optimized code correction prompt word that the large language model continues to output, and finally go to step 12.

3. The method according to claim 2, characterized in that Also included are: Get some homework data, including the description of the topic, submission status, the combination of correct and incorrect codes submitted by users for this topic, and the corresponding test input and output cases. Then group and clean the homework data according to the programming language, remove the interfering or contaminated data, and randomly select some data as test samples to form a test data set. Thus, calculating the evaluation score of the code error correction prompt word further includes: Step A1: select several test samples from the test data set, and then input the code correction prompt words, the erroneous code contained in each test sample and the target function description into the large language model one by one, and the large language model outputs the correction code corresponding to each test sample; Step A2: Pair each set of test input data with the correction code and correct code of each test sample one by one and input them into the verification system to obtain output results generated after the correction code and the correct code are executed respectively; Step A3: Compare the output results of the correction code and the correct code of each test sample to obtain the evaluation score of each test sample: if the output results are completely consistent, the evaluation score is 100 points, otherwise the evaluation score is 0 points, and then calculate the evaluation score of the code correction prompt word, that is, the average evaluation score of all test samples.

4. The method according to claim 3, characterized in that When constructing the test data set, it also includes: The correct code is input into the large language model, and the large language model generates an incorrect code based on the correct code. The incorrect code output by the large language model is then used as a test sample and added to the test data set.

5. The method according to claim 1, characterized in that: In step 3, after obtaining the corrected code output by the code error correction model, the following steps are also included: Step B1, displaying the error code and the correction code input by the user to the user through a comparison view; Step B2: When the user selects several code entries in the comparison view, the optimized code correction prompt word output by the prompt word optimization model, the error code and the description of the error input by the user, and the several code entries selected by the user are further input into the code correction model to obtain the corrected code output by the code correction model.

6. The method according to claim 5, characterized in that Step B1 further comprises: The error code and correction code input by the user are formatted so that the error code and the correction code are in the same format, and then a diff operation is performed on the formatted error code and correction code to obtain comparison data of the error code and the correction code.

7. The method according to claim 1, characterized in that Also included are: Step C1: Obtain business parameters and configuration information from the configuration information layer, complete the construction of the search service base, and start it; Step C2: connect with a headless browser and obtain the results returned by the search engine. Obtain the search results by matching regular expressions. Flatten the text and use it as the return of the search service base. Step C3, constructing a keyword extraction prompt: inputting the user's description of the error and the correction code output by the code error correction model into the large language model, thereby obtaining a number of keywords output by the large language model; Step C4: input a number of keywords output by the large language model into a search service base, and then divide the search results output by the search service base into graphic results and video results, and then push them to the user; Step C5: record the click data of each search result by the user, then count the clicks of each search result by all users, and sort the image, text and video results accordingly, so as to make recommendations to the user.

8. A code error correction system based on a large language model, characterized in that: Included are: A prompt word optimization model device is used to construct and train a prompt word optimization model, the prompt word optimization model is used to optimize the input initial code error correction prompt word, and output the optimized code error correction prompt word; A code error correction model device is used to construct and train a code error correction model, the code error correction model is used to correct the input error code and output the corrected correct code, that is, the corrected code; The code correction interactive device is used to obtain the error code and the description of the error input by the user, generate an initial code correction prompt word, and then input the initial code correction prompt word into a prompt word optimization model, and then input the optimized code correction prompt word output by the prompt word optimization model, the error code provided by the user and the description of the error into the code correction model, so as to obtain the correction code output by the code correction model.

9. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the code error correction method based on a large language model according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the code error correction method based on a large language model as described in any one of claims 1 to 7.

Images