Test method and device for code generation model

Through the testing method of the code generation model, the cutting and completion mechanism is used, combined with syntax check-up and inference speed evaluation, the problem of inefficient performance evaluation of the code generation model is solved, and efficient and accurate performance evaluation is achieved.

CN119003350APending Publication Date: 2024-11-22ALIBABA (CHINA) CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411076713.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently evaluate the performance of code generation models, which usually requires manpower observation, resulting in high cost and inefficiency.

Method used

Provide a test method, by trimming the code in the dataset, generating prompt instructions, submitting it to the code generation model for completion, and comparing the original code and the completed code to obtain performance test results, including tailoring strategies and syntax Checksum reasoning speed evaluation.

Benefits of technology

It realizes efficient performance evaluation without human observation, and the results are more objective and accurate, saving labor costs, and improving the utilization and accuracy of test data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003350A_ABST
    Figure CN119003350A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a test method and device for a code generation model. The method comprises more than one round of testing, each round of testing comprises the steps that codes are obtained from a data set to serve as first codes, and the data set comprises more than one codes; cutting off part of contents in the first code to obtain a second code; generating a prompt instruction by using the second code, and providing the prompt instruction to the code generation model to obtain a third code generated by the code generation model for the second code; obtaining a fourth code by using the second code and the third code; comparing the first code with the fourth code to obtain a comparison result; and obtaining a performance test result of the code generation model by using the comparison result obtained by the more than one round of test. The testing process does not depend on manual observation, the labor cost is saved, and the testing result is more objective and accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a testing method and device for a code generation model. Background Art

[0002] The code generation model is a technology that has emerged in recent years with the rapid development of artificial intelligence and machine learning technologies. It can automatically generate code using deep learning models. The core idea is to automate programming tasks. By inputting natural language descriptions, programming requirements or partial codes, the model can automatically generate code that meets the requirements, thereby improving development efficiency and reducing the workload of manual code writing.

[0003] However, after the code generation model is trained, it is difficult to evaluate its performance. Usually, it is necessary to evaluate the performance of the model by manually observing the adoption rate, which is labor-intensive and inefficient. Summary of the invention

[0004] In view of this, the present application provides a testing method and device for a code generation model, so as to efficiently implement performance evaluation of the code generation model and save manpower costs.

[0005] This application provides the following solutions:

[0006] In a first aspect, a testing method for a code generation model is provided, the method comprising more than one round of testing;

[0007] Each round of testing includes:

[0008] Obtaining a code from a data set as a first code, the data set comprising more than one code;

[0009] Cutting off part of the content of the first code to obtain a second code;

[0010] Generate a prompt instruction using the second code, provide the prompt instruction to the code generation model, and obtain a third code generated by the code generation model for the second code;

[0011] Obtain a fourth code using the second code and the third code;

[0012] Comparing the first code and the fourth code to obtain a comparison result;

[0013] The performance test results of the code generation model are obtained by using the comparison results obtained from the one or more rounds of testing.

[0014] According to an achievable method in an embodiment of the present application, the first code is trimmed of part of its content to obtain a second code including:

[0015] Parsing the first code to construct a code tree corresponding to the first code, the code tree including a hierarchical relationship between code blocks;

[0016] Starting from a non-starting position of at least one branch in the code tree, a subsequent part or all of the content of the at least one branch is pruned.

[0017] According to an achievable manner in an embodiment of the present application, trimming a subsequent part or all of the content of at least one branch from a non-starting position of at least one branch in the code tree includes: selecting at least one code block from the code tree, and trimming the code block according to a preset trimming strategy;

[0018] The trimming strategy includes:

[0019] Starting from a non-starting position of the at least one code block, trimming a subsequent portion of the at least one code block; or,

[0020] Trim the contents of the bracket structure in the code block; or,

[0021] A start trimming position is determined between a start position of the at least one code block and a start position of a child node of the code block, an end trimming position is determined between an end position of the at least one code block and an end position of a child node of the code block, and a portion between the start trimming position and the end trimming position is trimmed.

[0022] According to an achievable method in an embodiment of the present application, using the second code to generate a prompt instruction includes:

[0023] A prompt instruction is generated using the second code, the cropped position, and information prompting to complete the code at the cropped position.

[0024] According to an achievable manner in the embodiment of the present application, obtaining the fourth code by using the second code and the third code includes:

[0025] The third code is spliced ​​to the trimmed position in the second code to obtain a fourth code.

[0026] According to an achievable manner in the embodiment of the present application, before comparing the first code and the fourth code, the method further includes:

[0027] Performing syntax checking on the fourth code;

[0028] If the verification passes, it is determined that the syntax of this round of testing is correct, and the step of comparing the first code and the fourth code is continued; otherwise, the result of the syntax error of this round of testing is recorded in the error log.

[0029] According to an achievable method in an embodiment of the present application, the first code and the fourth code are compared to obtain a comparison result including:

[0030] Parsing the first code to construct a code tree corresponding to the first code, and parsing the fourth code to construct a code tree corresponding to the fourth code;

[0031] Comparing a code tree corresponding to the first code and a code tree corresponding to the fourth code;

[0032] If the code tree corresponding to the fourth code is compared with the code tree corresponding to the first code, and no code block is added at the level corresponding to the clipped position, it is determined that the code generation model is correctly completed in this round of testing;

[0033] The performance test results include the completion accuracy of the more than one round of tests.

[0034] According to an achievable manner in the embodiment of the present application, the method further includes:

[0035] The error log is used to determine the grammatical correctness of the one or more rounds of testing, and the performance test result includes the grammatical correctness.

[0036] According to an achievable manner in the embodiment of the present application, the method further includes:

[0037] Determine the reasoning speed of the code generation model by using the time from when the prompt instruction is input to when the third code is generated by the code generation model;

[0038] The inference speed of the code generation model in the one or more rounds of testing is used to obtain an estimated inference speed of the code generation model, and the performance test result includes the estimated inference speed.

[0039] According to an implementable manner in the embodiment of the present application, the number of code blocks included in the code in the data set is normally distributed.

[0040] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0041] According to a fourth aspect, there is provided an electronic device, comprising:

[0042] one or more processors; and

[0043] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0044] According to a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0045] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0046] 1) In each round of testing, the present application trims the first code in the data set to obtain the second code, provides the second code to the code generation model to generate the third code, and obtains the fourth code by combining the second code and the third code. The performance test results of the code generation model are obtained by comparing the first code and the fourth code in multiple rounds of testing. This method is essentially to trim the original code and submit it to the code generation model for "completion", and then compare the "completion" code with the original code to evaluate the performance of the code generation model. The whole process does not need to rely on human observation, which greatly saves labor costs, and the test results are more objective and accurate.

[0047] 2) When trimming the first code, the present application starts trimming based on the non-starting position of at least one branch in the code tree corresponding to the first code, so that other code blocks at the same level are not affected during trimming, ensuring the logical completion of the context, and being more in line with the actual "completion" scenario of the code generation model.

[0048] 3) In this application, a code block is selected from the code tree, and the code block can be trimmed based on a variety of trimming strategies. With this flexible selection of code blocks and trimming positions, theoretically one code can be trimmed into multiple different second codes, thereby improving the utilization rate of test data (i.e., the code in the database) and reducing the labor cost of obtaining more test data.

[0049] 4) The present application generates prompt instructions using the second code, the clipped location, and information prompting to complete the code at the clipped location, thereby clearly and accurately indicating that the code generation model can complete the code at the clipped location based on the second code to ensure the accuracy of the test.

[0050] 5) The present application can perform syntax checking on the fourth code, and based on this, generate a syntax correctness rate as one of the quantitative indicators in the performance test results, which can reflect whether the syntax rules used by the code generation model are reasonable.

[0051] 6) In this application, by comparing the code trees corresponding to the first code and the fourth code respectively, it is determined whether a code block is added to the level corresponding to the trimmed position to determine whether the code generation model completion is correct, and the completion accuracy is used as one of the quantitative indicators in the performance test results. This indicator can reflect whether the code generation logic used by the code generation model is reasonable.

[0052] 7) In this application, the reasoning speed of the code generation model in each round of testing can be further obtained to obtain the evaluation reasoning speed of the code generation model, and the evaluation reasoning speed can be used as one of the quantitative indicators in the performance test results, which can reflect the response performance of the code generation model.

[0053] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 is a system architecture diagram applicable to the embodiments of the present application;

[0056] Figure 2 A flow chart of a test method for a code generation model provided in an embodiment of the present application;

[0057] Figure 3a A code block example diagram of the first code provided in an embodiment of the present application;

[0058] Figure 3b A code tree example diagram of the first code provided in an embodiment of the present application;

[0059] Figure 4a A code tree example diagram of a fourth code provided in an embodiment of the present application;

[0060] Figure 4b A diagram showing an example code tree of another fourth code provided in an embodiment of the present application;

[0061] Figure 4c A code tree example diagram of yet another fourth code provided in an embodiment of the present application;

[0062] Figure 5 A schematic block diagram of a test device for a code generation model provided in an embodiment of the present application;

[0063] Figure 6A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0065] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0066] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0067] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0068] In order to facilitate the understanding of the present application, the system architecture on which the present application is based is first described. Figure 1 An exemplary system architecture to which the embodiments of the present application can be applied is shown. Figure 1 As shown in , the system architecture may include: a test device and a code generation model.

[0069] The code generation model is a trained model that generates code based on the input prompt. The code generation model is usually based on a deep learning framework and learns the syntax, structure, and common patterns of the programming language by training a large amount of code data. The input prompt usually contains text describing the required function and can also include partial code. The code generation model has the ability to generate code based on natural language text, and also has the ability to complete or continue code based on partial code.

[0070] Among them, the code generation model can be implemented based on an ordinary pre-trained language model or a large language model. A large language model, namely LLM (Large Language Model), refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. It is characterized by large scale and huge number of parameters (usually reaching more than 10 billion levels), and is usually based on a deep learning architecture such as a Transformer architecture. The difference between LLM and ordinary pre-trained language models lies in the parameter scale. When the parameter scale exceeds a certain level, the model achieves significant performance improvement and exhibits capabilities that do not exist in small models, such as in-context learning capabilities, which can learn complex patterns in language and perform a wide range of tasks, including text summarization, translation, sentiment analysis, multi-round dialogue, etc. Therefore, in order to distinguish it from traditional pre-trained language models, such models with parameter scales exceeding a certain level are called LLMs. In general, language models with parameter scales of more than 10 billion based on deep learning architectures can be called large language models. Common LLMs include: GTP-3 (Generative Pre-trained Transformer 3), T5 (Text-to-Text Transfer Transformer), GTP-4, PaLM (a large language model proposed by Google), LLaMA (Large Language Model Meta AI, a large language model released by Meta AI), and so on.

[0071] The test device is used to test the code generation model using a data set to obtain a performance test result of the code generation model. The test device can be set on a computer terminal or on a server (for example, on a cloud server). The test device can be an application located on a computer and a terminal or a server, or can also be a functional unit such as a plug-in or a software development kit (SDK) set in an application.

[0072] Figure 2 A flowchart of a test method for a code generation model provided in an embodiment of the present application, which can be performed by Figure 1 The method includes more than one round of testing, such as Figure 2 As shown in , each round of testing may include the following steps 201 to 209:

[0073] Step 201: Obtain a code from a data set as a first code, where the data set includes more than one code.

[0074] Step 203: Cut off part of the content of the first code to obtain a second code.

[0075] Step 205: Generate a prompt instruction using the second code, provide the prompt instruction to the code generation model, and obtain a third code generated by the code generation model for the second code.

[0076] Step 207: Obtain a fourth code using the second code and the third code.

[0077] Step 209: Compare the first code and the fourth code to obtain a comparison result.

[0078] After executing step 209 , if the test stop condition is not met, the process proceeds to step 201 to perform the next round of testing; if the test stop condition is met, the process proceeds to step 211 .

[0079] The test stop condition may be, for example, that the number of test rounds reaches a preset number of rounds, the code traversal in the data set (all of which are taken as the first code) is completed, and so on.

[0080] Step 211: Using the comparison results obtained from more than one round of testing, obtain the performance test results of the code generation model.

[0081] It can be seen from the above process that in each round of testing, the present application trims the first code in the data set to obtain the second code, provides the second code to the code generation model to generate the third code, and obtains the fourth code from the second code and the third code. The performance test results of the code generation model are obtained by comparing the first code and the fourth code in multiple rounds of testing. This method is essentially to trim the original code and submit it to the code generation model for "completion", and then compare the "completion" code with the original code to evaluate the performance of the code generation model. The whole process does not require human observation, which greatly saves labor costs, and the test results are more objective and accurate.

[0082] The following is a detailed description of each step in the above process and the effects that can be further produced in conjunction with the embodiments. It should be noted that the "first" and "second" limitations involved in the present disclosure do not have limitations in terms of size, order, and quantity, and are only used to distinguish them in name. For example, "first code", "second code", "third code", etc., are only used to distinguish the codes obtained at different stages in name.

[0083] First, the above step 201, namely "obtaining a code from a data set as a first code", is described in detail in conjunction with an embodiment.

[0084] In the embodiments of the present application, some existing data sets containing a large amount of code can be selected, for example, ThePile data set, humaneval-x data set, etc., which are open source data sets containing a large number of real code snippets. It is also possible to select part of the code from these existing data sets to construct a new data set to test the code generation model in the embodiments of the present application.

[0085] When constructing a data set, since the core idea adopted in this application is to submit the code to the code generation model for "completion" after trimming, for the code generation model, contexts of different lengths (i.e., the remaining part after trimming) have different effects on the prediction effect of the code generation model. Usually, the code generation model prediction of contexts of appropriate lengths is more accurate, while very long contexts will bring noise to the prediction of the code generation model. Therefore, in order to better meet the various problems that the code generation model may face in actual scenarios, in the embodiments of the present application, it is ensured as much as possible that the number of code blocks contained in the code of the data set (reflected by the context length) is diverse, for example, the number of code blocks contained in the code in the data set is normally distributed.

[0086] When obtaining codes from the data set in each round of testing, the codes may be obtained in sequence and used as the first codes, or the codes may be obtained randomly and used as the first codes.

[0087] The above step 203, namely "cutting off part of the content of the first code to obtain the second code", is described in detail below in conjunction with an embodiment.

[0088] When the first code is trimmed in the embodiment of the present application, the trimming is usually performed based on the code blocks. As one of the achievable methods, the first code can be first parsed to construct a code tree corresponding to the first code, the code tree including the hierarchical relationship between the code blocks; and the subsequent part or all of the content of at least one branch in the code tree is trimmed starting from the non-starting position of the at least one branch.

[0089] The first code may be parsed using a code parser such as tree-sitter. During the parsing process, the code is divided into code blocks, and a tree is constructed based on the inclusion relationship between the code blocks. The nodes of the constructed code tree are code blocks, and the edges are the inclusion relationships between the code blocks. If code block 2 is contained in code block 1, code block 1 is the parent node of code block 2.

[0090] For example, suppose Figure 3aThe first code shown in , where the numbers represent the number of lines, the code block corresponding to lines 1 to 10 is represented as code block a, the code block corresponding to lines 2 to 5 is represented as code block b, the code block corresponding to lines 3 to 4 is represented as code block c, the code block corresponding to lines 6 to 7 is represented as code block d, the code block corresponding to lines 8 to 9 is represented as code block e, and the code block corresponding to lines 11 to 14 is represented as code block f. The code tree formed can be as follows Figure 3b as shown in .

[0091] When trimming, you can trim one branch or multiple branches. Taking the trimming of one branch as an example, you can select a code block from the code tree and trim the code block according to the preset trimming strategy. The trimming strategy can be adopted but not limited to the following:

[0092] Strategy 1: Start from the non-starting position of the code block and trim the subsequent part of the code block.

[0093] This strategy is equivalent to starting from a certain position in the code block (which can be determined randomly) and cutting off all subsequent parts in the code block.

[0094] Strategy 2: Trim the contents of the bracket structure in the code block.

[0095] If the selected code block is a code block corresponding to function, if (conditional statement), for (loop statement), etc., which usually has a bracket structure, all the contents in the bracket structure can be trimmed.

[0096] Strategy 3: Determine the start trimming position from the start position of the code block to the start position of the child node of the code block, determine the end trimming position from the end position of the code block to the end position of the child node of the code block, and trim the part between the start trimming position and the end trimming position.

[0097] If the selected code block has a lower-level node, i.e., a child node, the minimum start offset and maximum end offset of the child node can be found. The start trimming position can be randomly selected from the start offset of the code block and the minimum start offset of the child node, and the end trimming position can be randomly selected from the maximum end offset of the child node and the end offset of the code block.

[0098] It should be noted that, in addition to the above-mentioned trimming methods, other trimming methods may also be used. The basic principle is that when trimming a certain code block, other code blocks at the same level as the code block should not be affected.

[0099] As one of the more preferred implementations, a code can be selected from a database as the first code, a code block can be randomly selected from the first code, and a non-starting position can be randomly selected from the code block to start trimming. In this way of randomly selecting a code block and a trimming position each time, in theory, one code can be trimmed into multiple different second codes, thereby improving the utilization rate of the test data (i.e., the code in the database) and reducing the labor cost of requiring more test data.

[0100] The remaining part of the first code after cutting is called the second code, and the second code is actually a code with a vacancy, and the cutting position can be recorded. The cutting position can be represented by a start cutting position and an end cutting position.

[0101] The following describes in detail the above step 205, i.e., "using the second code to generate a prompt instruction, providing the prompt instruction to the code generation model, and obtaining a third code generated by the code generation model for the second code" and step 207, i.e., "using the second code and the third code to obtain a fourth code", in conjunction with the embodiments.

[0102] The code generation model performs code generation based on the instruction, and the instruction here is called a prompt instruction. In the embodiment of the present application, the second code, the position of the clipping, and the information that prompts the code completion at the clipping position can be used to generate a prompt instruction. The prompt instruction actually instructs the code generation model to perform code completion at the clipping position in the second code, that is, to generate the code at the clipping position.

[0103] For the code generation model, the second code is equivalent to the context, and the code needs to be generated based on the context, and the generated code is called the third code. The present application does not limit the principle and prediction process of the code generation model.

[0104] After obtaining the third code, the third code can be spliced ​​to the position cut in the second code to obtain the fourth code. In addition to the direct splicing method, the generated third code can also be spliced ​​after a certain adaptation process. The fourth code obtained is actually the code obtained by "completing" the second code using the code generation model.

[0105] Furthermore, after obtaining the fourth code, the fourth code can be syntax-checked. If the check passes, it is determined that the syntax of this round of test is correct, and subsequent step 209 can be continued; otherwise, the result of the syntax error of this round of test is recorded in the error log.

[0106] When performing syntax checking, tools with syntax checking functions such as tree-sitter, Py l int, etc. can be used. This application does not limit the specific tools used.

[0107] The above step 209, i.e., "comparing the first code and the fourth code to obtain a comparison result", and step 211, i.e., "obtaining a performance test result of the code generation model using the comparison results obtained from more than one round of testing", are described in detail below in conjunction with the embodiments.

[0108] When comparing the first code and the fourth code, the main purpose is to verify whether the completion specification of the code generation model meets expectations and no cross-code block occurs. The so-called cross-code block refers to adding more code blocks to the level of the code block to be completed.

[0109] As one of the achievable methods, the first code may be parsed to construct a code tree corresponding to the first code, and the fourth code may be parsed to construct a code tree corresponding to the fourth code; and the code tree corresponding to the first code and the code tree corresponding to the fourth code may be compared. If the code tree corresponding to the fourth code does not add a code block at the level corresponding to the clipped position compared to the code tree corresponding to the first code, it is determined that the code generation model is correctly completed in this round of testing.

[0110] For example, suppose that Figure 3a In the first code shown, the code block f is trimmed to obtain the second code. The code tree corresponding to the fourth code obtained after "completion" by the code generation model is as follows Figure 4a As shown in the figure, the dotted line indicates the completed code part, that is, the code generation model generates code blocks g and h. This situation indicates that a code block is added to the level corresponding to the trimming position, that is, code block h is added, and the completion error in this round of testing.

[0111] For another example, suppose the code tree corresponding to the fourth code obtained after "completion" by the code generation model is as follows Figure 4b As shown, a code block, namely, code block h, is added to the next level of the level corresponding to the clipping position, but no code block is added to the level where the clipping position is located, so this round of test completion is correct.

[0112] For another example, suppose the code tree corresponding to the fourth code obtained after "completion" by the code generation model is as follows Figure 4c As shown in the figure, no code block is added to the level corresponding to the trimming position, and the completion of this round of tests is correct.

[0113] In addition to the above method to test whether the completion is correct, other methods can also be used, such as comparing whether the number of code blocks in the first code and the fourth code is consistent to determine whether the completion is correct. If the number of code blocks is consistent, it is determined that the completion is correct.

[0114] After multiple rounds of testing, the completion accuracy of the multiple rounds of testing can be counted and used as an indicator in the performance test results. The so-called completion accuracy can be, for example, the ratio of the number of rounds of correct completion to the number of test rounds.

[0115] In addition, if the fourth code is syntax-checked before this step, the syntax correctness of multiple rounds of testing can be further counted and used as one of the indicators of performance testing. The so-called syntax correctness can be, for example, the ratio of the number of grammatically correct rounds to the number of test rounds.

[0116] Furthermore, in step 205, the time when the prompt instruction is input to the code generation model and the time when the code generation model generates the third code can be recorded, and the time from when the prompt instruction is input to when the code generation model generates the third code can be used to determine the reasoning speed of the code generation model. By counting the reasoning speeds of multiple rounds of tests, the estimated reasoning speed of the code generation model can be obtained. For example, after averaging the reasoning speeds of multiple rounds of tests, taking the median, etc., the obtained value is used as the estimated reasoning speed of the code generation model. Then the estimated reasoning speed is also used as one of the indicators of performance testing.

[0117] It should be noted that when multiple rounds of testing are performed in the above test process, the codes in the database can be traversed and tested separately, and the test on the code generation model can be ended after all the codes are tested. Alternatively, codes can be randomly selected from the database and tested separately, and the test on the code generation model can be ended after reaching a preset number of test rounds.

[0118] In addition, the indicators and related test data (such as the first code, trimming position, generated third code, etc.) obtained in each round of testing can be recorded in the log, so that the test data that affects specific indicators can be effectively located later, and targeted optimization can be performed in the subsequent iteration process of the code generation model, thereby improving the performance and accuracy of the model.

[0119] The above testing method is not limited to a specific language, so it can better serve developers of all types, thereby improving the universality of testing the code generation model.

[0120] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0121] According to an embodiment of another aspect, a testing apparatus for a code generation model is provided. Figure 5 FIG. 1 is a schematic block diagram of the test device according to an embodiment. Figure 5 As shown, the device 500 includes: a code acquisition unit 501, a code trimming unit 502, a code completion unit 503, a code comparison unit 504 and a result determination unit 505, and may further include a syntax checking unit 506 and a speed determination unit 507. The main functions of each component unit are as follows:

[0122] The code acquisition unit 501 is configured to acquire a code from a data set as a first code in each round of testing, and the data set includes more than one code.

[0123] The code clipping unit 502 is configured to clip part of the first code to obtain a second code.

[0124] The code completion unit 503 is configured to generate a prompt instruction using the second code, provide the prompt instruction to the code generation model, obtain a third code generated by the code generation model for the second code; and obtain a fourth code using the second code and the third code.

[0125] The code comparison unit 504 is configured to compare the first code and the fourth code to obtain a comparison result.

[0126] The result determination unit 505 is configured to obtain the performance test result of the code generation model using the comparison results obtained from more than one round of testing.

[0127] As one of the implementable ways, the code trimming unit 502 may be specifically configured to parse the first code to construct a code tree corresponding to the first code, where the code tree includes a hierarchical relationship between code blocks; and trim a subsequent part or all of the content of at least one branch starting from a non-starting position of at least one branch in the code tree.

[0128] As one of the implementable ways, when the code trimming unit 502 trims the subsequent part of at least one branch from the non-starting position of at least one branch in the code tree, it can be specifically configured to: select at least one code block from the code tree, and trim the code block according to a preset trimming strategy; wherein the trimming strategy includes but is not limited to:

[0129] Strategy 1: Starting from a non-starting position of at least one code block, trim the subsequent part of at least one code block.

[0130] Strategy 2: Trim the content within the bracket structure in the code block.

[0131] Strategy 3: Determine the start trimming position from the start position of at least one code block to the start position of the child node of the code block, determine the end trimming position from the end position of at least one code block to the end position of the child node of the code block, and trim the part between the start trimming position and the end trimming position.

[0132] As one achievable manner, when the code completion unit 503 generates a prompt instruction using the second code, it can be specifically configured to generate a prompt instruction using the second code, the cropped position, and information prompting code completion at the cropped position.

[0133] As one of the implementable ways, when the code completion unit 503 obtains the fourth code by using the second code and the third code, it can be specifically configured to: splice the third code to the trimmed position in the second code to obtain the fourth code.

[0134] Furthermore, the grammar checking unit 506 can be configured to: perform grammar checking on the fourth code; if the check passes, it is determined that the syntax of this round of test is correct, and the code comparison unit 504 is triggered to execute the step of comparing the first code and the fourth code; otherwise, the result of the grammar error of this round of test is recorded in the error log.

[0135] As one of the achievable ways, the code comparison unit 504 can be configured to: parse the first code to construct a code tree corresponding to the first code, and parse the fourth code to construct a code tree corresponding to the fourth code; compare the code tree corresponding to the first code and the code tree corresponding to the fourth code; if the code tree corresponding to the fourth code does not add a code block at the level corresponding to the clipped position compared to the code tree corresponding to the first code, it is determined that the code generation model is correctly completed in this round of testing. In this case, the performance test results can include the completion accuracy of more than one round of testing.

[0136] Furthermore, the result determination unit 505 may also be configured to: determine the grammatical correctness of the above-mentioned one or more rounds of testing using the error log. In this case, the performance test result may further include the grammatical correctness.

[0137] Furthermore, the speed determination unit 507 can be configured to determine the reasoning speed of the code generation model by using the time from when the prompt instruction is input to when the third code is generated. The result determination unit 505 can also be configured to obtain the estimated reasoning speed of the code generation model by using the reasoning speed of the code generation model in the above-mentioned one or more rounds of testing. In this case, the performance test result can further include the estimated reasoning speed.

[0138] As one of the more preferred implementations, the number of code blocks contained in the code in the above data set is normally distributed.

[0139] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0141] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0142] And an electronic device, comprising:

[0143] one or more processors; and

[0144] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0145] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0146] in, Figure 6The architecture of the electronic device is shown as an example, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620 may be communicatively connected via a communication bus 630.

[0147] Among them, the processor 610 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0148] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store an operating system 621 for controlling the operation of the electronic device 600, and a basic input and output system (BIOS) 622 for controlling the low-level operation of the electronic device 600. In addition, a web browser 623, a data storage management system 624, and a test device 625, etc. can also be stored. The above-mentioned test device 625 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0149] The input / output interface 613 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0150] The network interface 614 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0151] The bus 630 comprises a pathway for transmitting information between the various components of the device (eg, the processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620).

[0152] It should be noted that, although the above device only shows a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, a memory 620, a bus 630, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0153] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0154] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A testing method for a code generation model, characterized in that: The method includes more than one round of testing; Each round of testing includes: Obtaining a code from a data set as a first code, the data set comprising more than one code; Cutting off part of the content of the first code to obtain a second code; Generate a prompt instruction using the second code, provide the prompt instruction to the code generation model, and obtain a third code generated by the code generation model for the second code; Obtain a fourth code using the second code and the third code; Comparing the first code and the fourth code to obtain a comparison result; The performance test results of the code generation model are obtained by using the comparison results obtained from the one or more rounds of testing.

2. The method according to claim 1, characterized in that The first code is trimmed of some contents, and the second code is obtained including: Parsing the first code to construct a code tree corresponding to the first code, the code tree including a hierarchical relationship between code blocks; Starting from a non-starting position of at least one branch in the code tree, a subsequent part or all of the content of the at least one branch is pruned.

3. The method according to claim 2, characterized in that Trimming a subsequent part or all of the content of at least one branch from a non-starting position of at least one branch in the code tree includes: selecting at least one code block from the code tree, and trimming the code block according to a preset trimming strategy; The tailoring strategy includes: Starting from a non-starting position of the at least one code block, trimming a subsequent portion of the at least one code block; or, Trim the contents of the bracket structure in the code block; or, A start trimming position is determined between a start position of the at least one code block and a start position of a child node of the code block, an end trimming position is determined between an end position of the at least one code block and an end position of a child node of the code block, and a portion between the start trimming position and the end trimming position is trimmed.

4. The method according to claim 1, characterized in that: Generating a prompt instruction using the second code includes: A prompt instruction is generated using the second code, the cropped position, and information prompting to complete the code at the cropped position.

5. The method according to claim 1, characterized in that Obtaining a fourth code using the second code and the third code comprises: The third code is spliced ​​to the trimmed position in the second code to obtain a fourth code.

6. The method according to claim 1, characterized in that Before comparing the first code and the fourth code, the method further includes: Performing syntax checking on the fourth code; If the verification passes, it is determined that the syntax of this round of testing is correct, and the step of comparing the first code and the fourth code is continued; otherwise, the result of the syntax error of this round of testing is recorded in the error log.

7. The method according to any one of claims 1 to 6, characterized in that Comparing the first code and the fourth code, obtaining a comparison result includes: Parsing the first code to construct a code tree corresponding to the first code, and parsing the fourth code to construct a code tree corresponding to the fourth code; Comparing a code tree corresponding to the first code and a code tree corresponding to the fourth code; If the code tree corresponding to the fourth code is compared with the code tree corresponding to the first code, and no code block is added at the level corresponding to the clipped position, it is determined that the code generation model is correctly completed in this round of testing; The performance test results include the completion accuracy of the more than one round of tests.

8. The method according to claim 6, characterized in that The method further comprises: The error log is used to determine the grammatical correctness of the one or more rounds of testing, and the performance test result includes the grammatical correctness.

9. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Determine the reasoning speed of the code generation model by using the time from when the prompt instruction is input to when the third code is generated by the code generation model; The inference speed of the code generation model in the one or more rounds of testing is used to obtain an estimated inference speed of the code generation model, and the performance test result includes the estimated inference speed.

10. The method according to any one of claims 1 to 6, characterized in that The number of code blocks contained in the code in the data set is normally distributed.

11. A testing device for a code generation model, for performing more than one round of testing, characterized in that: The device comprises: A code acquisition unit is configured to acquire a code from a data set as a first code in each round of testing, wherein the data set includes more than one code; A code clipping unit, configured to clip part of the first code to obtain a second code; A code completion unit is configured to generate a prompt instruction using the second code, provide the prompt instruction to the code generation model, obtain a third code generated by the code generation model for the second code; and obtain a fourth code using the second code and the third code; a code comparison unit, configured to compare the first code and the fourth code to obtain a comparison result; The result determination unit is configured to obtain the performance test result of the code generation model by using the comparison results obtained from the one or more rounds of testing.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

13. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Citation Information

Cited By

  • Method and system for testing reliability of air conditioner based on monitoring information

    CN119915540A

  • Large language model code generation system and method oriented to complex requirements

    CN119938006A

  • Model training method and apparatus, and device and storage medium

    WO2025256147A1