Method and apparatus for evaluating large code model

Abstract

Provided in the embodiments of the present description are a method and apparatus for evaluating a large code model. The method comprises: on the basis of a test problem for code generation, determining prompt information, inputting the prompt information into a large code model, so as to obtain output information of the large code model, and extracting code text from the output information; separating an execution code and a code annotation from the code text, compiling the execution code, and determining a compilation pass rate on the basis of whether the compilation is passed, and if the compilation is passed, executing a corresponding test case for the execution code, and determining an overall pass rate on the basis of whether the test case is passed; determining a semantic similarity on the basis of the code annotation and a reference annotation corresponding to the code annotation; and determining a comprehensive score on the basis of the compilation pass rate, the overall pass rate and the semantic similarity.

Description

A method and device for evaluating a large code model

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 15, 2023, with application number 202311197791.7 and application name “A method and device for evaluating a large code model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] One or more embodiments of this specification relate to the field of large code models, and more particularly, to a method and apparatus for evaluating large code models. Background Art

[0003] Large code models are deep learning models used for code generation and can be based on large language models. Large language models (LLMs) are deep learning models based on natural language processing, trained on large text corpora and containing tens of billions or more parameters. Currently, there is a lack of comprehensive and accurate methods for evaluating the code output and related content of large code models.

[0004] Therefore, a new evaluation method for large code models is needed.

[0005] Summary of the Invention

[0006] The embodiments of this specification aim to provide a method and apparatus for evaluating a large code model. This method can determine a comprehensive score for the quality of the code and code-related content output by the large code model based on the compilation pass rate of the executed code output by the large code model, the overall pass rate of both compilation and testing, and the semantic similarity between the code annotations and explanations output by the model and the reference annotations and explanations. This method enables a comprehensive and accurate evaluation of the syntactic and functional correctness of the code generated by the large code model, as well as the quality of the code annotations and explanations, thereby addressing the shortcomings of the existing technology.

[0007] According to a first aspect, a method for evaluating a large code model is provided, comprising:

[0008] Determine prompt information according to a test question for code generation, input the prompt information into a large code model, obtain output information of the large code model, and extract code text from the output information;

[0009] Separating execution code and code comments from the code text; compiling the execution code, and determining a compilation pass rate based on whether the compilation passes; if the compilation passes, executing corresponding test cases for the execution code, and determining an overall pass rate based on whether the test cases pass;

[0010] determining semantic similarity based on the code annotation and a reference annotation corresponding to the code annotation;

[0011] A comprehensive score is determined based on the compilation pass rate, the overall pass rate, and the semantic similarity.

[0012] In a possible implementation, extracting the code text from the output information includes: extracting the code text and a textual interpretation of the code text from the output information;

[0013] Determining semantic similarity based on the code annotation and a reference annotation corresponding to the code annotation includes:

[0014] Semantic similarity is determined based on the code annotation and the reference annotation corresponding to the code annotation, and the text interpretation and the reference interpretation corresponding to the text interpretation.

[0015] In a possible implementation, determining prompt information according to a test question for code generation includes: determining prompt information according to the test question for code generation and first indication information of an output format of the output information.

[0016] In a possible implementation, determining prompt information according to a test question for code generation includes determining prompt information according to the test question for code generation and second indication information of a programming language based on which the code text included in the output information is based.

[0017] In a possible implementation, determining prompt information according to a test problem for code generation includes: determining prompt information according to the test problem for code generation and third indication information of a test case format of the test case.

[0018] In a possible implementation, determining a comprehensive score based on the compilation pass rate, the overall pass rate, and the semantic similarity includes:

[0019] A comprehensive score is determined based on a weighted average of the compilation pass rate, the overall pass rate, and the semantic similarity.

[0020] In a possible implementation, the test cases include positive and negative functional test cases.

[0021] In a possible implementation, the output information includes multiple answers to the test question; and extracting the code text from the output information includes extracting multiple code texts corresponding to the multiple answers from the output information;

[0022] The separating the execution code and the code comment from the code text comprises: separating the execution code and the code comment corresponding to the plurality of answers respectively from the plurality of code texts;

[0023] Compiling the execution code and determining a compilation pass rate according to whether the compilation passes includes: respectively compiling multiple execution codes corresponding to the multiple answers, and determining the compilation pass rate according to the number of execution codes that pass the compilation.

[0024] In a possible implementation, if the compilation passes, executing corresponding test cases for the execution code, and determining the overall pass rate based on whether the test cases pass, includes:

[0025] For each execution code, if the compilation passes, the corresponding test case is executed for each execution code, and the overall pass rate is determined according to the number of execution codes that pass the test case in the multiple execution codes.

[0026] According to a second aspect, a device for evaluating a large code model is provided, the device comprising:

[0027] an extraction unit configured to determine prompt information according to a test question for code generation, input the prompt information into a large code model, obtain output information of the large code model, and extract code text from the output information;

[0028] The first processing unit is configured to separate execution code and code comments from the code text; compile the execution code, and determine a compilation pass rate based on whether the compilation passes; if the compilation passes, execute corresponding test cases for the execution code, and determine an overall pass rate based on whether the test cases pass;

[0029] a second processing unit configured to determine semantic similarity based on the code annotation and a reference annotation corresponding to the code annotation;

[0030] The scoring unit is configured to determine a comprehensive score based on the compilation pass rate, the overall pass rate and the semantic similarity.

[0031] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the first method described above.

[0032] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the first method described is implemented.

[0033] By utilizing one or more of the methods, apparatuses, computing devices, and storage media in the above aspects, a comprehensive and accurate evaluation can be performed on the syntactic correctness, functional correctness, and quality of code annotations and code interpretations of the code generated by the code model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 shows a schematic diagram of the input and output of the code model;

[0036] FIG2 is a schematic diagram showing the principle of a method for evaluating a large code model according to an embodiment of this specification;

[0037] FIG3 shows a flow chart of a method for evaluating a large code model according to an embodiment of this specification;

[0038] FIG4 is a schematic diagram showing prompt information according to an embodiment of this specification;

[0039] FIG5 shows a structural diagram of a device for evaluating a large code model according to an embodiment of this specification. DETAILED DESCRIPTION

[0040] The solution provided in this invention will be described below with reference to the accompanying drawings.

[0041] As previously mentioned, the code model is a deep learning model for code generation and for generating annotations or explanations corresponding to the code. The code model can be based on a large language model (LLM). A large language model refers to a deep learning model based on natural language processing that is trained on a large text corpus and contains tens of billions (or more) of parameters. Generally speaking, the code model can generate corresponding code (code) and annotations in the code based on the natural language description entered by the user. In some examples, a natural language explanation of the code can also be generated. Figure 1 shows a schematic diagram of the input and output of the code model. As shown in Figure 1, for example, a user enters a natural language description into the code model. This description can generally be the problem that the code model is expected to generate code to solve. For example, in the example of Figure 1, the user enters "Write a Python function to find the maximum of two numbers." The code model can output a source code written in the Python programming language based on the user input information. The source code includes not only the code for execution (or execution code), such as the definition of the function findmax, but also the test code for that function. Code comments (e.g., text comments beginning with the symbol "#") may also be included. Code comments are explanations and descriptions of the code, with the purpose of allowing users to understand the code more conveniently. In different examples, comments may be prompts for a statement, program segment, or function, etc., to improve the readability of the program code. However, comments are only for improving readability and will not be compiled by the computer, or in other words, they will not be executed. For example, in the example shown in Figure 1, in line 14 of the source code, "print(find_max(10,5))" is the execution code for printing the execution result of the function find_max(10,5). "# output: 10" is a code comment, which is a prompt for the correct printing result. In some examples, in addition to outputting the generated code and code comments, the code model can also output a natural language explanation of the code generation. In different examples, the content contained in the natural language explanation may be different. In one example, the natural language explanation may, for example, include an explanation of the idea of ​​generating the code (the main statements and / or algorithms used). For example, as shown in Figure 1, the natural language explanation includes "The basic idea of ​​this function is to use an if statement to compare...", which is a description of the main statements and algorithms of the generated code. In another example, it can also include a description of the comments in the code. For example, as shown in Figure 1, the natural language explanation includes "In the comments of the function definition, ...provides a description of the parameters and return values..." In some examples, the code model also outputs natural language conversation continuation statements, such as "Okay, here is a Python function that is used to..." as shown in Figure 1, making its overall output more similar to the natural question-and-answer format of human conversation.

[0042] In real-world production scenarios, users or manufacturers who use or produce large code models often need to measure the quality of their output, for example to determine how to use or improve them. However, due to the complexity of large code model output, there is currently no comprehensive evaluation method for measuring the quality of the code and code-related content within the output of large code models.

[0043] In order to solve the above technical problems, an embodiment of this specification provides a method for evaluating a large code model. Figure 2 shows a schematic diagram of the principle of a method for evaluating a large code model according to an embodiment of this specification. As shown in Figure 2, for example, a predetermined template can be combined with natural language questions for testing the code, and format prompts or programming language prompts for the output results of the large code model, and the large code model can be input. Then, the output results of the large code model are received, and the execution code, code comments and code explanations are separated from the output results. The execution code is compiled, and the compilation pass rate is determined based on whether the compilation passes. For the execution code that passes the compilation, the test case is executed. The overall pass rate is determined based on whether the test case passes. In addition, based on the code comments and the corresponding reference comments, the text similarity is determined based on the code comments and the corresponding reference explanations. Thereafter, based on the compilation pass rate, the overall pass rate and the semantic similarity, a comprehensive score for the quality of the code and code-related content output by the large code model is determined.

[0044] This method has the following advantages: it can comprehensively evaluate the execution code, code comments, and code explanations output by the large code model based on the compilation pass rate, overall pass rate (i.e., the proportion of code that passes both compilation and use case testing in the total generated code), and the semantic similarity between the code comments and code explanations and the reference comments and reference explanations. Furthermore, based on the compilation pass rate, overall pass rate, and semantic similarity, a comprehensive score for the code and code-related content of the large code model can be obtained. Through this method, for example, the syntactic correctness and functional correctness of the code generated by the large code model, as well as the quality of the code comments and code explanations, can be comprehensively and accurately evaluated.

[0045] The detailed process of this method is further described below. FIG3 shows a flow chart of a method for evaluating a large code model according to an embodiment of this specification. As shown in FIG3 , the method includes at least the following steps:

[0046] Step S301, determining prompt information based on a test question for code generation, inputting the prompt information into a large code model, obtaining output information of the large code model, and extracting code text from the output information;

[0047] Step S302: Separate the execution code and code comments from the code text; compile the execution code, and determine the compilation pass rate based on whether the compilation passes; if the compilation passes, execute the corresponding test case for the execution code, and determine the overall pass rate based on whether the test case passes;

[0048] Step S303, determining semantic similarity based on the code annotation and the reference annotation corresponding to the code annotation;

[0049] Step S304: determining a comprehensive score based on the compilation pass rate, the overall pass rate, and the semantic similarity.

[0050] First, in step S301, based on the test problem for code generation, prompt information is determined, the prompt information is input into the code model, the output information of the code model is obtained, and the code text is extracted from the output information. The test problem is the problem based on which the code model generates code, or it can be said that it is a problem used to test the code generation capability of the code model. In one embodiment, the test problem can be a problem expressed in natural language. In different embodiments, the specific test problems used for code generation can be different, and this specification does not limit this. For example, in the example shown in Figure 4, the test problem can be, for example, "find the first repeated element in a given sequence." Furthermore, after the prompt information containing the test problem is input into the code model, the code model can generate corresponding code, such as code that solves the problem, based on the problem.

[0051] Prompt information is input into the code model. In different embodiments, the specific method for determining the prompt information may vary depending on the test question. In one embodiment, the prompt information may be determined based on the test question used for code generation and first information indicating the output format of the output information. In different embodiments, the specific content of the first prompt information may vary depending on the desired output format of the code model. For example, as shown in FIG4 , the first prompt information may be, for example, "All output is returned in Markdown format." Markdown is a simple, easy-to-read and easy-to-write text format. In one embodiment, the prompt information may be determined based on the test question used for code generation and second information indicating the programming language based on the code text included in the output information. In different embodiments, the desired programming language of the code model output code may vary, and therefore, the specific content of the second prompt information may also vary. For example, as shown in FIG4 , the second prompt information may be, for example, "Please use Python to implement...". Python is an interpreted, high-level programming language. In one embodiment, the prompt information may be determined based on the test question used for code generation and third information indicating the test case format of the test case. In different specific embodiments, the specific content of the third prompt information may be different depending on the different test cases that are desired to be executed thereafter. For example, as shown in FIG4 , the third prompt information may be, for example, “use assert to prompt the first repeating element”, which is used to prompt the code large model to use the assert function to prompt the first repeating element to be found in the output code. The advantage is that if the test case for executing the code thereafter has a predetermined use case format, for example, the repeating element to be output is determined according to the assert function. By providing the third prompt information to the code large model, the code generated by the code large model can better meet the requirements of the test case format, so that the test case can more accurately test the real code generation capability of the code large model. In different embodiments, one or more of the first prompt information, the second prompt information, and the third prompt information can be combined with the test question to obtain the prompt information.

[0052] After inputting the prompt information into the code model, output information can be received from the code model, and code text can be extracted from the output information. The code text can typically be source code generated by the code model. The specific method for extracting the code text from the output information can vary in different embodiments, and this specification does not limit this. In one embodiment, for example, the code text can be extracted from the output information using a regular expression.

[0053] In actual code generation scenarios, the code model often outputs a textual explanation of the generated code. Therefore, in one embodiment, the code text and the textual explanation of the code text can also be extracted from the output information. The textual explanation of the code text can also be referred to as the code explanation. The code explanation is typically not part of the source code, but rather a textual explanation of the generated code, such as programming ideas, main algorithms, and statements, that the code model mimics.

[0054] In some scenarios, a single test question may correspond to multiple answers. For example, a test question may require the generation of multiple code solutions in multiple programming languages. Therefore, in one embodiment, the output information may include multiple answers to the test question. Furthermore, the multiple code texts corresponding to the multiple answers can be extracted from the output information.

[0055] Then, in step S303, the execution code and code comments can be separated from the code text; the execution code can be compiled, and the compilation pass rate can be determined based on whether the compilation passes; if the compilation passes, the corresponding test cases can be executed for the execution code, and the overall pass rate can be determined based on whether the test cases pass.

[0056] As previously mentioned, the code text may generally be source code generated by a large code model, which may include execution code for execution and code comments not for execution. In various specific embodiments, depending on the programming language, the code comments may be annotated with annotation symbols predetermined in the programming language, so that the compiler can identify the code comments from the source code. Therefore, in various embodiments, the execution code and code comments may be separated from the code text based on annotation symbols predetermined in the programming language.

[0057] Compile refers to the process of generating a computer executable program based on a code written in a high-level programming language. In this step, the executable code can be compiled, and the compilation pass rate is determined based on whether the compilation is successful. Since compilation will check the syntax of the compiled code, only grammatically correct code can pass the compilation. Therefore, through compilation, it is possible to determine whether the syntax of the executable code is correct. Test cases are test inputs, execution conditions, and expected results compiled for a predetermined goal in order to test whether the tested code has achieved the predetermined function. Therefore, by executing test cases, the functional correctness of the executable code can be tested. Therefore, if the executable code compiles successfully, the corresponding test case can be executed for the compiled executable code, and the overall pass rate can be determined based on whether the test case passes. In different embodiments, the corresponding test cases may be different depending on the execution code, and this specification does not limit this. In one embodiment, the test cases may include positive and negative functional test cases.

[0058] In the embodiment where the above-mentioned test question corresponds to multiple answers, the execution codes and code comments corresponding to the multiple answers can be separated from the multiple code texts respectively. Thus, the multiple execution codes corresponding to the multiple answers can be compiled respectively, and the compilation pass rate can be determined based on the number of execution codes that have passed the compilation. Moreover, for each execution code, if the compilation passes, the corresponding test case is executed for each execution code, and the overall pass rate is determined based on the number of execution codes that pass the test case in the multiple execution codes. For example, there are 10 execution codes, of which 2 are compiled successfully but the use case execution fails, and 2 are compiled successfully and the use case passes. The compilation success rate is 40%, and the overall pass rate is 20%.

[0059] In different embodiments, the compilation pass rate and the overall pass rate can also be determined based on an unbiased estimation algorithm based on the number of all executed codes corresponding to all answers to multiple questions, the number of executed codes that passed compilation, and the number of executed codes that passed compilation and use cases. Different unbiased estimation algorithms can be used in different specific embodiments, and this specification does not limit this.

[0060] Next, in step S305, semantic similarity can be determined based on the code annotation and the reference annotation corresponding to the code annotation. In this step, semantic similarity can be determined based on the code annotation and the reference annotation corresponding to the pre-acquired code annotation. In different embodiments, the specific method for determining semantic similarity can vary. For example, in one embodiment, semantic similarity can be determined based on a cosine similarity algorithm. In one embodiment, semantic similarity can be determined based on the BLEURT metric.

[0061] In the above embodiment of extracting code text and code interpretation from output information, semantic similarity can also be determined based on the code annotation and the reference annotation corresponding to the code annotation, as well as the text interpretation and the reference interpretation corresponding to the text interpretation. In one example, based on a predetermined algorithm, the semantic similarity between the code annotation and the reference annotation (or annotation similarity) and the semantic similarity between the text interpretation and the reference interpretation (or interpretation similarity) can be calculated, and the semantic similarity can be determined based on the average of the annotation similarity and the interpretation similarity.

[0062] In the embodiment where the test question corresponds to multiple answers, semantic similarity can be determined based on the code annotations and code explanations corresponding to the multiple answers and the corresponding reference annotations and reference explanations. In one example, for example, the annotation similarity and explanation similarity between the code annotations and code explanations corresponding to the multiple answers and the corresponding reference annotations and reference explanations can be calculated respectively. Then, the semantic similarity is determined based on the average of the annotation similarities and explanation similarities.

[0063] In various embodiments, semantic similarity can also be determined based on the code annotations and code explanations corresponding to all answers to the multiple questions and the corresponding reference annotations and reference explanations. In one example, for example, the annotation similarity and explanation similarity between the code annotations and code explanations corresponding to all answers to the multiple questions and the corresponding reference annotations and reference explanations can be calculated. Semantic similarity can then be determined based on the average of the annotation similarities and explanation similarities.

[0064] Afterwards, a comprehensive score can be determined in step S307 based on the compilation pass rate, the overall pass rate, and the semantic similarity. In different embodiments, the specific manner of determining the comprehensive score may be different. In one embodiment, the comprehensive score can be determined based on the arithmetic mean of the compilation pass rate, the overall pass rate, and the semantic similarity. In one embodiment, the comprehensive score can be determined based on the weighted average of the compilation pass rate, the overall pass rate, and the semantic similarity. In different specific embodiments, the weighted weights of the compilation pass rate, the overall pass rate, and the semantic similarity may be different, and this specification does not impose any limitation on this.

[0065] According to another embodiment, a device for evaluating a large code model is also provided. FIG5 shows a structural diagram of a device for evaluating a large code model according to an embodiment of this specification. As shown in FIG5 , the device 500 includes:

[0066] The extraction unit 501 is configured to determine prompt information according to a test question for code generation, input the prompt information into a large code model, obtain output information of the large code model, and extract code text from the output information;

[0067] The first processing unit 502 is configured to separate the execution code and the code comments from the code text; compile the execution code, and determine the compilation pass rate based on whether the compilation passes; if the compilation passes, execute the corresponding test case for the execution code, and determine the overall pass rate based on whether the test case passes;

[0068] The second processing unit 503 is configured to determine semantic similarity based on the code annotation and a reference annotation corresponding to the code annotation;

[0069] The scoring unit 504 is configured to determine a comprehensive score according to the compilation pass rate, the overall pass rate and the semantic similarity.

[0070] Another aspect of the embodiments of this specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute any one of the above methods.

[0071] Another aspect of the embodiments of this specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, any one of the above methods is implemented.

[0072] It should be understood that the descriptions such as “first” and “second” in this article are only used to distinguish similar concepts for the sake of simplicity of description and do not have any other limiting effect.

[0073] Although one or more embodiments of this specification provide the method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, an environment of parallel processors or multi-threaded processing, or even a distributed data processing environment). The term "comprise", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0074] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0075] Those skilled in the art will appreciate that one or more embodiments of the present specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In distributed computing environments, program modules may be located in local and remote computer storage media, including storage devices.

[0077] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0078] The foregoing is merely an example of one or more embodiments of this specification and is not intended to limit the one or more embodiments of this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification shall be included within the scope of the claims.

Claims

1. A method for evaluating a large code model, comprising: Determine prompt information according to the test questions for code generation, input the prompt information into the code big model, obtain output information of the code big model, and extract code text from the output information; Separating execution code and code comments from the code text; Compiling the execution code, and determining the compilation pass rate according to whether the compilation passes; if the compilation passes, executing the corresponding test case for the execution code, and determining the overall pass rate according to whether the test case passes; Determining semantic similarity based on the code annotation and a reference annotation corresponding to the code annotation; A comprehensive score is determined based on the compilation pass rate, the overall pass rate and the semantic similarity.

2. The method according to claim 1, wherein: Extracting the code text from the output information includes: extracting the code text and a text explanation for the code text from the output information; Determining semantic similarity according to the code annotation and a reference annotation corresponding to the code annotation includes: Semantic similarity is determined based on the code annotation and the reference annotation corresponding to the code annotation, and the text interpretation and the reference interpretation corresponding to the text interpretation.

3. The method according to claim 1, wherein: Determining prompt information according to a test problem for code generation includes: determining prompt information according to the test problem for code generation and first indication information of an output format of the output information.

4. The method according to claim 1, wherein: Determining prompt information according to a test question for code generation includes: determining prompt information according to the test question for code generation and second indication information of a programming language based on which the code text included in the output information is based.

5. The method according to claim 1, wherein: Determining prompt information according to a test problem for code generation includes: determining prompt information according to the test problem for code generation and third indication information of a test case format of the test case.

6. The method according to claim 1, wherein: According to the compilation pass rate, the overall pass rate and the semantic similarity, a comprehensive score is determined, including: A comprehensive score is determined according to a weighted average of the compilation pass rate, the overall pass rate and the semantic similarity.

7. The method according to claim 1, wherein: The test cases include positive and negative functional test cases.

8. The method according to claim 1, wherein: The output information includes a plurality of answers to the test question; the extracting the code text from the output information includes extracting a plurality of code texts corresponding to the plurality of answers from the output information; The separating the execution code and the code comment from the code text comprises: separating the execution code and the code comment corresponding to the multiple answers respectively from the multiple code texts; Compiling the execution code and determining the compilation pass rate according to whether the compilation is successful, including: respectively compiling multiple execution codes corresponding to the multiple answers, and determining the compilation pass rate according to the number of execution codes that are successfully compiled.

9. The method according to claim 8, wherein: If the compilation is successful, executing the corresponding test case for the execution code, and determining the overall pass rate according to whether the test case is passed, including: For each execution code, if the compilation passes, the corresponding test case is executed for each execution code, and the overall pass rate is determined according to the number of execution codes that pass the test case in the multiple execution codes.

10. A device for evaluating a large code model, the device comprising: An extraction unit is configured to determine prompt information according to a test question for code generation, input the prompt information into a large code model, obtain output information of the large code model, and extract code text from the output information; A first processing unit is configured to separate execution code and code comments from the code text; Compiling the execution code, and determining the compilation pass rate according to whether the compilation passes; if the compilation passes, executing the corresponding test case for the execution code, and determining the overall pass rate according to whether the test case passes; A second processing unit is configured to determine a semantic similarity based on the code annotation and a reference annotation corresponding to the code annotation; The scoring unit is configured to determine a comprehensive score according to the compilation pass rate, the overall pass rate and the semantic similarity.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 9.

12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 9 is implemented.

Images