Code generation model evaluation method and device, program product and electronic equipment

Abstract

The invention provides a code generation model evaluation method and device, a program product and electronic equipment, and relates to the technical field of computers. The method comprises the steps that a to-be-detected code generation model is determined; generating at least one business code based on the to-be-detected code generation model; analyzing and processing the at least one business code to obtain data information of different dimensions; and obtaining an evaluation result corresponding to the to-be-detected code generation model according to the data information of different dimensions and the evaluation rule. According to the method, the data information of different dimensions of the business code generated by the to-be-detected code generation model is fully automatically determined, and the evaluation condition is judged based on the data information of different dimensions, so that the quality of the to-be-detected code generation model is quickly and accurately determined, and obviously, the detection efficiency is improved. In this way, the evaluation efficiency and the evaluation accuracy of the to-be-detected code generation model can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a code generation model evaluation method, device, program product and electronic device. Background Art

[0002] At present, generative models represented by large language models (LLMs) are developing at a rapid pace and have attracted widespread attention in the industry. By learning massive amounts of natural language data, these models can not only generate fluent and natural texts, but also have the ability to understand deep semantics and perform complex tasks. With the continuous advancement of technology, LLMs are increasingly performing well in the field of code generation. They are playing an increasingly important role in improving software development efficiency, lowering programming barriers, and promoting software technology innovation, providing innovative solutions for automated software development and opening up new possibilities.

[0003] However, how to accurately and efficiently evaluate the code generation capability of generative models in order to obtain large-scale language models that can be quickly put into use has become an urgent problem to be solved. Summary of the invention

[0004] The present disclosure provides a code generation model evaluation method, a code generation model evaluation device, a computer program product and an electronic device, so as to quickly and accurately evaluate the code generation capability of a large language model.

[0005] According to a first aspect of the present disclosure, a code generation model evaluation method is provided, the method comprising:

[0006] Determine the code generation model to be tested;

[0007] Generate at least one service code based on the code generation model to be detected;

[0008] Analyze and process the at least one business code to obtain data information of different dimensions;

[0009] According to the data information and evaluation rules of the different dimensions, an evaluation result corresponding to the code generation model to be tested is obtained.

[0010] In one implementation, the data information of different dimensions includes: at least two of: code style data, code security data, code compilation data, code execution data, code memory leakage data, and code calling resource data.

[0011] In one embodiment, obtaining the evaluation result corresponding to the code generation model to be tested according to the data information and evaluation rules of different dimensions includes:

[0012] Determine the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule; the evaluation sub-rule belongs to the evaluation rule;

[0013] According to the sub-evaluation value corresponding to each dimension, the evaluation result corresponding to the code generation model to be tested is obtained.

[0014] In one implementation, if the data information is code style data, determining the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule includes:

[0015] Determine whether the sub-data in the code style data conforms to the sub-specifications in the first preset specification; the first preset specification includes coding specifications, naming specifications, comment specifications, abbreviation specifications, and comment-to-code ratio specifications;

[0016] According to the sub-data in the code style data conforming to the corresponding sub-specification data, a sub-evaluation value corresponding to the code style data is determined.

[0017] In one implementation, if the data information is code security data, determining the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule includes:

[0018] Determine whether the sub-data in the code security data conforms to the sub-specifications in the second preset specification; the second preset specification includes a vulnerability specification, an exception handling specification, a return information specification, and a variation initialization specification;

[0019] According to the sub-data in the code safety data conforming to the corresponding sub-specification data, a sub-evaluation value corresponding to the code safety data is determined.

[0020] In one implementation, if the data information is code compilation data, determining the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule includes:

[0021] Determining information about the number of errors in the code compilation data;

[0022] According to the error quantity information, a sub-evaluation value corresponding to the code compilation data is determined.

[0023] In one implementation, if the data information is code execution data, determining the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule includes:

[0024] According to the number of passed test cases and the number of test cases in the code execution data, a sub-evaluation value corresponding to the code execution data is determined.

[0025] In one implementation, if the data information is code memory leakage data, determining the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule includes:

[0026] Determine the leakage quantity information in the code memory leakage data;

[0027] According to the leakage quantity information, a sub-evaluation value corresponding to the code leakage data is determined.

[0028] In one implementation, if the data information is code call resource data, determining the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule includes:

[0029] Determine a code performance metric value according to an average memory usage rate, an average central processing unit CPU utilization rate, an average graphics processing unit GPU utilization rate, an average graphics processing unit GPU video memory utilization rate, and a code running time in the code call resource data;

[0030] According to the range of the code performance metric value, a sub-evaluation value corresponding to the code call resource data is determined.

[0031] In one embodiment, the method further comprises:

[0032] When it is determined that the evaluation result does not meet the preset condition, determining the submodule to be adjusted in the code generation model to be detected; the submodule to be adjusted is the module corresponding to the dimension whose sub-evaluation value does not meet the standard;

[0033] Adjusting the submodule to be adjusted to obtain an adjusted code generation model, and determining an evaluation result of the adjusted code generation model;

[0034] If the evaluation result of the adjusted code generation model meets the preset condition, it is determined that the adjusted code generation model can be put into use online.

[0035] According to a second aspect of the present disclosure, a code generation model evaluation device is provided, the device comprising:

[0036] A determination unit, used for determining a code generation model to be detected;

[0037] A generating unit, configured to generate at least one service code based on the code generation model to be detected;

[0038] An obtaining unit, configured to analyze and process the at least one business code to obtain data information of different dimensions;

[0039] The evaluation unit is used to obtain the evaluation result corresponding to the code generation model to be tested according to the data information of different dimensions and the evaluation rules.

[0040] In one implementation, the data information of different dimensions includes: at least two of: code style data, code security data, code compilation data, code execution data, code memory leakage data, and code calling resource data.

[0041] In one embodiment, the evaluation unit is used to:

[0042] Determine the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule; the evaluation sub-rule belongs to the evaluation rule;

[0043] According to the sub-evaluation value corresponding to each dimension, the evaluation result corresponding to the code generation model to be tested is obtained.

[0044] In one embodiment, if the data information is code style data, the evaluation unit is used to:

[0045] Determine whether the sub-data in the code style data conforms to the sub-specifications in the first preset specification; the first preset specification includes coding specifications, naming specifications, comment specifications, abbreviation specifications, and comment-to-code ratio specifications;

[0046] According to the sub-data in the code style data conforming to the corresponding sub-specification data, a sub-evaluation value corresponding to the code style data is determined.

[0047] In one embodiment, if the data information is code security data, the evaluation unit is used to:

[0048] Determine whether the sub-data in the code security data conforms to the sub-specifications in the second preset specification; the second preset specification includes a vulnerability specification, an exception handling specification, a return information specification, and a variation initialization specification;

[0049] According to the sub-data in the code safety data conforming to the corresponding sub-specification data, a sub-evaluation value corresponding to the code safety data is determined.

[0050] In one embodiment, if the data information is code compilation data, the evaluation unit is used to:

[0051] Determining information about the number of errors in the code compilation data;

[0052] According to the error quantity information, a sub-evaluation value corresponding to the code compilation data is determined.

[0053] In one embodiment, if the data information is code execution data, the evaluation unit is used to:

[0054] According to the number of passed test cases and the number of test cases in the code execution data, a sub-evaluation value corresponding to the code execution data is determined.

[0055] In one implementation, if the data information is code memory leakage data, the evaluation unit is used to:

[0056] Determine the leakage quantity information in the code memory leakage data;

[0057] According to the leakage quantity information, a sub-evaluation value corresponding to the code leakage data is determined.

[0058] In one implementation, if the data information is code call resource data, the evaluation unit is used to:

[0059] Determine a code performance metric value according to an average memory usage rate, an average central processing unit CPU utilization rate, an average graphics processing unit GPU utilization rate, an average graphics processing unit GPU video memory utilization rate, and a code running time in the code call resource data;

[0060] According to the range of the code performance metric value, a sub-evaluation value corresponding to the code call resource data is determined.

[0061] In one embodiment, the device further comprises a processing unit, configured to:

[0062] When it is determined that the evaluation result does not meet the preset conditions, determining the submodule to be adjusted in the code generation model to be detected; the submodule to be adjusted is the module corresponding to the dimension whose sub-evaluation value does not meet the standard;

[0063] Adjusting the submodule to be adjusted to obtain an adjusted code generation model, and determining an evaluation result of the adjusted code generation model;

[0064] If the evaluation result of the adjusted code generation model meets the preset condition, it is determined that the adjusted code generation model can be put into use online.

[0065] According to a third aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method of the first aspect and possible implementations thereof are implemented.

[0066] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method of the above-mentioned first aspect and its possible implementation methods by executing the executable instructions.

[0067] The technical solution disclosed in this disclosure has the following beneficial effects:

[0068] In an embodiment of the present disclosure, the code generation model to be tested can be determined first; then at least one business code can be generated based on the code generation model to be tested; further, at least one business code can be analyzed and processed to obtain data information of different dimensions; and finally, based on the data information of different dimensions and the evaluation rules, the evaluation result corresponding to the code generation model to be tested can be obtained. The present disclosure determines the data information of different dimensions of the business code generated by the code generation model to be tested fully automatically, and determines the evaluation status based on the data information of different dimensions, so as to quickly, objectively and accurately determine the quality of the code generation model to be tested. Obviously, this method can improve the evaluation efficiency and evaluation accuracy of the code generation model to be tested.

[0069] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 A schematic diagram of an application scenario in this exemplary embodiment is shown;

[0072] Figure 2 A system architecture diagram in this exemplary embodiment is shown;

[0073] Figure 3 A flowchart showing a code generation model evaluation method in this exemplary embodiment is shown;

[0074] Figure 4 A schematic diagram showing the structure of a code generation model evaluation device in this exemplary embodiment is shown;

[0075] Figure 5A schematic structural diagram of an electronic device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure. In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0077] The term "comprising" and any variations thereof in the specification and claims of the present disclosure are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0078] In the embodiments of the present disclosure, one or more, "many" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or plural.

[0079] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order, sequence, size, and priority. For example, the first sub-evaluation value and the second sub-evaluation value in the embodiment of the present disclosure are only used to distinguish different sub-evaluation values. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0080] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, which are schematic diagrams of the present disclosure and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings may be functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in hardware modules or integrated circuits, or in networks, processors or microcontrollers. The implementation method can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein. The features, structures or characteristics described in the present disclosure may be combined in one or more implementations in any suitable manner. In the following description, many specific details are provided to provide a full description of the implementation methods of the present disclosure. However, those skilled in the art should be aware that one or more specific details may be omitted when implementing the technical solution of the present disclosure, or one or more specific details may be replaced by other methods, components, devices, steps, etc.

[0081] It should be noted that in the embodiments of the present disclosure, some existing solutions in the industry such as software, components, models, etc. may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution. In the technical solution of the present disclosure, the terminal, dissemination, and use of casting data are in compliance with relevant national laws and regulations.

[0082] In the related art, Large Language Models (LLM) refers to machine learning models with large-scale parameters and complex computing structures. Specifically, large language models are usually built by deep neural networks and have billions or even hundreds of billions of parameters. Based on different business needs, different types of generative models represented by large language models can be built. For example, a code generation model that uses large language models (such as the GPT series) to generate code. The code generation model is trained to understand and generate the syntax and logical structure of the programming language, thereby generating code snippets based on natural language descriptions.

[0083] However, it is a difficult problem to evaluate the code generation capability of the code generation model comprehensively, accurately and automatically.

[0084] In view of one or more of the above problems, an exemplary embodiment of the present disclosure provides a code generation model evaluation method, through which the code generation model to be tested can be first determined; then at least one business code is generated based on the code generation model to be tested; further, at least one business code is analyzed and processed to obtain data information of different dimensions; and finally, based on the data information of different dimensions and evaluation rules, the evaluation result corresponding to the code generation model to be tested is obtained. The present disclosure determines the data information of different dimensions of the business code generated by the code generation model to be tested fully automatically, and determines the evaluation status based on the data information of different dimensions, so as to quickly and accurately determine the quality of the code generation model to be tested. Obviously, this method can improve the evaluation efficiency and evaluation accuracy of the code generation model to be tested.

[0085] In order to better understand the technical solutions provided by the embodiments of the present disclosure, the following briefly introduces the application scenarios to which the technical solutions provided by the embodiments of the present disclosure are applicable. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present disclosure and are not intended to limit the present disclosure. In specific implementation, the technical solutions provided by the embodiments of the present disclosure can be flexibly applied according to actual needs.

[0086] In the embodiments of the present disclosure, the code generation model evaluation technology can be applied to various code generation models that need to be built based on different scenarios, such as social marketing, video recommendation, business processing, and the like, and business scenarios that require the corresponding code generation models to be evaluated, which are not limited in the embodiments of the present disclosure.

[0087] See also Figure 1 As shown, Figure 1This is an application scenario to which the technical solution of the embodiment of the present disclosure can be applied. In the scenario diagram, multiple terminal devices 101 and electronic devices 102 are included. Among them, the terminal device 101-1, the terminal device 101-2, ..., the terminal device 101-n can be used by different users, that is, the code generation models corresponding to different services can be evaluated synchronously or asynchronously.

[0088] The terminal device 101 and the electronic device 102 , as well as each terminal device 101 , may be directly or indirectly connected to each other through one or more networks 103 .

[0089] In the disclosed embodiment, a user can trigger an evaluation request for a code generation model based on the terminal device 101, and then the terminal device 101 sends the evaluation request for the code generation model to the electronic device 102, so that the electronic device 102 can determine the code generation model to be tested; generate at least one business code based on the code generation model to be tested; analyze and process the at least one business code to obtain data information of different dimensions; and obtain the evaluation result corresponding to the code generation model to be tested according to the data information of different dimensions and the evaluation rules.

[0090] In the disclosed embodiment, Figure 1 The terminal devices 101 may be mobile phones, tablet computers (PADs), personal computers (PCs), smart TVs, smart watches, smart speakers, smart vehicle-mounted devices, and wearable devices, but are not limited thereto. These devices may have functions that support triggering instructions for evaluating the code generation model.

[0091] as well as, Figure 1 The electronic device 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server or cloud server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms, but is not limited to these.

[0092] Of course, the method provided in the embodiment of the present disclosure is not limited to Figure 1 The application scenario shown can also be used in other possible application scenarios, such as an application scenario where the code generation model evaluation method is implemented only by an electronic device, and the embodiments of the present disclosure are not limited thereto.

[0093] See also Figure 2 As shown, Figure 2The system architecture diagram of the operating environment of this exemplary embodiment is shown. The system architecture includes an evaluation system user management module 210 , an evaluation system authority management module 220 , an evaluation task management module 230 , an evaluation agent module 240 and an evaluation result display module 250 .

[0094] In the disclosed embodiment, the evaluation system user management module 210 is used to manage the information of users that can be linked to the evaluation system, such as the user's basic identity information, enterprise internal level information, etc. The evaluation system authority management module 220 is used to manage the user's evaluation authority for each submodule in the code generation model. For example, only the authority to evaluate submodule N in the code generation model is open to user 1, and the authority to evaluate submodule N and submodel M in the code generation model is open to user 5. Optionally, the user's evaluation authority can establish a corresponding relationship with the user's enterprise internal level information, so that the evaluation authority of each user can be determined based on the corresponding relationship.

[0095] In the disclosed embodiment, the evaluation task management module 230 is used to manage the received evaluation request tasks, such as determining that several evaluation request tasks are executed in parallel, or the execution order of the evaluation request tasks, etc., which is not limited in the disclosed embodiment. And, the evaluation result display module 250 is used to display the total evaluation and sub-evaluation values ​​of the code generation model to be tested, and the display method includes graphic filling, line length, etc. For example, assuming that the full score of the sub-evaluation value is 5 points, and the evaluation is 4 points, then fill four-fifths of the graphic, such as a star shape, a cartoon character pattern, etc.

[0096] In the disclosed embodiment, the evaluation agent module 240 includes a data set management submodule, a model management submodule, a reasoning submodule, an evaluation submodule, and a comprehensive scoring submodule. Among them, the data set management submodule is responsible for the management of the code evaluation data set, such as the addition, deletion, modification and query of the data set. The model management submodule is responsible for the download, storage, query, deletion, modification, model details, model KEY data addition, deletion, modification and query, model prompt format, model execution environment management such as container environment, conda environment management and other functions of the model data. The reasoning submodule is responsible for executing the reasoning of the code model and feeding back the reasoning results, such as the reasoning of the local model, the API call reasoning of the cloud model, etc. The evaluation submodule is responsible for implementing the evaluation execution of different dimensions of the code model. The evaluation of each dimension of the code generation model is implemented by an agent, such as the code style consistency agent, the code security processing capability agent, the code compilation pass rate agent, the code execution accuracy agent, the code memory leak assessment agent, the code execution performance agent, etc., supporting the heartbeat function and the evaluation result storage function. The comprehensive scoring submodule is responsible for comprehensively calculating the output results of the dimension agent and outputting the scoring results.

[0097] To further illustrate the technical solution provided by the embodiments of the present disclosure, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present disclosure provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present disclosure. The method may be executed in the order of the method shown in the embodiments or drawings or in parallel during the actual processing process or when the device is executed.

[0098] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of a code generation model evaluation method in an embodiment of the present disclosure. The flow of the method may be executed by an electronic device, for example, and the electronic device may be Figure 1 The electronic device 102 or the electronic device 102 is configured with Figure 2 The server of the system shown in the figure executes the method, and the specific implementation process of the method is as follows:

[0099] Step 301: Determine the code generation model to be tested.

[0100] In the embodiments of the present disclosure, the electronic device can determine the code generation model to be tested based on the identifier of the code generation model carried in the evaluation request sent by the terminal device, or it can determine the code generation model to be tested based on the identifier of the code generation model corresponding to the indication information output by the evaluation task management module. Of course, other methods are also possible, which are not limited to the embodiments of the present disclosure.

[0101] Step 302: Generate at least one service code based on the code generation model to be detected.

[0102] Step 303: Analyze and process at least one business code to obtain data information of different dimensions.

[0103] In an embodiment of the present disclosure, an electronic device can perform operations such as code generation on a code generation model to be tested based on a code evaluation data set, thereby generating at least one corresponding business code. A business code refers to a code generated by a code generation model to be tested based on a prompt in a code evaluation data set, which is used to implement the requirements or functions in the prompt information and may include any number of lines or characters. Then, at least one business code is analyzed and processed to obtain data corresponding to the code generation process and the code generation result, and the obtained data corresponding to the code generation process and the code generation result are classified to obtain data information of different dimensions.

[0104] In the embodiment of the present disclosure, data information of different dimensions includes: at least two of code style data, code security data, code compilation data, code execution data, code memory leakage data, and code call resource data.

[0105] Step 304: Obtain the evaluation result corresponding to the code generation model to be tested according to the data information and evaluation rules of different dimensions.

[0106] In the disclosed embodiment, the electronic device can determine the sub-evaluation value corresponding to each dimension based on the data information of each dimension and the corresponding evaluation sub-rule; the evaluation sub-rule belongs to the evaluation rule; according to the sub-evaluation value corresponding to each dimension, the evaluation result corresponding to the code generation model to be tested is obtained. In other words, in the disclosed embodiment, the electronic device can first determine the sub-evaluation value corresponding to each dimension, and then determine the evaluation result corresponding to the code generation model to be tested in combination with the dimensions required for evaluating the code generation model. For example, the sub-evaluation values ​​of 6 dimensions can be calculated first, but it is determined that 4 dimensions are required for the evaluation, then the sub-evaluation values ​​of the 4 dimensions are selected to determine the evaluation result of the code generation model to be tested.

[0107] In one implementation, considering that the consistency of code style is crucial to maintaining the readability, maintainability and overall quality of the code, the code generation model to be tested can be evaluated from the dimension of code style consistency.

[0108] The electronic device can determine whether the sub-data in the code style data conforms to the sub-specifications in the first preset specification; the first preset specification includes coding specifications, naming specifications, comment specifications, abbreviation specifications, and comment-to-code ratio specifications; based on the data that the sub-data in the code style data conforms to the corresponding sub-specifications, determine the sub-evaluation value corresponding to the code style data.

[0109] Optionally, the electronic device can check whether the coding standard set by the prompt is followed, such as PEP 8--StyleGuide for Python Code for Python or Google Java Style Guide for Java, Google C++ style, Google Objective-C style, Google JavaScript style guide, etc. If the child data meets the coding standard, 1 point is scored.

[0110] Optionally, the electronic device can check whether the naming method set by the prompt is followed to unify the naming method of variables, functions, classes and modules, such as using camelCase or underscores. If the sub-data complies with the naming standard, it will score 1 point.

[0111] Optionally, the electronic device can check whether the code contains comments to explain the function, parameters and return value, as well as any complex logic. If the sub-data meets the annotation specifications, it will score 1 point.

[0112] Optionally, the electronic device can check whether a uniform indentation method is used, such as 4 spaces or 2 spaces, and do not mix spaces and tabs. If the sub-data meets the indentation specification, 1 point will be scored.

[0113] Optionally, the electronic device can check whether the ratio of comments to codes is within a preset range, so as to ensure that each part of the code has appropriate comments. The preset range is, for example, 1:1 or 1:2. If the sub-data meets the specification of the ratio of comments to codes, 1 point is scored.

[0114] It can be seen that the electronic device can evaluate the code style consistency of the code generation model to be tested from five sub-dimensions. If all five sub-dimensions meet the preset sub-specifications, the code style consistency evaluation dimension can get a full score of 5 points.

[0115] In one implementation, the code generation model to be tested can be evaluated from the evaluation dimension of code security capability. The electronic device can determine whether the sub-data in the code security data meets the sub-specifications in the second preset specification; the second preset specification includes vulnerability specifications, exception handling specifications, return information specifications, and variable initialization specifications; according to the data that the sub-data in the code security data meets the corresponding sub-specifications, the sub-evaluation value corresponding to the code security data is determined.

[0116] Optionally, the electronic device can detect security vulnerabilities and potential attack vectors in the code through automated security scanning tools. For example, check whether functions and methods with known security vulnerabilities are used, such as using the gets() function in C++ to detect security vulnerabilities and potential attack vectors. If the sub-data meets any of the security vulnerabilities and potential attack vectors in the vulnerability specification, 1 point is scored.

[0117] Optionally, the electronic device can check whether errors and exceptions are handled correctly and whether sensitive error information is displayed to the user. If the sub-data meets the exception handling specification, 1 point is scored.

[0118] Optionally, the electronic device can check whether the function return value contains null. If the sub-data meets the return information specification, 1 point is scored.

[0119] Optionally, the electronic device may check the global variables to determine whether the function variables have been initialized. If the sub-data meets the variation initialization specification, 1 point is scored.

[0120] It can be seen that electronic devices can evaluate the code security processing capabilities of the code generation model to be tested from four sub-dimensions. If all four sub-dimensions meet the preset sub-specifications, the code style consistency evaluation dimension can get a full score of 5 points.

[0121] In one implementation, the code generation model to be tested can be evaluated from the evaluation dimension of code compilation pass rate. The electronic device can determine the number of errors in the code compilation data; and determine the sub-evaluation value corresponding to the code compilation data according to the number of errors.

[0122] For example, as shown in Table 1, the number of compile errors can be checked and scored according to the score corresponding to the number of compile errors to obtain the corresponding sub-evaluation value.

[0123] compile error number of errors Score Number of errors = 0 5 0 < number of errors <= 2 4 2 < number of errors <= 5 3 5 < number of errors <= 10 2 10 < number of errors <= 20 1 20 <Number of errors 0

[0124] Table 1

[0125] In one embodiment, the code generation model to be tested can be evaluated from the evaluation dimension of code execution accuracy. The code execution accuracy is used to evaluate whether the code can return the correct result after execution. Use predefined test cases to verify the functional correctness of the code. The electronic device can determine the sub-evaluation value corresponding to the code execution data based on the number of test cases passed and the number of test cases in the code execution data. Optionally, the test cases provided in the code evaluation data set can cover the code capabilities, thereby ensuring that both key functions and edge cases are tested.

[0126] For example, if the score is calculated based on the proportion of correctly executed test cases, the total score is 5 points. If there is one test case, each score is 5 points; if there are five test cases, each score is 1 point; if there are ten test cases, each score is 0.5 points. Sub-evaluation value = 5*(number of passed test cases / number of test cases).

[0127] In one embodiment, the code generation model to be detected can be evaluated from the evaluation dimension of code memory leakage. The code memory leakage evaluation uses a code leakage tool to evaluate whether the code has a memory leak. Optionally, the electronic device can determine the leakage quantity information in the code memory leakage data; and determine the sub-evaluation value corresponding to the code leakage data based on the leakage quantity information.

[0128] For example, as shown in Table 2, the number of memory leaks can be checked, and scores can be calculated according to the scores corresponding to the number of memory leaks to obtain corresponding sub-evaluation values.

[0129] Number of memory leaks Score Number of memory leaks = 0 5 Number of memory leaks = 1 4 Number of memory leaks = 2 3 Number of memory leaks = 3 2 Number of memory leaks = 4 1 Number of memory leaks > 4 0

[0130] Table 2

[0131] In one implementation, considering that efficient code is crucial for practical applications, performance evaluation helps to identify potential bottlenecks and optimize code, so the code generation model to be tested can be evaluated from the evaluation dimension of code execution performance. Among them, code execution performance is used to evaluate the performance of the code, the time and resource usage of the code to return the correct result under the same hardware configuration, and measure the code running time, memory usage, central processing unit (CPU) utilization, graphics processing unit (GPU) utilization, GPU memory utilization and other performance indicators.

[0132] Optionally, the electronic device can determine the code performance measurement value based on the average memory usage, average central processing unit CPU utilization, average graphics processing unit GPU utilization, average graphics processing unit GPU video memory utilization and code running time in the code call resource data; and determine the sub-evaluation value corresponding to the code call resource data based on the range of the code performance measurement value.

[0133] Optionally, code performance metric value = test case benchmark running time * test case benchmark memory usage * test case benchmark CPU utilization * test case benchmark GPU utilization * test case benchmark GPU video memory utilization) / (test case actual running time * average memory utilization * average CPU utilization * average GPU utilization * average GPU video memory utilization).

[0134] For example, as shown in Table 3, the code performance metric value may be determined, and the code performance metric value may be scored according to the score corresponding to the code performance metric value to obtain the corresponding sub-evaluation value.

[0135] Code performance metrics Score Code performance metric value > 1 5 1>=Code performance metric value>0.8 4 0.8>=Code performance metric value>0.6 3 0.6>=Code performance metric value>0.4 2 0.4>=Code performance metric value>0.2 1 0.2>=Code performance metric value 0

[0136] Table 3

[0137] In the embodiment of the present disclosure, the electronic device can determine the weight coefficient corresponding to each dimension. The weight coefficient is, for example, 0.2 or 0.3. The weight coefficient can be determined based on actual implementation, and this is not limited in the embodiment of the present disclosure.

[0138] In the embodiment of the present disclosure, the electronic device can determine the sub-evaluation value corresponding to each evaluation dimension based on the evaluation dimension of the code generation model to be tested, and then combine the product of the sub-evaluation value and the corresponding weight coefficient to finally determine the evaluation result of the code generation model to be tested.

[0139] For example, assuming that the evaluation dimensions of the code generation model to be tested are code performance dimension, code execution accuracy and code compilation pass rate, and the sub-evaluation values ​​corresponding to the three dimensions are 3 points, 5 points and 2 points respectively, and the corresponding weight coefficients are 0.2, 0.3 and 0.5 respectively, then the evaluation result of the code generation model to be tested can be determined as: 3*0.2+5*0.3+2*0.5=3.1 points.

[0140] In a possible implementation, the electronic device may also determine the evaluation result to determine whether the large code generation model to be tested can be put into use online.

[0141] Optionally, when it is determined that the evaluation result meets the preset condition, it is determined that the code generation model to be tested can be put into use. The preset condition may be that the evaluation result is greater than a preset threshold, such as 3 or 5, etc. The preset threshold may be determined based on actual implementation, and this is not limited in the embodiments of the present disclosure.

[0142] That is to say, when it is determined that the evaluation result meets the preset conditions, it can be determined that the code generation model to be tested has good performance in all aspects, thereby determining that the code generation model to be tested can be put into use online.

[0143] Optionally, when it is determined that the evaluation result does not meet the preset conditions, the sub-module to be adjusted in the code generation model to be tested is determined; the sub-module to be adjusted is the module corresponding to the dimension whose sub-evaluation value does not meet the standard; the sub-module to be adjusted is adjusted to obtain the adjusted code generation model, and the evaluation result of the adjusted code generation model is determined; if the evaluation result of the adjusted code generation model meets the preset conditions, it is determined that the adjusted code generation model can be put into use online.

[0144] It can be seen that in the embodiments of the present disclosure, when the code generation model to be tested does not meet the preset conditions, the code generation model to be tested can be adjusted in reverse based on the situation indicated by the sub-evaluation value obtained previously, so that the code generation model that can be used online can be quickly obtained, thereby improving the online success rate of the code generation model.

[0145] The exemplary embodiment of the present disclosure also provides a code generation model evaluation device. Figure 4 As shown, the code generation model evaluation device 400 includes the following program modules:

[0146] A determination unit 401 is used to determine a code generation model to be detected;

[0147] A generating unit 402, configured to generate at least one service code based on the code generation model to be detected;

[0148] The obtaining unit 403 is used to analyze and process the at least one business code to obtain data information of different dimensions;

[0149] The evaluation unit 404 is used to obtain the evaluation result corresponding to the code generation model to be tested according to the data information of different dimensions and the evaluation rules.

[0150] In one implementation, the data information of different dimensions includes: at least two of: code style data, code security data, code compilation data, code execution data, code memory leakage data, and code calling resource data.

[0151] In one implementation, the evaluation unit 404 is used to:

[0152] Determine the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule; the evaluation sub-rule belongs to the evaluation rule;

[0153] According to the sub-evaluation value corresponding to each dimension, the evaluation result corresponding to the code generation model to be tested is obtained.

[0154] In one implementation, if the data information is code style data, the evaluation unit 404 is used to:

[0155] Determine whether the sub-data in the code style data conforms to the sub-specifications in the first preset specification; the first preset specification includes coding specifications, naming specifications, comment specifications, abbreviation specifications, and comment-to-code ratio specifications;

[0156] According to the sub-data in the code style data conforming to the corresponding sub-specification data, a sub-evaluation value corresponding to the code style data is determined.

[0157] In one implementation, if the data information is code security data, the evaluation unit 404 is used to:

[0158] Determine whether the sub-data in the code security data conforms to the sub-specifications in the second preset specification; the second preset specification includes a vulnerability specification, an exception handling specification, a return information specification, and a variation initialization specification;

[0159] According to the sub-data in the code safety data conforming to the corresponding sub-specification data, a sub-evaluation value corresponding to the code safety data is determined.

[0160] In one implementation, if the data information is code compilation data, the evaluation unit 404 is used to:

[0161] Determining information about the number of errors in the code compilation data;

[0162] According to the error quantity information, a sub-evaluation value corresponding to the code compilation data is determined.

[0163] In one implementation, if the data information is code execution data, the evaluation unit 404 is used to:

[0164] According to the number of passed test cases and the number of test cases in the code execution data, a sub-evaluation value corresponding to the code execution data is determined.

[0165] In one implementation, if the data information is code memory leakage data, the evaluation unit 404 is used to:

[0166] Determine the leakage quantity information in the code memory leakage data;

[0167] According to the leakage quantity information, a sub-evaluation value corresponding to the code leakage data is determined.

[0168] In one implementation, if the data information is code call resource data, the evaluation unit 404 is used to:

[0169] Determine a code performance metric value according to an average memory usage rate, an average central processing unit CPU utilization rate, an average graphics processing unit GPU utilization rate, an average graphics processing unit GPU video memory utilization rate, and a code running time in the code call resource data;

[0170] According to the range of the code performance metric value, a sub-evaluation value corresponding to the code call resource data is determined.

[0171] In one embodiment, the device further comprises a processing unit, configured to:

[0172] When it is determined that the evaluation result does not meet the preset condition, determining the submodule to be adjusted in the code generation model to be detected; the submodule to be adjusted is the module corresponding to the dimension whose sub-evaluation value does not meet the standard;

[0173] Adjusting the submodule to be adjusted to obtain an adjusted code generation model, and determining an evaluation result of the adjusted code generation model;

[0174] If the evaluation result of the adjusted code generation model meets the preset condition, it is determined that the adjusted code generation model can be put into use online.

[0175] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0176] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0177] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned code generation model evaluation method is implemented.

[0178] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0179] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0180] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0181] Computer programs can be carried or transmitted through electrical, magnetic, optical, electromagnetic, infrared and other signals. Electronic devices can convert signals carrying computer programs into digital signals to run computer programs. When a computer program runs on an electronic device, its code is used to enable the electronic device to execute (more specifically, it can enable the processor of the electronic device to execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned code generation model evaluation method, which includes the following steps: Step 301: Determine the code generation model to be tested; Step 302: Generate at least one business code based on the code generation model to be tested; Step 303: Analyze and process at least one business code to obtain data information of different dimensions; Step 304: According to the data information of different dimensions and the evaluation rules, obtain the evaluation results corresponding to the code generation model to be tested.

[0182] By implementing the above method steps through a computer program, the code generation model to be tested can be first determined; then at least one business code can be generated based on the code generation model to be tested; further, at least one business code is analyzed and processed to obtain data information of different dimensions; and finally, based on the data information of different dimensions and the evaluation rules, the evaluation result corresponding to the code generation model to be tested is obtained. The present invention determines the data information of different dimensions of the business code generated by the code generation model to be tested fully automatically, and determines the evaluation situation based on the data information of different dimensions, so as to quickly and accurately determine the quality of the code generation model to be tested. Obviously, in this way, the evaluation efficiency and accuracy of the code generation model to be tested can be improved.

[0183] The exemplary embodiments of the present disclosure also provide an electronic device, which may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions.

[0184] Reference below Figure 5 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Figure 5 The electronic device 500 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0185] like Figure 5 As shown, the electronic device 500 may include: a processor 510 , a memory 520 , a bus 530 , an I / O (input / output) interface 540 , a network adapter 550 , and a display 580 .

[0186] The memory 520 may include a volatile memory, such as a RAM 521, a cache unit 522, and may also include a non-volatile memory, such as a ROM 523. The memory 520 may also include one or more program modules 524, such program modules 524 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 524 may include each module in the above-mentioned device.

[0187] The processor 510 may include one or more processing units. For example, the processor 510 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0188] The processor 510 can be used to execute executable instructions stored in the memory 520, such as executing the above-mentioned code generation model evaluation method, which includes the following steps: Step 301: determine the code generation model to be tested; Step 302: generate at least one business code based on the code generation model to be tested; Step 303: analyze and process at least one business code to obtain data information of different dimensions; Step 304: obtain the evaluation result corresponding to the code generation model to be tested according to the data information of different dimensions and the evaluation rules.

[0189] By executing the above method steps through the processor 510, the code generation model to be tested can be first determined; then at least one business code can be generated based on the code generation model to be tested; further, at least one business code is analyzed and processed to obtain data information of different dimensions; and finally, based on the data information of different dimensions and the evaluation rules, the evaluation result corresponding to the code generation model to be tested is obtained. The present invention determines the data information of different dimensions of the business code generated by the code generation model to be tested fully automatically, and determines the evaluation situation based on the data information of different dimensions, so as to quickly and accurately determine the quality of the code generation model to be tested. Obviously, in this way, the evaluation efficiency and evaluation accuracy of the code generation model to be tested can be improved.

[0190] The bus 530 is used to realize the connection between different components of the electronic device 500, and may include a data bus, an address bus, and a control bus.

[0191] The electronic device 500 can communicate with one or more external devices 600 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 540 .

[0192] The electronic device 500 can communicate with one or more networks through the network adapter 550. For example, the network adapter 850 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 550 can communicate with other modules of the electronic device 500 through the bus 530.

[0193] The electronic device 500 may display a graphical user interface, such as sub-assessment values, assessment results, etc., through the display 580 .

[0194] although Figure 5 Not shown, other hardware and / or software modules may also be provided in the electronic device 500, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0195] As can be seen from the above, the technical solution of the present disclosure can be implemented as a method, an apparatus, a system, a computer program product, a storage medium, an electronic device, etc. Those skilled in the art can understand that various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, such as being respectively referred to as a "circuit", "module" or "system".

[0196] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. A code generation model evaluation method, characterized in that: The method comprises: Determine the code generation model to be tested; Generate at least one service code based on the code generation model to be detected; Analyze and process the at least one business code to obtain data information of different dimensions; According to the data information and evaluation rules of the different dimensions, an evaluation result corresponding to the code generation model to be tested is obtained.

2. The method according to claim 1, characterized in that The data information of different dimensions includes: at least two of: code style data, code security data, code compilation data, code execution data, code memory leakage data and code calling resource data.

3. The method according to claim 2, characterized in that Obtaining the evaluation result corresponding to the code generation model to be tested according to the data information and evaluation rules of the different dimensions, including: Determine the sub-evaluation value corresponding to each dimension according to the data information of each dimension and the corresponding evaluation sub-rule; the evaluation sub-rule belongs to the evaluation rule; According to the sub-evaluation value corresponding to each dimension, the evaluation result corresponding to the code generation model to be tested is obtained.

4. The method according to claim 3, characterized in that: If the data information is code style data, then according to the data information of each dimension and the corresponding evaluation sub-rule, the sub-evaluation value corresponding to each dimension is determined, including: Determine whether the sub-data in the code style data conforms to the sub-specifications in the first preset specification; the first preset specification includes coding specifications, naming specifications, comment specifications, abbreviation specifications, and comment-to-code ratio specifications; According to the sub-data in the code style data conforming to the corresponding sub-specification data, a sub-evaluation value corresponding to the code style data is determined.

5. The method according to claim 3, characterized in that: If the data information is code execution data, then according to the data information of each dimension and the corresponding evaluation sub-rule, the sub-evaluation value corresponding to each dimension is determined, including: According to the number of passed test cases and the number of test cases in the code execution data, a sub-evaluation value corresponding to the code execution data is determined.

6. The method according to claim 3, characterized in that If the data information is code call resource data, then according to the data information of each dimension and the corresponding evaluation sub-rule, the sub-evaluation value corresponding to each dimension is determined, including: Determine a code performance metric value according to an average memory usage rate, an average central processing unit CPU utilization rate, an average graphics processing unit GPU utilization rate, an average graphics processing unit GPU video memory utilization rate, and a code running time in the code call resource data; According to the range of the code performance metric value, a sub-evaluation value corresponding to the code call resource data is determined.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When it is determined that the evaluation result does not meet the preset conditions, determining the submodule to be adjusted in the code generation model to be detected; the submodule to be adjusted is the module corresponding to the dimension whose sub-evaluation value does not meet the standard; Adjusting the submodule to be adjusted to obtain an adjusted code generation model, and determining an evaluation result of the adjusted code generation model; If the evaluation result of the adjusted code generation model meets the preset condition, it is determined that the adjusted code generation model can be put into use online.

8. A code generation model evaluation device, characterized in that: The device comprises: A determination unit, used for determining a code generation model to be detected; A generating unit, configured to generate at least one service code based on the code generation model to be detected; An obtaining unit, configured to analyze and process the at least one business code to obtain data information of different dimensions; The evaluation unit is used to obtain the evaluation result corresponding to the code generation model to be tested according to the data information of different dimensions and the evaluation rules.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images