Evaluation method, device and equipment of code large model and medium

Abstract

The invention relates to a code large model evaluation method and device, equipment and a medium. The method comprises the following steps: analyzing code attributes of running codes of a model to obtain a code attribute set; performing clustering analysis on the code attribute set to generate an attribute clustering result; determining a model evaluation scene according to the attribute clustering result; based on the model evaluation scene, generating a model evaluation index according to a preset scene-index mapping rule; based on the model evaluation scene, evaluating the operation code by adopting the model evaluation index to obtain model performance data; and performing multi-dimensional analysis on the model performance data to generate a comprehensive evaluation report. By adopting the method, the performance evaluation of the code large model can be automatically and efficiently completed, and the evaluation pertinence and the result credibility are improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence model evaluation and verification, and in particular relates to a method, apparatus, device and medium for evaluating large code models. Background Technology

[0002] With the development of technologies at the intersection of artificial intelligence and software engineering, CodeLarge Model (CodeLLM) technology has emerged. This technology supports the entire software engineering process, including code generation, software testing, defect detection, and code maintenance, significantly improving the automation level of software development. Current CodeLarge Model evaluation methods often focus on single task scenarios, such as using tools like HumanEval to evaluate code generation tasks and verifying functional correctness through syntax checking or unit testing. Current CodeLarge Model evaluation methods suffer from the following problems: insufficient scenario coverage, lacking a systematic evaluation of engineering scenarios such as software testing, defect detection, and software maintenance; insufficient scientific rigor of metrics, failing to cover multi-dimensional indicators such as robustness and engineering integration efficiency; and a fragmented evaluation system, with existing evaluation tools only targeting specific sub-tasks, failing to form a complete evaluation system covering the entire software engineering lifecycle. This results in biased and unreliable evaluation results, making it difficult to guide model optimization and engineering applications. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, device, and medium for evaluating large code models that can solve the above problems.

[0004] Firstly, this application provides a method for evaluating large code models, including:

[0005] Analyze the code attributes of the model's running code to obtain a set of code attributes;

[0006] Perform cluster analysis on the code attribute set and generate attribute clustering results;

[0007] Based on the attribute clustering results, determine the model evaluation scenario;

[0008] Based on the model evaluation scenario, model evaluation metrics are generated according to the preset scenario-metric mapping rules;

[0009] Based on the model evaluation scenario, model evaluation metrics are used to evaluate the running code and obtain model performance data;

[0010] Perform multi-dimensional analysis on model performance data to generate a comprehensive evaluation report.

[0011] In one embodiment, the code attributes of the model's running code are analyzed to obtain a set of code attributes, including:

[0012] Perform functional identification on the running code to determine the code's functional type;

[0013] For each code functionality type, perform the following operations:

[0014] Extract surface layer attributes from the running code to obtain a subset of surface layer attributes;

[0015] The running code is parsed using an abstract syntax tree to obtain a subset of syntax attributes;

[0016] Perform semantic dependency analysis on the running code to obtain a subset of semantic attributes;

[0017] The surface layer attribute subset, syntactic attribute subset, and semantic attribute subset are concatenated into feature vectors to form attribute feature vectors;

[0018] Integrate the attribute feature vectors corresponding to all code function types to form a code attribute set.

[0019] In one embodiment, cluster analysis is performed on the code attribute set to generate attribute clustering results, including:

[0020] Obtain the task requirements for the corresponding functional type of the code attribute set;

[0021] Construct a task-aware attention network based on task requirements;

[0022] Based on the code attribute set, a task-aware attention network is used to generate task feature weights;

[0023] The attribute feature vectors in the code attribute set are weighted using task feature weights to obtain enhanced attribute feature vectors;

[0024] Based on a preset clustering threshold, the enhanced attribute feature vectors are clustered to form attribute clustering results.

[0025] In one embodiment, the model evaluation scenario is determined based on the attribute clustering results, including:

[0026] Within each cluster in the attribute clustering results, statistical analysis is performed on the enhanced attribute feature vectors to obtain cluster attributes and attribute distribution characteristics;

[0027] Based on cluster attributes, scene elements are determined according to preset attribute-element mapping rules;

[0028] Based on attribute distribution characteristics, the scene frame is determined according to the preset pattern-frame mapping rules;

[0029] Combine scene elements and scene framework to form a model evaluation scene.

[0030] In one embodiment, based on a model evaluation scenario, model evaluation metrics are used to evaluate the running code to obtain model performance data, including:

[0031] Based on the model evaluation metrics, the corresponding evaluation program is matched from the preset evaluation program library;

[0032] Based on the model evaluation scenario, the test parameters of the evaluation program are determined, and the test parameters are configured into the corresponding evaluation program to obtain the evaluation program with complete parameter configuration.

[0033] Associate the running code with the evaluation program with configured parameters to form an indicator-program-code association group;

[0034] For each metric-program-code association group, an evaluation program with configured parameters is used to perform evaluation operations on the running code to obtain raw evaluation data;

[0035] Integrate the raw evaluation data of all related metrics, programs, and code groups to form model performance data.

[0036] In one embodiment, the model performance data is analyzed from multiple dimensions to generate a comprehensive evaluation report, including:

[0037] Based on the model evaluation metrics, determine the analytical dimensions for multi-dimensional analysis;

[0038] Based on the code function type, the model performance data is split according to the code function type to obtain categorized performance data;

[0039] Based on the analysis dimensions, the categorized performance data are scored according to the preset performance scoring rules to obtain the categorized performance score;

[0040] Based on the preset performance scoring weights, the performance scores of different categories are weighted and calculated to generate a comprehensive evaluation score.

[0041] By combining the performance scores by category and the overall evaluation score, and organizing the content according to the preset report framework, a comprehensive evaluation report is generated.

[0042] In one embodiment, the categorical performance scores are weighted according to preset performance scoring weights to generate a comprehensive evaluation score, which is achieved through the following formula:

[0043]

[0044] in, For the overall evaluation score, L represents the total number of code function types. The weight coefficient for the function of the l-th type of code. Let Q be the weight vector for the multidimensional analysis, and let Q be the number of dimensions in the multidimensional analysis. Let l be the categorical performance score vector for the l-th type of code functionality. , where 'a' is the dimension of the attribute feature vector. The standard deviation of the performance scores for all categories. Here is the calibration function, where .

[0045] Secondly, this application also provides an evaluation device for a large code model, comprising:

[0046] The code attribute analysis module is used to analyze the code attributes of the model's running code and obtain a set of code attributes;

[0047] The attribute clustering analysis module is used to perform clustering analysis on the code attribute set and generate attribute clustering results;

[0048] The evaluation scenario determination module is used to determine the model evaluation scenario based on the attribute clustering results;

[0049] The evaluation metric generation module is used to generate model evaluation metrics based on the model evaluation scenario and according to the preset scenario-metric mapping rules.

[0050] The model performance evaluation module is used to evaluate the running code based on model evaluation scenarios and model evaluation metrics to obtain model performance data.

[0051] The comprehensive evaluation report module is used to perform multi-dimensional analysis of model performance data and generate a comprehensive evaluation report.

[0052] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described evaluation method for a large code model.

[0053] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described evaluation method for a large code model.

[0054] The aforementioned method, apparatus, equipment, and medium for evaluating large-scale code models obtain a code attribute set by analyzing the code attributes of the model's running code. Cluster analysis is then performed on this attribute set to generate attribute clustering results. Based on these results, a model evaluation scenario is determined. Based on the evaluation scenario, model evaluation indicators are generated according to a preset scenario-indicator mapping rule. These indicators are then used to evaluate the running code to obtain model performance data. Multi-dimensional analysis of the performance data is then performed to generate a comprehensive evaluation report. This system can systematically cover the entire software engineering lifecycle, including software testing, defect detection, and software maintenance, addressing the shortcomings of existing evaluation scenario coverage. By using scenario-adapted scientific evaluation indicators, it overcomes the limitations of single indicators, forming a unified evaluation system, avoiding fragmented evaluation, and enabling more comprehensive and objective evaluation results through multi-dimensional analysis. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart of an evaluation method for a large code model according to the present invention;

[0057] Figure 2 This is a structural diagram of a code large model evaluation device according to the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] In one embodiment, such as Figure 1 As shown, a method for evaluating large code models is provided. This embodiment illustrates the method using a system including terminals and servers as an example. It is understood that this method can also be applied independently to servers or terminals integrating evaluation modules. When there is a need for full-process capability evaluation, such as large code model R&D optimization and engineering selection, the terminal uploads the running code of the model to be evaluated and related task requirements to the server. The server calls dataset and attribute analysis tools to complete code attribute extraction and clustering to determine the evaluation scenario. Through network interaction, it calls the adapted evaluation program of the preset evaluation program library server, configures the parameters, and executes the evaluation. The generated performance data is analyzed by the server to generate a comprehensive evaluation report and fed back to the terminal. In this embodiment, the method includes the following steps:

[0060] S01, analyze the code attributes of the running code of the model to obtain the code attribute set.

[0061] Among them, "running code" refers to the code fragments, files, and related code data output or run by the large code model under evaluation when performing target tasks such as code generation, software testing, defect detection, and software maintenance; "code attributes" refers to attribute information that can characterize the characteristics and functional relationships of the running code itself, including but not limited to surface features, syntactic features, semantic features, and functional type features; and "code attribute set" refers to the set of attribute information formed by extracting and characterizing various attributes of the running code. Functional type identification can be performed on the model's running code. For the identified functional type, relevant attributes of the running code are extracted from multiple dimensions. After feature vectorization or standardization of each extracted attribute, they are summarized and integrated to obtain a code attribute set that comprehensively reflects the characteristics of the running code.

[0062] S02, perform cluster analysis on the code attribute set and generate attribute clustering results.

[0063] Cluster analysis refers to a data analysis and processing method that groups and classifies attribute information with similar characteristics in a code attribute set based on preset clustering rules or algorithms. Attribute clustering results are the analysis results data formed after cluster analysis, containing several clusters and the characteristics of each cluster (such as attribute composition within a cluster, attribute similarity degree, and differences between clusters). In implementation, clustering algorithms (such as K-means clustering, hierarchical clustering, density clustering, etc.) or customized clustering strategies designed according to the needs of large code model evaluation scenarios can be used. Clustering parameters (such as the range of the number of clusters, attribute similarity threshold, etc.) are set according to preset clustering objectives (such as the correlation between attributes and code functions, attribute feature similarity, etc.). The clustering algorithm groups attribute information with similar characteristics or functional correlation in the code attribute set into the same cluster, generating attribute clustering results containing the attribute composition of each cluster, statistical values ​​of attribute features within a cluster, and distinguishing features between clusters, providing data support for subsequent determination of model evaluation scenarios.

[0064] S03. Determine the model evaluation scenario based on the attribute clustering results.

[0065] Among them, the model evaluation scenario refers to a specific evaluation scenario framework determined for the capabilities to be evaluated in the large code model and adapted to the actual application needs of software engineering. It covers task scenario types related to the application of the large code model, such as code generation, software testing, defect detection, and software maintenance. In implementation, the core features of each cluster in the attribute clustering results can be extracted and interpreted to determine the code functional attribute category corresponding to each cluster (such as test case generation related attributes, defect repair related attributes, etc.). It can be combined with the cluster feature-evaluation scenario mapping rules (pre-constructed based on the typical application scenario requirements of the large code model in the software engineering field, such as mapping clusters containing test coverage attributes and assertion generation attributes to software testing scenarios). By matching the core features of the clusters with the preset scenario features, the basic evaluation scenario direction corresponding to each cluster is determined. The basic scenario directions are integrated to form a model evaluation scenario that covers the core capabilities of the model to be evaluated and is consistent with the actual engineering application requirements, thus meeting the targeted needs of subsequent evaluations.

[0066] S04. Based on the model evaluation scenario, generate model evaluation metrics according to the preset scenario-metric mapping rules.

[0067] Among them, the preset scenario-metric mapping rule refers to the set of rules pre-built based on the technical specifications, engineering practice requirements and performance evaluation logic in the field of large code model evaluation, which associates different evaluation scenarios with corresponding core evaluation dimensions and specific measurement standards. This rule can determine the correspondence between scenarios and metrics (such as test case coverage, assertion validity and other metrics corresponding to software testing scenarios). The model evaluation metrics refer to the specific metrics used to quantitatively or qualitatively evaluate the performance of large code models in the target evaluation scenario, which can cover core evaluation dimensions such as functional correctness, efficiency, robustness and engineering adaptability. In implementation, a rule base storing preset scenario-index mapping rules can be called. Scenario matching techniques (such as keyword matching and scenario feature vector similarity matching) are used to match the determined model evaluation scenario with the scenario type in the rule base, obtaining the initial set of evaluation indicators corresponding to the scenario. Combined with the actual application scenario of the model to be evaluated (such as strengthening the engineering adaptability index verification for models oriented towards enterprise-level software engineering), the initial indicator set is verified for rationality and adjusted as necessary (such as supplementing personalized indicators for specific scenarios or filtering redundant indicators), generating scenario-adaptive model evaluation indicators to achieve the core evaluation requirements of model performance covering the scenario.

[0068] S05, based on the model evaluation scenario, uses model evaluation metrics to evaluate the running code and obtain model performance data.

[0069] Among them, model performance data refers to the raw results and preliminary processed data set obtained through evaluation operations, reflecting the specific performance of the model under various evaluation indicators. In implementation, based on the characteristics of the model evaluation scenario (e.g., software testing scenarios require associating the code under test with the testing environment, defect detection scenarios require loading a defect sample library) and the type of model evaluation indicators (e.g., quantitative indicators require configuring data acquisition precision, qualitative indicators require configuring judgment criteria), evaluation resource matching technology (e.g., calling suitable code syntax analysis tools, unit testing frameworks, performance monitoring tools, etc. from a preset evaluation tool library) is adopted. The parameters of the evaluation tools are configured according to the scenario requirements (e.g., configuring the context input length for code generation scenarios, configuring the timing start / end nodes for efficiency indicators). The running code is input into the configured evaluation tool, and evaluation operations are executed according to the preset process (e.g., syntax verification, test case execution, response time statistics, etc.), collecting the raw evaluation results corresponding to each indicator in real time. Basic processing (e.g., outlier removal, data format unification, unit standardization) is performed on the raw results, and they are categorized and integrated according to indicators to form model performance data covering all evaluation indicators.

[0070] S06 performs multi-dimensional analysis of model performance data and generates a comprehensive evaluation report.

[0071] Among them, multi-dimensional analysis refers to the systematic data processing process of statistical calculation, intra-dimensional verification, and cross-dimensional correlation analysis of model performance data based on the evaluation dimensions of the large code model evaluation (consistent with the dimensions to which the model evaluation indicators belong, such as functional performance, running efficiency, and environmental adaptability); the comprehensive evaluation report refers to a structured document that integrates the results of multi-dimensional analysis and presents the model's performance, strengths and weaknesses, and evaluation conclusions in the target evaluation scenario. In implementation, specific dimensions for multi-dimensional analysis can be determined based on model evaluation metrics (such as syntax accuracy and generation speed for code generation scenarios, and detection rate and response time for defect detection scenarios). Data statistical tools (such as Python Pandas library and Excel data analysis functions) or visualization analysis tools (such as Matplotlib and Tableau) can be used to perform dimensional statistics on model performance data (such as calculating the indicator compliance rate and numerical distribution analysis) and cross-dimensional comparisons (such as verifying the differences in robustness indicators under different scenarios). Combined with preset performance scoring rules (such as calculating the comprehensive score by configuring weights according to the importance of indicators), performance scores for each dimension and an overall comprehensive score are generated. All analysis content is integrated according to a preset report framework (which may include modules such as data overview, dimensional analysis results, comprehensive score, and conclusions and suggestions) to form a comprehensive evaluation report that reflects the overall performance of the model.

[0072] The aforementioned method for evaluating a large code model analyzes the code attributes of the model's running code to obtain a set of code attributes, forming a comprehensive attribute set reflecting the characteristics of the code. Cluster analysis is performed on the code attribute set to generate attribute clustering results, achieving intelligent grouping based on attribute similarity. Based on the attribute clustering results, the model evaluation scenario is determined, and cluster cluster attributes and distribution characteristics are mapped to specific scenario elements and frameworks, dynamically adapting to actual software engineering needs. Based on the model evaluation scenario, model evaluation indicators are generated according to preset scenario-indicator mapping rules, ensuring that the indicators scientifically cover dimensions such as functional correctness and robustness. These indicators are used to evaluate the running code to obtain model performance data, and automated testing is achieved by matching the evaluation program and configuring parameters. Multi-dimensional analysis of the model performance data generates a comprehensive evaluation report, integrating categorical scoring and weighted comprehensive scores. This method overcomes the limitations of existing technologies that focus on a single task; the scenario-indicator mapping, based on domain knowledge, improves the relevance of the indicators and overcomes insufficient scientific rigor; multi-dimensional analysis constructs a unified evaluation system, avoiding fragmented evaluation and improving the overall efficiency, relevance, and reliability of automated evaluation.

[0073] In one embodiment, the code attributes of the model's running code are analyzed to obtain a set of code attributes, including:

[0074] S11, perform function identification on the running code to determine the code function type;

[0075] S12, for each code function type, perform the following operations:

[0076] S13, extract surface layer attributes from the running code to obtain a subset of surface layer attributes;

[0077] S14, perform abstract syntax tree parsing on the running code to obtain a subset of syntax attributes;

[0078] S15, Perform semantic dependency analysis on the running code to obtain a subset of semantic attributes;

[0079] S16, concatenate the surface layer attribute subset, syntactic attribute subset and semantic attribute subset into feature vectors to form attribute feature vectors;

[0080] S17 integrates the attribute feature vectors corresponding to all code function types to form a code attribute set.

[0081] Specifically, functional identification of running code can be based on the actual output of a large code model in software engineering tasks such as code generation, software testing, defect detection, or software maintenance. Functional classification algorithms (such as keyword matching or machine learning models) can be used to determine code function types, such as dividing code into generation, testing, or repair categories. For each identified code function type, multi-level attribute extraction operations are performed. This involves extracting surface-level attributes from the running code to obtain subsets of surface-level attributes such as line count, comment density, or identifier naming patterns. The code structure is analyzed using abstract syntax tree parsing tools (such as ANTLR or Tree-sitter) to extract subsets of syntactic attributes such as control flow nodes or data type definitions. Semantic dependency analysis (such as using program dependency graphs or symbolic execution techniques) is then performed to capture subsets of semantic attributes such as variable scope or API call relationships. These subsets are then processed into feature vectors (e.g., converted to numerical vectors through normalization or embedding models) and concatenated into a unified attribute feature vector. Finally, the attribute feature vectors corresponding to all code function types are integrated, and a code attribute set can be formed through array or matrix aggregation.

[0082] In one embodiment, cluster analysis is performed on the code attribute set to generate attribute clustering results, including:

[0083] S21, Obtain the task requirements for the corresponding functional type of the code attribute set;

[0084] S22, Construct a task-aware attention network according to task requirements;

[0085] S23, Based on the code attribute set, a task-aware attention network is used to generate task feature weights;

[0086] S24, The attribute feature vectors in the code attribute set are weighted using task feature weights to obtain the enhanced attribute feature vectors;

[0087] S25. Based on a preset clustering threshold, cluster the enhanced attribute feature vectors to form attribute clustering results.

[0088] For example, task requirements corresponding to the code function type (such as code generation, software testing, defect detection, software maintenance, etc.) of the code attribute set can be obtained from the evaluation task configuration file or the evaluation requirements input by the user. The evaluation objectives corresponding to each function type (such as the syntax correctness of code generation, the improvement of test case coverage in software testing, etc.) can be clarified. Based on these task requirements, a task-aware attention network can be constructed using neural network frameworks such as TensorFlow or PyTorch. This network includes an input layer, a hidden layer, and an output layer. The input layer receives attribute feature vectors and task requirement features, the hidden layer learns the relationship between task requirements and attribute features through an attention mechanism, and the output layer outputs task feature weights. The attribute feature vectors from the code attribute set and the extracted task requirement features are input into the constructed task-aware attention network. The network's internal attention score is then calculated (i.e., based on the relationship between attribute features and task requirements). The process involves calculating attention weights based on semantic similarity and functional relevance of task requirements, generating task feature weights for each attribute feature vector corresponding to the current task requirements, representing the importance of attribute features to task requirements. These task feature weights are then used to perform element-wise weighting operations on the corresponding attribute feature vectors in the code attribute set; that is, each dimension value of the attribute feature vector is multiplied by its corresponding task feature weight, resulting in enhanced attribute feature vectors that highlight task-related attribute features and suppress interference from irrelevant attributes. Based on a preset clustering threshold (this threshold is pre-set based on the attribute similarity requirements of the evaluation scenario, such as a similarity threshold of 0.7), K-means clustering or hierarchical clustering algorithms can be used to calculate the similarity of all enhanced attribute feature vectors. Enhanced attribute feature vectors with similarity greater than the preset clustering threshold are grouped into the same cluster, forming an attribute clustering result containing several clusters, the attribute composition of each cluster, and statistical values ​​of attribute similarity within each cluster.

[0089] In one embodiment, the model evaluation scenario is determined based on the attribute clustering results, including:

[0090] S31, within each cluster in the attribute clustering results, perform statistical analysis on the enhanced attribute feature vectors to obtain cluster attributes and attribute distribution characteristics;

[0091] S32, based on cluster attributes, determine scene elements according to preset attribute-element mapping rules;

[0092] S33, based on attribute distribution characteristics, determine the scene frame according to the preset pattern-frame mapping rules;

[0093] S34 combines scene elements and scene framework to form a model evaluation scene.

[0094] Specifically, within each cluster included in the attribute clustering results, the Pandas or NumPy libraries in Python can be used to perform statistical analysis on all enhanced attribute feature vectors (i.e., attribute vectors that highlight task-related features after being weighted by task feature weights). This includes calculating the mean, variance, maximum and minimum values, and frequency of occurrence of each dimension of the enhanced attribute feature vectors within the cluster. These statistics are used to extract cluster attributes that characterize the core features of the cluster. (For example, if the mean of test case coverage and assertion validity-related dimensions in the enhanced attribute feature vectors within the cluster exceeds 60%, then the cluster is considered a cluster.) Cluster attributes are associated with software testing, and attribute distribution characteristics are obtained simultaneously (e.g., a variance of <0.2 indicates concentrated attribute distribution, variance ≥0.2 indicates dispersed distribution, or a certain type of feature value appearing more than 70% of the time indicates that the feature dominates the distribution). Based on the cluster attributes obtained above, preset attribute-element mapping rules can be invoked (associating cluster attributes with specific evaluation scenario elements in key-value pair form, such as software testing associated attributes corresponding to test case generation elements and assertion generation elements, and defect detection associated attributes corresponding to static detection elements and dynamic fuzz elements), which can be matched through keywords ( For example, text matching based on Jaccard similarity matches cluster cluster attributes with keys in the rules to determine one or more scene elements corresponding to the cluster. Based on the acquired attribute distribution features, preset pattern-frame mapping rules can be invoked (associating attribute distribution feature patterns with evaluation scene frame types; for example, a concentrated attribute distribution (variance < 0.2) corresponds to a single task scene frame (containing three modules: task objective, core operation, and result verification), while a dispersed attribute distribution (variance ≥ 0.2) corresponds to a composite task scene frame (containing five modules: task collaboration, subtask decomposition, and cross-subtask verification). If a certain feature dominates the distribution (frequency accounting for more than 70%), it corresponds to a focused scenario framework. After determining the pattern of the attribute distribution feature through statistical tools (such as SPSS), the matching rules determine the corresponding scenario framework. The determined scenario elements are then placed into the corresponding modules of the scenario framework according to their functional relationships. For example, test case generation elements and assertion generation elements are embedded into the core operation module of a single task scenario framework. The task objectives are supplemented as software testing capability evaluation and result verification as test coverage and assertion validity verification, forming a complete model evaluation scenario (such as software test case and assertion generation evaluation scenario).

[0095] In one embodiment, based on a model evaluation scenario, model evaluation metrics are used to evaluate the running code to obtain model performance data, including:

[0096] S41, based on the model evaluation metrics, match the corresponding evaluation program from the preset evaluation program library;

[0097] S42, Based on the model evaluation scenario, determine the test parameters of the evaluation program, and configure the test parameters to the corresponding evaluation program to obtain the evaluation program with complete parameter configuration;

[0098] S43 associates the running code with the evaluation program with the completed parameter configuration to form an indicator-program-code association group;

[0099] S44, for each indicator-program-code association group, the evaluation program with configured parameters performs evaluation operations on the running code to obtain raw evaluation data;

[0100] S45 integrates the raw evaluation data of all indicator-program-code related groups to form model performance data.

[0101] For example, the preset evaluation program library can be a pre-built and stored structured program collection on a local server or in the cloud (such as Alibaba Cloud OSS, AWS S3). The library stores corresponding executable evaluation programs according to evaluation metrics (such as syntax checking programs like ESLint (JavaScript) and PyLint (Python) for code generation syntax correctness metrics, and coverage statistics programs like JaCoCo (Java) and Coverage.py (Python) for test case coverage metrics). Each program is associated with a metric tag. Based on the name and dimension of the current model evaluation metric (such as functional correctness dimension and efficiency dimension), the library uses a keyword matching algorithm (such as Jaccard similarity matching) to search for programs with the same or similar metric tags. For example, the test case coverage metric matches JaCoCov0.8.10.exe, thus completing the matching of the corresponding evaluation program. The test parameters of the evaluation program are determined based on the specific needs of the model evaluation scenario. For example, in the file-level code generation evaluation scenario, the evaluation program corresponding to the code generation speed indicator (such as a custom timing program) needs to be configured with parameters such as context input length (such as 1000 lines of project-level code dependencies), code generation termination judgment condition (such as generating a complete function body or file end mark), timing start node (when the model receives input), and timing termination node (when the model outputs complete code). In the software test case generation scenario, the JaCoCo program corresponding to the test case coverage indicator needs to be configured with parameters such as the path of the code under test (such as D:\test\paymentModule\), the test case execution path (such as D:\test\testCases\), and the coverage statistics granularity (such as class level, method level). After the parameters are determined, they are configured to the corresponding evaluation program through the configuration file or command line parameter input method. For example, for the JaCoCo program, the sourcepath and destfile field values ​​are modified in the jacoco.exec configuration file, or the parameters are passed through the command line to obtain the evaluation program with the parameters configured. Establish an association mapping table. Each record in the table contains three fields: model evaluation metric, the path of the evaluation program with configured parameters, and the storage path of the running code. A single model evaluation metric, its matching and configured evaluation program, and the corresponding running code (e.g., the payment module code `payment.java` in the test case generation scenario) under the same evaluation task are associated through field values ​​to form a complete metric-program-code association group. For example, metric: test case coverage; program path: D:\tools\JaCoCo\jacoco.exe; code path: D:\test\paymentModule\payment.java. All association groups are stored in the association group table of a relational database.For each metric-program-code association group, the evaluation program path and running code path are read from the database. The configured evaluation program is launched via the server operating system's process call interface (such as the CreateProcess function in Windows or the fork / exec function in Linux), and the running code is passed as input to the program. For example, when calling the JaCoCo program, the program reads the payment.java code and corresponding test cases, executes the test cases, and counts the number of covered classes and methods, outputting raw evaluation data such as: Number of covered classes: 8, Total number of classes: 10; Number of covered methods: 25, Total number of methods: 30. The generation speed evaluation program records the time taken from receiving input to outputting complete code, which is 12.5 seconds. The original evaluation data of all related groups can be read through Python's Pandas library or SQL query statements, and integrated according to model evaluation indicators. For example, all original data related to test case coverage (number of covered classes, total number of classes, number of covered methods, total number of methods) are summarized into the test case coverage data table, and the time-consuming data related to code generation speed are summarized into the code generation efficiency data table. The integrated data includes fields such as related group ID, indicator name, original data item 1, original data item 2, and evaluation time, forming structured model performance data for subsequent analysis.

[0102] In one embodiment, the model performance data is analyzed from multiple dimensions to generate a comprehensive evaluation report, including:

[0103] S51, Based on model evaluation metrics, determine the analytical dimensions for multi-dimensional analysis;

[0104] S52, based on the code function type, the model performance data is split according to the code function type to obtain categorized performance data;

[0105] S53, based on the analysis dimensions, scores the categorized performance data according to the preset performance scoring rules to obtain the categorized performance score;

[0106] S54, according to the preset performance scoring weights, perform weighted calculations on the performance scores of different categories to generate a comprehensive evaluation score;

[0107] S55 combines categorized performance scores and comprehensive evaluation scores, organizes content according to a preset report framework, and generates a comprehensive evaluation report.

[0108] Specifically, model evaluation metrics can cover dimensions such as functional correctness (e.g., code generation syntax correctness, test case coverage), efficiency (e.g., code generation speed, defect detection response time), robustness (e.g., ability to adapt to input noise), and engineering adaptability (e.g., CI / CD integration difficulty). Based on the core evaluation scope to which each model evaluation metric belongs, the analysis dimensions are determined to be consistent with the metric dimensions: functional correctness, efficiency, robustness, and engineering adaptability. For example, code generation syntax correctness and test case coverage metrics are mapped to the functional correctness analysis dimension, and code generation speed and defect detection response time metrics are mapped to the efficiency analysis dimension. The code function types include four categories: code generation, software testing, defect detection, and software maintenance. Subsets can be extracted from the structured model performance data based on the code function type using SQL queries or filtering functions in the Python Pandas library. For example, filtering out all data with "code function type = 'defect detection'" will generate defect detection sub-type performance data. This data contains the original evaluation results of various evaluation metrics (such as defect detection rate and false positive rate) in the defect detection scenario. This process can be repeated to obtain sub-type performance data for all code function types. The pre-defined performance scoring rules clearly define the correspondence between the raw data and scores for different indicators under each analysis dimension. For example, in the functional correctness dimension, a code generation syntax accuracy rate ≥90% earns 10 points, 80%-90% earns 8 points, 70%-80% earns 6 points, and <70% earns 4 points. In the efficiency dimension, a code generation speed ≤10 seconds / file earns 10 points, 10-15 seconds / file earns 8 points, 15-20 seconds / file earns 6 points, and >20 seconds / file earns 4 points. Based on the defined analysis dimensions, performance data for each sub-category can be read. Based on the original values ​​of the indicators, a matching score is assigned according to the preset performance scoring rules. For example, in the defect detection sub-category performance data, a defect detection rate of 85% corresponds to a score of 9 points under the functional correctness dimension, where a defect detection rate ≥ 80% earns 9 points. Therefore, the functional correctness dimension score for this indicator is 9 points. The scores of each indicator under different analysis dimensions are summarized to form a sub-category performance score (e.g., the software testing sub-category performance score includes 8 points for functional correctness, 7 points for efficiency, 6 points for robustness, and 8 points for engineering adaptability). The preset performance score weights are determined through domain expert evaluation. For example, the code functional type weight coefficients are 0.3 for code generation, 0.25 for software testing, 0.25 for defect detection, and 0.2 for software maintenance. The multi-dimensional analysis weights are 0.4 for functional correctness, 0.2 for efficiency, 0.2 for robustness, and 0.2 for engineering adaptability. A weighted calculation is then performed to generate a comprehensive evaluation score.The preset report framework can include five fixed modules: data overview, detailed performance scores by category, comprehensive evaluation score and interpretation, strengths and weaknesses analysis, and optimization suggestions. The performance scores by category are organized into a table (e.g., rows represent code function types, columns represent analysis dimensions, and cells represent corresponding scores) and entered into the detailed performance scores by category module. The comprehensive evaluation score (e.g., 58.74 out of 100) and a brief description of the calculation process are entered into the comprehensive evaluation score and interpretation module. Based on the performance scores by category, the strengths and weaknesses analysis module points out, for example, that the code generation function scores 9 points in the functional correctness dimension, showing outstanding performance; while the defect detection function scores 6 points in the robustness dimension, indicating a deficiency. The optimization suggestions module proposes supplementing training data to enhance the defect detection function's ability to adapt to input noise in the scenario. A comprehensive evaluation report containing data tables, score calculation explanations, and analysis conclusions is generated.

[0109] In one embodiment, S61, the performance scores of different categories are weighted according to preset performance scoring weights to generate a comprehensive evaluation score, which is achieved by the following formula:

[0110]

[0111] in, For the overall evaluation score, L represents the total number of code function types. The weight coefficient for the function of the l-th type of code. Let Q be the weight vector for the multidimensional analysis, and let Q be the number of dimensions in the multidimensional analysis. Let l be the categorical performance score vector for the l-th type of code functionality. The attribute feature vector of the l-th type of code function The transpose of is used to ensure dimensionality consistency in matrix operations, where 'a' is the dimension of the attribute feature vector. The standard deviation of the performance scores for all categories. Here is the calibration function, where .

[0112] For example, the total number L of code function types can be determined (e.g., four categories: code generation, software testing, defect detection, and software maintenance, with L taking a value of 4). The weight coefficient of the l-th type of code function can be determined by domain expert evaluation. (e.g., code generation corresponding) Software testing Defect detection corresponding Software maintenance Determine the weight vector for multidimensional analysis. (It is a 1×Q real vector, where Q is the number of the four analytical dimensions: functional correctness, efficiency, robustness, and engineering adaptability, i.e., Q=4, and the weights of each dimension are as follows) ), the categorical performance score vector of the l-th code function Let Q be a real vector of 1×Q, where each element represents the score of that type of function in the corresponding analysis dimension (e.g., software testing functions). = ), the attribute feature vector of the l-th type of code function After transposition, we get (A 1×Q real vector satisfies the dimensionality consistency requirement for matrix operations), all classification performance scores (i.e., each) can be calculated using statistical tools (such as Python's numpy library). The standard deviation of the rating data across all dimensions. ,Will Substitute into calibration function In China, according to The calibration function result g(σs) is obtained by calculating the formula; then the denominator is calculated, that is, for all l from 1 to L. Perform a summation operation; calculate the numerator, performing the operation sequentially for each l from 1 to L: and and Hadamaji (i.e.) and Multiply the corresponding elements by themselves, and then multiply the result by the product of the elements of the corresponding dimensions. Perform a vector dot product operation, sum the results for all l, and then combine the sum with... Multiply; divide the result of the calculation in the numerator by the result in the denominator. You can then obtain a comprehensive evaluation score. .

[0113] The aforementioned method for evaluating large-scale code models obtains a set of code attributes by analyzing the code attributes of the model's running code. Cluster analysis is then performed on this attribute set to generate clustering results, determining a model evaluation scenario suitable for actual software engineering needs. Based on this scenario, scientific evaluation metrics are generated according to preset scenario-metric mapping rules. These metrics are then used to evaluate the running code to obtain model performance data. Multi-dimensional analysis of the performance data is then performed to generate a comprehensive evaluation report. This method can cover the entire lifecycle of code generation, software testing, defect detection, and software maintenance, overcoming the shortcomings of existing evaluation scenarios. By using scenario-adapted multi-dimensional metrics, it addresses the lack of scientific rigor in existing metrics, avoiding a fragmented evaluation system, making the evaluation results more comprehensive and objective, improving credibility, and effectively guiding model optimization and engineering applications.

[0114] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0115] Based on the same inventive concept, this application also provides an apparatus for evaluating a large code model to implement the above-described method for evaluating a large code model. The solution provided by this apparatus is similar to the implementation described in the above-described method. Therefore, the specific limitations of one or more embodiments of the apparatus for evaluating a large code model provided below can be found in the limitations of the method for evaluating a large code model described above, and will not be repeated here.

[0116] In one exemplary embodiment, such as Figure 2 As shown, a large code model evaluation device is provided, including:

[0117] The code attribute analysis module 101 is used to analyze the code attributes of the running code of the model and obtain a code attribute set;

[0118] The attribute clustering analysis module 102 is used to perform clustering analysis on the code attribute set and generate attribute clustering results.

[0119] The evaluation scenario determination module 103 is used to determine the model evaluation scenario based on the attribute clustering results;

[0120] The evaluation index generation module 104 is used to generate model evaluation indexes based on the model evaluation scenario and according to the preset scenario-index mapping rules.

[0121] The model performance evaluation module 105 is used to evaluate the running code based on the model evaluation scenario and the model evaluation index to obtain model performance data.

[0122] The comprehensive evaluation report module 106 is used to perform multi-dimensional analysis of model performance data and generate a comprehensive evaluation report.

[0123] In one embodiment, the code attribute analysis module 101 is further configured to:

[0124] Perform functional identification on the running code to determine the code's functional type;

[0125] For each code functionality type, perform the following operations:

[0126] Extract surface layer attributes from the running code to obtain a subset of surface layer attributes;

[0127] The running code is parsed using an abstract syntax tree to obtain a subset of syntax attributes;

[0128] Perform semantic dependency analysis on the running code to obtain a subset of semantic attributes;

[0129] The surface layer attribute subset, syntactic attribute subset, and semantic attribute subset are concatenated into feature vectors to form attribute feature vectors;

[0130] Integrate the attribute feature vectors corresponding to all code function types to form a code attribute set.

[0131] In one embodiment, the attribute clustering analysis module 102 is further configured to:

[0132] Obtain the task requirements for the corresponding functional type of the code attribute set;

[0133] Construct a task-aware attention network based on task requirements;

[0134] Based on the code attribute set, a task-aware attention network is used to generate task feature weights;

[0135] The attribute feature vectors in the code attribute set are weighted using task feature weights to obtain enhanced attribute feature vectors;

[0136] Based on a preset clustering threshold, the enhanced attribute feature vectors are clustered to form attribute clustering results.

[0137] In one embodiment, the evaluation scenario determination module 103 is further configured to:

[0138] Within each cluster in the attribute clustering results, statistical analysis is performed on the enhanced attribute feature vectors to obtain cluster attributes and attribute distribution characteristics;

[0139] Based on cluster attributes, scene elements are determined according to preset attribute-element mapping rules;

[0140] Based on attribute distribution characteristics, the scene frame is determined according to the preset pattern-frame mapping rules;

[0141] Combine scene elements and scene framework to form a model evaluation scene.

[0142] In one embodiment, the model performance evaluation module 105 is further configured to:

[0143] Based on the model evaluation metrics, the corresponding evaluation program is matched from the preset evaluation program library;

[0144] Based on the model evaluation scenario, the test parameters of the evaluation program are determined, and the test parameters are configured into the corresponding evaluation program to obtain the evaluation program with complete parameter configuration.

[0145] Associate the running code with the evaluation program with configured parameters to form an indicator-program-code association group;

[0146] For each metric-program-code association group, an evaluation program with configured parameters is used to perform evaluation operations on the running code to obtain raw evaluation data;

[0147] Integrate the raw evaluation data of all related metrics, programs, and code groups to form model performance data.

[0148] In one embodiment, the comprehensive evaluation report module 106 is further configured to:

[0149] Based on the model evaluation metrics, determine the analytical dimensions for multi-dimensional analysis;

[0150] Based on the code function type, the model performance data is split according to the code function type to obtain categorized performance data;

[0151] Based on the analysis dimensions, the categorized performance data are scored according to the preset performance scoring rules to obtain the categorized performance score;

[0152] Based on the preset performance scoring weights, the performance scores of different categories are weighted and calculated to generate a comprehensive evaluation score.

[0153] By combining the performance scores by category and the overall evaluation score, and organizing the content according to the preset report framework, a comprehensive evaluation report is generated.

[0154] In one embodiment, the comprehensive evaluation report module 106 is further configured to generate a comprehensive evaluation score by performing a weighted calculation on the categorical performance scores according to preset performance scoring weights using the following formula:

[0155]

[0156] in, For the overall evaluation score, L represents the total number of code function types. The weight coefficient for the function of the l-th type of code. Let Q be the weight vector for the multidimensional analysis, and let Q be the number of dimensions in the multidimensional analysis. Let l be the categorical performance score vector for the l-th type of code functionality. , where 'a' is the dimension of the attribute feature vector. The standard deviation of the performance scores for all categories. Here is the calibration function, where .

[0157] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the evaluation method for a large code model as described above.

[0158] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0159] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0160] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for evaluating large code models, characterized in that, The method includes: Analyze the code attributes of the model's running code to obtain a set of code attributes; Perform cluster analysis on the code attribute set to generate attribute clustering results; Based on the attribute clustering results, determine the model evaluation scenario; Based on the model evaluation scenario, model evaluation metrics are generated according to the preset scenario-metric mapping rules. Based on the model evaluation scenario, the running code is evaluated using the model evaluation metrics to obtain model performance data; The model performance data is analyzed from multiple dimensions to generate a comprehensive evaluation report.

2. The method according to claim 1, characterized in that, The analysis model's running code attributes are used to obtain a set of code attributes, including: The code is functionally identified to determine its function type. For each code functionality type, perform the following operations: Surface layer attributes are extracted from the running code to obtain a subset of surface layer attributes; The running code is parsed using an abstract syntax tree to obtain a subset of syntax attributes; Perform semantic dependency analysis on the running code to obtain a subset of semantic attributes; The surface layer attribute subset, the syntactic attribute subset, and the semantic attribute subset are concatenated into feature vectors to form an attribute feature vector; The attribute feature vectors corresponding to all code function types are integrated to form the code attribute set.

3. The method according to claim 2, characterized in that, The step of performing cluster analysis on the code attribute set to generate attribute clustering results includes: Obtain the task requirements for the functional type corresponding to the code attribute set; Based on the task requirements, a task-aware attention network is constructed; Based on the code attribute set, the task-aware attention network is used to generate task feature weights; The attribute feature vectors in the code attribute set are weighted using the task feature weights to obtain enhanced attribute feature vectors; Based on a preset clustering threshold, the enhanced attribute feature vectors are clustered to form the attribute clustering results.

4. The method according to claim 3, characterized in that, The step of determining the model evaluation scenario based on the attribute clustering results includes: Within each cluster of the attribute clustering results, statistical analysis is performed on the enhanced attribute feature vector to obtain the cluster attributes and attribute distribution characteristics. Based on the cluster attributes, scene elements are determined according to preset attribute-element mapping rules; Based on the attribute distribution characteristics, the scene frame is determined according to the preset pattern-frame mapping rules; The scene elements and the scene framework are combined to form the model evaluation scene.

5. The method according to claim 1, characterized in that, The process of evaluating the running code based on the model evaluation scenario and using the model evaluation metrics to obtain model performance data includes: Based on the model evaluation metrics, a corresponding evaluation program is matched from a preset evaluation program library; Based on the model evaluation scenario, the test parameters of the evaluation program are determined, and the test parameters are configured into the corresponding evaluation program to obtain the evaluation program with complete parameter configuration. The running code is associated with the evaluation program with the configured parameters to form an indicator-program-code association group; For each of the aforementioned indicator-program-code association groups, the evaluation program configured with the aforementioned parameters is used to perform evaluation operations on the running code to obtain raw evaluation data; The original evaluation data of all the aforementioned indicator-program-code association groups are integrated to form the model performance data.

6. The method according to claim 2, characterized in that, The process of performing multi-dimensional analysis on the model performance data and generating a comprehensive evaluation report includes: Based on the model evaluation indicators, the analytical dimensions of the multi-dimensional analysis are determined; Based on the code function type, the model performance data is split according to the code function type to obtain categorized performance data; Based on the analysis dimensions, the categorized performance data are scored according to preset performance scoring rules to obtain categorized performance scores; Based on preset performance scoring weights, the categorized performance scores are weighted and calculated to generate a comprehensive evaluation score. The comprehensive evaluation report is generated by combining the categorized performance scores and the overall evaluation score, and organizing the content according to a preset report framework.

7. The method according to claim 6, characterized in that, The comprehensive evaluation score is generated by weighting the categorized performance scores according to preset performance scoring weights, and is achieved through the following formula: in, For the overall evaluation score, L represents the total number of code function types. The weight coefficient for the function of the l-th type of code is... Let Q be the weight vector for the multidimensional analysis, and let Q be the number of dimensions in the multidimensional analysis. Let l be the categorical performance score vector for the l-th type of code functionality. , where 'a' is the dimension of the attribute feature vector. The standard deviation of the performance scores for all categories. Here is the calibration function, where .

8. A device for evaluating large code models, characterized in that, The device includes: The code attribute analysis module is used to analyze the code attributes of the model's running code and obtain a set of code attributes; The attribute clustering analysis module is used to perform clustering analysis on the code attribute set and generate attribute clustering results; The evaluation scenario determination module is used to determine the model evaluation scenario based on the attribute clustering results. The evaluation index generation module is used to generate model evaluation indexes based on the model evaluation scenario and according to the preset scenario-index mapping rules. The model performance evaluation module is used to evaluate the running code based on the model evaluation scenario and the model evaluation metrics to obtain model performance data. The comprehensive evaluation report module is used to perform multi-dimensional analysis on the model performance data and generate a comprehensive evaluation report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images