Large language model evaluation method and device in data science field and storage medium

Abstract

The invention relates to the technical field of data science, in particular to a big language model evaluation method, device and equipment in the field of data science and a computer storage medium. According to the large language model evaluation method, from the perspective of coarse granularity, firstly, task types, functions and the range of corresponding codes are summarized, then from the perspective of fine granularity, programming rules are defined for output of each function according to specific tasks, results are compared with standard answers, and finally, the large language model is evaluated. Evaluating the target large language model through the generated test case set; according to the method, the standard answer is generated for each sub-task from a comprehensive perspective, and the specific evaluation index is defined, so that the method can be used for large language model evaluation of a more complex real world task consisting of a plurality of sub-tasks.

Description

Technical Field

[0001] The present invention relates to the field of data science technology, and in particular to a large language model evaluation method, device, equipment and computer storage medium in the field of data science. Background Art

[0002] Large Language Models (LLMs) are increasingly used in data science and scientific fields, e.g., data analysis, protein generation, and scientific reasoning. For data science tasks, LLMs offer the potential to (semi-)automate data analysis and data visualization for given public known problems by calling a code interpreter with the corresponding Python library. These works are benchmarked on relatively simple tasks for which the true labels can be obtained accurately.

[0003] However, many real-world data analyses require reasoning about more complex scenarios, for example, to evaluate the quality of images generated by data visualization tasks. Proper evaluation of these more complex data science tasks remains an open research direction. Although there are some existing benchmarks for evaluating the challenges associated with LLM, these benchmarks generally focus on evaluating narrower tasks with easily available standard answers and evaluation metrics. For example, MLAgentBench provides a machine learning research benchmark by building an LLM agent pipeline. SWEBench benchmarks the ability of LLM to solve real-world software issues from GitHub. InfiAgent–DABench generates labels and calculates accuracy through GPT-4 to complete data analysis tasks. The forefront of LLM evaluation is towards more complex real-world tasks consisting of multiple subtasks. For these challenging problems, how to generate standard answers for each subtask from a comprehensive perspective and define specific evaluation metrics is a problem worth exploring. Summary of the invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the problem that the large language model evaluation benchmark in the prior art has poor applicability to complex real-world tasks.

[0005] In order to solve the above technical problems, the present invention provides a large language model evaluation method, comprising:

[0006] Define typical data science task types and collect questions that contain at least one typical data science task type to create a question set;

[0007] Generate and verify an answer for each question in the question set using a large language model to obtain a verified standard answer;

[0008] Define corresponding evaluation functions for different typical data science task types, aggregate the evaluation functions of each typical data science task type by function category, and obtain the aggregated evaluation function of each typical data science task type;

[0009] Generate a corresponding evaluation code for each question according to the aggregate evaluation function, and generate a test case set for large language model evaluation in combination with the standard answer corresponding to each question;

[0010] The target large language model is evaluated based on the test case set.

[0011] Preferably, the typical data science task types include, but are not limited to: data cleaning and preprocessing tasks, data exploration and statistical tasks, data visualization tasks, predictive modeling tasks, data mining and pattern recognition tasks, and interpretability and report generation tasks.

[0012] Preferably, the collection includes at least one question of a typical data science task type, and establishing the question set includes:

[0013] Collect the problems from preset channels, including but not limited to: data science task problems in actual programming problems, data science task problems in code bases, data science task problems generated by large language models, and data science task problems designed by domain experts;

[0014] The problems are filtered using preset filtering rules, including but not limited to: keyword screening filtering, exclusion filtering requiring code update, error finding or concept explanation, and exclusion filtering requiring writing consistent with human preferences and large language models;

[0015] Review the correctness, applicability and ease of assessment of the filtered questions;

[0016] Format the reviewed questions and ensure data availability.

[0017] Preferably, generating and verifying an answer for each question in the question set by using a large language model to obtain a verified standard answer includes:

[0018] For each question, the large language model is sampled multiple times to generate multiple answer code snippets and corresponding answer outputs;

[0019] Verify the answer output through preset test cases, or,

[0020] A self-consistency strategy is used to cross-validate all generated answer code snippets;

[0021] The verified answer is determined as the standard answer.

[0022] Preferably, the defining corresponding evaluation functions for different typical data science task types includes:

[0023] Define corresponding evaluation functions for different typical data science task types;

[0024] Generate a corresponding evaluation code for each problem according to the evaluation function;

[0025] The typical data science task type, the evaluation function and the corresponding evaluation code of each problem are encapsulated into a triple;

[0026] creating a directed acyclic graph in a hierarchy for each question based on the triples;

[0027] The evaluation function is executed using a large language model to obtain expected results for each typical data science task type as a benchmark for evaluating the large language model.

[0028] Preferably, the function category aggregation of the evaluation function of each typical data science task type to obtain the aggregated evaluation function of each typical data science task type includes:

[0029] For each typical data science task type, select K optimal evaluation functions;

[0030] Based on the K optimal evaluation functions, all evaluation functions of each typical data science task type are clustered into K clustered evaluation functions.

[0031] Preferably, generating a corresponding evaluation code for each question according to the aggregate evaluation function, and generating a test case set for large language model evaluation in combination with a standard answer corresponding to each question includes:

[0032] Generate a corresponding evaluation code for each question according to the aggregate evaluation function;

[0033] Determine the evaluation result by judging the consistency between the predicted output of the evaluation code corresponding to the aggregate evaluation function and the standard answer, and unify the evaluation result into a Boolean type or a decimal type between 0 and 1;

[0034] The decimal evaluation result is converted into a Boolean value by setting a corresponding threshold.

[0035] The present invention also provides a large language model evaluation device, comprising:

[0036] A data acquisition module is used to define typical data science task types and collect questions containing at least one typical data science task type to establish a question set;

[0037] A data processing module, used to generate and verify an answer for each question in the question set through a large language model to obtain a verified standard answer;

[0038] An evaluation function building module is used to define corresponding evaluation functions for different typical data science task types, aggregate the evaluation functions of each typical data science task type by function category, and obtain the aggregated evaluation function of each typical data science task type;

[0039] A test case set building module, used to generate a corresponding evaluation code for each question according to the aggregate evaluation function, and generate a test case set for large language model evaluation in combination with the standard answer corresponding to each question;

[0040] A target evaluation module is used to evaluate the target large language model based on the test case set.

[0041] The present invention also provides a large language model evaluation device, comprising:

[0042] Memory for storing computer programs;

[0043] A processor is used to implement the steps of the above-mentioned large language model evaluation method when executing the computer program.

[0044] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned large language model evaluation method are implemented.

[0045] The above technical solution of the present invention has the following advantages compared with the prior art:

[0046] The large language model evaluation method described in the present invention, from a coarse-grained perspective, first summarizes the scope of task types, functions and corresponding codes, then, from a fine-grained perspective, defines programming rules for the output of each function according to specific tasks, and compares the results with standard answers, and finally, evaluates the target large language model through the generated test case set; the present invention generates standard answers for each subtask from a comprehensive perspective and defines specific evaluation indicators, thereby enabling large language model evaluation for more complex real-world tasks composed of multiple subtasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0048] Figure 1 It is a flow chart of the implementation of a large language model evaluation method provided by the present invention;

[0049] Figure 2 It is a framework diagram of DataSciBench, a large language model evaluation benchmark provided by the present invention;

[0050] Figure 3 It is a schematic diagram of all the results and average results of all the evaluation models;

[0051] Figure 4 This is a diagram showing the average completion rate of all the evaluation models on problems of different difficulty levels;

[0052] Figure 5 Schematic diagram of Pass@1 comparison of all test LLMs between DataSciBench and HumanEval. DETAILED DESCRIPTION

[0053] The core of the present invention is to provide a large language model evaluation method, device, equipment and computer storage medium in the field of data science, which effectively provides a large language model evaluation benchmark suitable for complex real-world tasks.

[0054] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0055] Please refer to Figure 1 , Figure 1 This is a flow chart of the implementation of a large language model evaluation method provided by the present invention; the specific operation steps are as follows:

[0056] S101: define typical data science task types, collect questions that contain at least one typical data science task type, and establish a question set;

[0057] S102: Generate an answer for each question in the question set by using a large language model and verify it to obtain a verified standard answer;

[0058] S103: defining corresponding evaluation functions for different typical data science task types, and performing function category aggregation on the evaluation functions of each typical data science task type to obtain an aggregated evaluation function of each typical data science task type;

[0059] S104: Generate a corresponding evaluation code for each question according to the aggregate evaluation function, and generate a test case set for large language model evaluation in combination with the standard answer corresponding to each question;

[0060] S105: Evaluate the target large language model based on the test case set.

[0061] Please refer to Figure 2 , Figure 2 A framework diagram of DataSciBench, a large language model evaluation benchmark provided by the present invention;

[0062] DataSciBench consists of three important components:

[0063] 1. Problem definition and collection, where 6 task types are defined and a set of 222 real, challenging, and high-quality problems are collected through problem filtering and expert review.

[0064] 2. Response integration and verification, we propose a novel Task-Function-Code (TFC) to evaluate the key tasks of each question, and define aggregation functions and programming rules to effectively evaluate specific task descriptions and compare the predictions with the standard answers.

[0065] 3. LLM Evaluation,Evaluate 6 API-based models, 8 open source general models, and 9 open source code generation models from coarse-grained (e.g., success rate, completion rate) and fine-grained (e.g., VLM-as-a-judge, aggregation function) perspectives.

[0066] This paper introduces a new benchmark, called DataSciBench, to evaluate the data science capabilities of LLMs and help LLMs improve their data analysis and data visualization capabilities. For the collected questions and their corresponding responses and evaluation indicators, this paper hopes to meet the following characteristics: (1) More natural, challenging, and high-quality questions are needed to promote the development of LLMs. (2) Each task requires multi-hop reasoning, and sequential tasks require strong correlations to distinguish models well. (3) Multiple types of results are required for comprehensive evaluation.

[0067] Based on the above embodiments, this embodiment describes step S101 in detail:

[0068] Typical data science task types include, but are not limited to: data cleaning and preprocessing tasks, data exploration and statistics tasks, data visualization tasks, predictive modeling tasks, data mining and pattern recognition tasks, and interpretability and report generation tasks, among which:

[0069] 1. Data cleaning and preprocessing tasks. This task detects and handles missing values, outliers, and duplicate data; as well as standardizing data formats, such as the unified format of date and time.

[0070] 2. Data exploration and statistical tasks. This task is to calculate the basic statistical indicators of the data (mean, median, standard deviation, etc.), generate data distribution diagrams (histograms, box plots, etc.), calculate the correlation between variables, and draw correlation matrices or correlation diagrams.

[0071] 3. Data visualization task. The goal of this task is to visualize and analyze data and create interactive charts so that users can freely explore the data.

[0072] 4. Predictive modeling task. This task aims to select appropriate machine learning algorithms, such as linear regression, decision tree, random forest, etc.; perform feature engineering, such as feature selection, feature transformation, feature combination, etc.; divide the data set into training set and test set, train and evaluate the model; and select corresponding evaluation indicators for different prediction problems such as classification, regression or clustering.

[0073] 5. Data mining and pattern recognition tasks. This task uses methods such as association rule mining and frequent item set mining to find interesting patterns in the data; text mining technology is used to extract keywords, topics and other information from text data; and cluster analysis, classification algorithms, etc. are applied to identify the underlying patterns and structures. Pattern recognition tasks can complete the following functions: image recognition, text clustering, and time series detection.

[0074] 6. Interpretability and report generation task. This task aims to provide explanations of model results, such as feature importance, model parameters, etc., and automatically generate reports and summaries to present the analysis results in a way that is easy to understand and share.

[0075] In order to increase the difficulty of the evaluation problem, this embodiment selects more complex problems containing multiple tasks. These sequential tasks can be any combination of tasks.

[0076] The collection contains at least one question of a typical data science task type, and establishing a question set includes:

[0077] The problems are collected from preset channels, including but not limited to: data science task problems in actual programming problems, data science task problems in code bases, data science task problems generated by large language models, and data science task problems designed by domain experts, wherein, in a specific embodiment:

[0078] Collect users’ real problems from an online code generation platform CodeGeeX to achieve coarse-grained collection from real-world online platforms.

[0079] Select high-quality data science-related problems from BigCodeBench and rewrite them to meet the requirements of this invention.

[0080] Use a few-shot problem approach to ask the LLM to generate similar problems.

[0081] Detailed questions were written with reference to relevant websites to increase the difficulty and robustness of the evaluation benchmark.

[0082] The problems are filtered using preset filtering rules, including but not limited to: keyword screening filtering (such as task name, deep learning and machine learning), exclusion filtering that requires code update, error finding or concept explanation, and exclusion filtering that requires writing consistency with human preferences and large language models; namely:

[0083] Low-quality questions are filtered by the following three principles: (1) Select questions containing keywords, such as task names, deep learning, and machine learning. (2) Filter questions that require code updates, bug finding, and concept explanations. (3) Filter questions that need to be rewritten to be consistent with human preferences and LLM.

[0084] Review the correctness, applicability and ease of assessment of the filtered questions, format the questions that pass the review, and ensure the availability of data; namely:

[0085] To ensure the quality of the collected questions, the present invention is reviewed by experts in computer science and data analysis. The review process includes three stages: (1) In the first stage, experts verify the correctness of the questions and adjust the applicability of the questions. In addition, experts ensure that the answers to the questions are easy to evaluate. (2) In the second stage, experts format the input, output, and instruction questions of the questions, which makes LLM follow unified instructions. (3) In the third stage, experts ensure the availability of the input question dataset, including generating random datasets and collecting public datasets.

[0086] Based on the above embodiments, this embodiment describes step S102 in detail:

[0087] For each question, the large language model is sampled multiple times to generate multiple answer code snippets and corresponding answer outputs; for each collected question, the LLM is required to generate multiple independent responses, which may include code snippets, data analysis results, or other forms of answers, and execute the code generated by the LLM in a safe sandbox environment to monitor its running status and output results.

[0088] The present invention increases the diversity and reliability of results through multiple sampling, reduces the deviation caused by randomness, and verifies whether the code snippets generated by LLM can be executed correctly to produce expected outputs.

[0089] In order to ensure the rationality and reliability of the answers generated by LLM, the present invention uses two different methods for verification:

[0090] Verify the answer output through preset test cases; For questions from BigCodeBench, which provide reliable test cases, the generated answers are verified by executing all test cases, and then experts review the answers that pass the test cases to confirm whether they meet the expected solutions.

[0091] A self-consistency strategy is used to cross-validate all generated answer code snippets; For other questions without test cases, a self-consistency strategy is used to cross-validate all generated codes, that is, multiple answers generated for the same question are compared to check whether they are consistent and whether they can all reasonably solve the given problem.

[0092] The verified answer is determined as the standard answer.

[0093] Through the above strategies, we can ensure that the answers generated by LLM are not only technically feasible, but also reasonable and reliable in practical applications. Multiple sampling and cross-validation help reduce the bias of a single output and improve the stability of the evaluation results. Through clear verification steps, the transparency of the evaluation process is improved, making the evaluation results easier to understand and accept.

[0094] In order to generate standard answers and obtain evaluation indicators for each subtask, the present invention proposes a semi-automatic framework called Task-Function-Code (TFC) generation and evaluation. Based on the above embodiment, this embodiment describes step S103 in detail:

[0095] Corresponding evaluation functions are defined for different typical data science task types; in one embodiment:

[0096] Identify all relevant data science task types, such as data cleaning, data exploration, predictive modeling, etc., and design and predefine evaluation functions for each task type that can quantify the completion and quality of the task;

[0097] Leveraging the capabilities of GPT-4o-mini, it selects task types based on predefined criteria (such as complexity, commonality, importance, etc. of the task) and returns the corresponding evaluation functions of these selected task types.

[0098] Generate corresponding evaluation codes for each problem according to the evaluation function; that is, generate evaluation codes for each task problem according to the selected task type and the predefined evaluation function, and these codes will be used to actually perform the evaluation and generate evaluation results.

[0099] The typical data science task type, the evaluation function and the corresponding evaluation code for each problem are encapsulated into a triple in:

[0100] T (Task Type): task type;

[0101] F (Evaluation Function): evaluation function;

[0102] C (Evaluation Code): Evaluation code;

[0103] N: the number of valuable task types in each problem;

[0104] For each task problem, determine its corresponding task type, select the corresponding evaluation function, generate the evaluation code, and then encapsulate this information into a tuple.

[0105] A directed acyclic graph (DAG) in a hierarchical structure is created for each problem based on the triples; a data interpreter (DI) is used to analyze the task problem, identify task types and their dependencies, and generate a DAG for each task problem, wherein each task type is defined as a node in the DAG.

[0106] The evaluation functions are executed using the large language model to obtain the expected results for each typical data science task type as a benchmark for evaluating the large language model. Based on the generated DAG, all evaluation functions are run with a strong LLM as the backbone. The expected results (ground truth) for each task type are obtained, which will serve as a benchmark for evaluating the LLM output.

[0107] Based on the above embodiments, this embodiment further describes step S103 in detail:

[0108] In order to unify key functions and improve the scalability of evaluation, this paper selects top-K functions for each task type and aggregates all generated functions into top-K function categories:

[0109] For each typical data science task type, select K optimal evaluation functions;

[0110] Based on the K optimal evaluation functions, all evaluation functions of each typical data science task type are clustered into K clustered evaluation functions.

[0111] In a specific embodiment, typical data science task types may be specifically divided into the following categories:

[0112] Data cleaning and preprocessing

[0113] Data cleaning integrity: Ensure that the records in the data set remain complete after cleaning without unnecessary loss.

[0114] DataFrame Shape Validation: Verify that the dimensions of a DataFrame are as expected, such as the number of rows and columns.

[0115] Data integrity: Check whether missing values ​​in the data are handled properly to ensure data integrity.

[0116] Normalization range check: Verify that the data is within the expected range after normalization or standardization, such as 0 to 1 or -1 to 1.

[0117] Data quality scoring: Scoring the overall quality of the data, which may include aspects such as consistency, accuracy, and reliability.

[0118] Data exploration and statistical analysis

[0119] Descriptive statistics: calculate the central tendency (mean, median), dispersion (variance, standard deviation), etc. of the data.

[0120] Correlation analysis: Evaluate the correlation between variables, such as Pearson's correlation coefficient or Spearman's rank correlation coefficient.

[0121] Distribution analysis: Analyze the distribution characteristics of data, such as normality test.

[0122] Outlier Detection: Identifying and handling outliers in data.

[0123] Statistical visualization: Generate statistical charts, such as histograms, box plots, etc., to intuitively display data characteristics.

[0124] Data visualization

[0125] Interactive charts: Create charts that users can interact with, such as a zoomable scatter plot or a filterable bar chart.

[0126] Infographic Design: Design intuitive infographics to communicate key data points.

[0127] Multidimensional data visualization: Use heat maps, parallel coordinates plots and other techniques to display multidimensional data.

[0128] Geospatial Data Visualization: Visualize geographic data, such as maps and heat maps.

[0129] Dynamic data visualization: Create dynamic charts to show how data changes over time.

[0130] Predictive modeling

[0131] Model selection: Choose an appropriate prediction model, such as linear regression, decision tree, random forest, etc.

[0132] Feature engineering: perform feature selection, feature transformation, and feature combination.

[0133] Model training and evaluation: Train the model and evaluate its performance using techniques such as cross-validation.

[0134] Hyperparameter optimization: Adjust the model's hyperparameters to optimize performance.

[0135] Prediction accuracy: Evaluate the accuracy of the model’s predictions, such as mean squared error (MSE) or accuracy rate.

[0136] Data mining and pattern recognition

[0137] Association rule mining: discovering frequent itemsets and association rules in data.

[0138] Cluster analysis: Clustering data to identify natural groupings in the data.

[0139] Classification and classifier learning: Train classifiers to categorize data.

[0140] Anomaly detection: Identifying unusual patterns or outliers in data.

[0141] Trend and Seasonality Analysis: Identify trends and seasonal patterns in time series data.

[0142] Interpretability and report generation

[0143] Model interpretability: Provide explanations of model predictions, such as feature importance analysis.

[0144] Results Summary: Generates a summary of key results for quick understanding.

[0145] Report Automation: Automatically generate detailed analysis reports.

[0146] Data Storytelling: Tell the story behind your data through data visualization and narrative.

[0147] Decision Support: Providing analysis and recommendations to support decision making.

[0148] Based on the above embodiments, this embodiment describes step S104 in detail:

[0149] Generate a corresponding evaluation code for each question according to the aggregate evaluation function;

[0150] Determine the evaluation result by judging the consistency between the predicted output of the evaluation code corresponding to the aggregate evaluation function and the standard answer, and unify the evaluation result into a Boolean type or a decimal type between 0 and 1;

[0151] For the decimal type evaluation results, appropriate thresholds are set for different aggregation functions according to the specific requirements and expected results of the task and converted into Boolean values; the selection of thresholds should be based on domain knowledge and experience to ensure the rationality and accuracy of the evaluation results.

[0152] In a specific embodiment, all outputs are unified into Boolean or decimal values ​​between 0 and 1, where the Boolean value indicates whether the task is successfully completed (1 for success, 0 for failure), and the decimal value provides a more detailed performance measurement, such as accuracy, similarity, etc.

[0153] The following are two specific examples to illustrate the above method:

[0154] Data cleansing integrity

[0155] Task Description: Verify the integrity of data after data preprocessing to ensure that no unnecessary data is lost.

[0156] Verification method:

[0157] Calculate the final number of rows or columns after preprocessing.

[0158] Compare the calculated result with the standard answer (the expected number of rows or columns).

[0159] If the two are the same, the output is 1 (success); if they are different, the output is 0 (failure).

[0160] Data Mining and Pattern Recognition

[0161] Task Description: Evaluate the performance of data mining tasks, such as the silhouette coefficient of cluster analysis.

[0162] Verification method:

[0163] Calculate the silhouette coefficient, which is a measure of clustering effectiveness, with values ​​typically ranging from -1 to 1.

[0164] Set a threshold (for example, 0.5) to convert the value of the silhouette coefficient to a Boolean value:

[0165] If the silhouette coefficient is greater than or equal to 0.5, the output is 1 (indicating a good clustering result).

[0166] If the silhouette coefficient is less than 0.5, the output is 0 (indicating poor clustering).

[0167] Suppose there is a data cleaning task where 100 rows of data are expected after data preprocessing. The result obtained after executing the code generated by LLM is 98 rows. According to the defined rules:

[0168] Calculate the difference in the number of rows: |100-98|=2

[0169] Since the number of rows does not match, the final output is 0 (failure).

[0170] For a clustering task, the silhouette coefficient obtained after executing the code generated by LLM is 0.55. According to the defined rules:

[0171] Since the silhouette coefficient is greater than the threshold of 0.5, the final output is 1 (success).

[0172] Through these unified rules and threshold settings, the DataSciBench of the present invention can provide a clear and quantifiable evaluation standard for the performance of LLM in data science tasks. This approach not only improves the consistency and comparability of the evaluation, but also simplifies the evaluation process, making the results easier to understand and operate.

[0173] Based on the above embodiments, this embodiment describes step S105 in detail:

[0174] In a specific application, we obtained 222 valid questions and corresponding test cases, which helped to evaluate the following API-based and open source models:

[0175] To evaluate the performance of different models (e.g., API-based models and open source general / code-generated models), this embodiment builds a comprehensive benchmark on the collected problems.

[0176] (1) Experimental setup

[0177] Six API-based models include o1-mini / GPT-4o-mini / GPT-4o-2024-05-13 / GPT-4-Turbo, Claude-3.5-Sonnet, and GLM-4-Flash.

[0178] 8 open source general models, including Llama3.1-8B-Instruct, Llama3-8B-Instruct, Qwen2.5-7B-Instruct, Qwen2-1.5 / 7B-Instruct, Gemma2-9B-it, GLM-4-9B-Chat, Yi-1.5-9B-Chat-16k.

[0179] ·9 open source code generation models including Deepseek-Coder-1.3 / 6.7 / 33B-Instruct, Codellama-7 / 13 / 34B-Instruct, Qwen2.5-Coder-1.5 / 7B-Instruct, StarCoder2-15B.

[0180] (2) Evaluation indicators

[0181] Coarse-grained metrics. The present invention defines coarse-grained metrics (CR and SR) for evaluating LLM.

[0182] Completion Rate (CR) Based on Data Interpreter, the present invention calculates the completion rate of a given TFC. For each TFC in the TFC list, the present invention gives it a completion score, with the lowest score being 0 and the highest score being 2. The given step completion scores are as follows: skip (score of 0), failure (score of 0), success-non-compliant (1 point), and success-compliant (2 points). The final completion rate is then calculated as follows:

[0183]

[0184] The numerator is the sum of the scores completed for each step, while the denominator is the sum of the maximum possible scores for all steps, which is 2×T, where T is the number of TFCs.

[0185] Success rate (SR). Similar to Codex, the success rate of the present invention is defined as the estimated complete success rate of a single problem under 10 runs. Specifically, if all TFCs pass in a problem run, it will be considered a success. Otherwise, it will be considered a failure. Note that for problems obtained from BigCodeBench, the present invention compares the output of the completion function with the output of the ground truth completion function to determine whether a single run passes or not, because the TFC is derived based on the desired function output in this case.

[0186] Fine-grained aggregation indicators. The present invention also defines fine-grained aggregation indicators for evaluating all LLMs in detail.

[0187] Visual Language Model (VLM)-as-a-judge evaluates the overall score of two inputs based on predefined criteria, providing a step-by-step rationale for its evaluation.

[0188] Data quality score (F1) in data cleaning and preprocessing, which aims to evaluate the cleanliness of the data after preprocessing. If it matches the standard answer, it produces a Boolean output of 1, otherwise 0.

[0189] Plot Validity (F2) in data visualization is related to the accuracy of the visual representation, for example, checking whether the shape of the correlation matrix is ​​consistent with the standard answer. If it is consistent, the final value is 1, otherwise it is 0.

[0190] Data accuracy (F3) in Data Exploration and Statistical Understanding focuses on understanding the quality of the data, which can be quantified using the Mean Squared Error (MSE). The final value is derived by comparing it to the true value with a predefined threshold.

[0191] Visualization completeness in data visualization (F4) evaluates the comprehensiveness of generated images (e.g., PNG, jpeg, PDF) by checking their existence against ground truth answers. If the file exists, it is scored 1, otherwise 0.

[0192] Model accuracy (F5) in predictive modeling is used to measure the predictive performance of the model and provides a Boolean accuracy value or a decimal between 0 and 1.

[0193] (3) Overall experimental results

[0194] The present invention is in Table 1 and Figure 3 The overall experimental results are shown in:

[0195] Table 1 Overall experimental results

[0196]

[0197] (1) In terms of average performance, API-based models significantly outperform open-source models. Among API-based models, GPT-4o has the highest total score of 64.43%, which is 9.84% higher than GPT-4-Turbo with a total score of 54.59%. It is worth noting that GPT-4o also surpasses all other models in all indicators except VLM, indicating its comprehensive capabilities in all aspects. (2) For open-source models, the performance gap between general models and code generation models is not significant. Among them, Deepseek-Coder-33B-Instruct has the highest score of 56.74%, even surpassing various closed-source models such as o1-mini and GPT-4-Turbo. Other models such as Qwen2.5-Coder-7B-Instruct and Qwen2.5-7b-Instruct also showed quite good capabilities, with total scores of 47.67% and 45.99% respectively. In contrast, there are also some models that only passed a few tasks, and their total scores are even lower than 5.0%. Among them, CodeLlama-34B-Instruct unexpectedly obtained a score of 1.33%, even lagging behind its smaller version CodeLlama-7B-Instruct. The present invention analyzes this anomaly. (3) In addition, the present invention Figure 3The total and average scores of the rankings are shown in Figure 2. It can be seen that the average performance of the API model is basically better than that of the open source model, and the average score of the open source acquisition model is 49.15% to 23.01%. In contrast, the performance difference between the API-based models is smaller than that of the open source models.

[0198] (4) Experimental results of problems of different difficulty

[0199] In order to evaluate the ability of multiple LLMs to complete problems of different difficulty levels, the present invention classifies tasks using BCB and CSV format data, human handwriting problems, and deep learning tasks related to data science into easy, medium, and difficult levels. The present invention evaluates multiple LLMs by combining different difficulty levels, the overall average CR, and the average CR of each difficulty level. Figure 4 It can be observed that: (1) Consistency across difficulty: Some LLMs, such as GPT-4o, GPT-4o-mini, GPT-4-Turbo, and Deepseek-Coder-33B-Instruct, show consistent performance across all difficulty levels, indicating robustness. (2) Models with the best performance across difficulty levels: Models such as the GPT-4 series and the Deepseek-Coder-Instruct series are among the best performing models, achieving high average CRs, especially performing well in complex, data-driven tasks. (3) Performance variance: There is a significant difference in the average CR between general models and small-scale models, and some models have lower overall scores, indicating that general models are less efficient or accurate in data science tasks.

[0200] (5) Comparison with other benchmarks

[0201] and Figure 5 The present invention compares the proposed DataSciBench with HumanEval. Figure 5 As shown, the present invention observes that most LLMs are located in the upper triangular area of ​​the graph, and all the tested models are divided into two groups, where the circle marks represent API-based models, and the other marks represent various open source LLMs. The LLMs in the dotted area perform well on the two benchmarks, and the performance of the solid area on the two datasets is relatively mismatched.

[0202] Correlation analysis with other benchmarks in Table 2:

[0203] Table 2 Correlation analysis

[0204]

[0205] The present invention performs correlation analysis to evaluate the consistency between the benchmark of the present invention and the coding evaluation benchmark (such as BigCodeBench and LiveCodeBench). To achieve this, the present invention calculates Pearson's r and Spearman's p correlation coefficients, which provide insights into the strength and direction of the relationship between the benchmark and these established indicators. This analysis not only verifies the results of the present invention, but also ensures robustness in different evaluation dimensions. The research results of the present invention show a strong positive correlation, indicating that the benchmark of the present invention is well consistent with these established coding evaluation indicators.

[0206] (6) Research or engineering observation

[0207] Through carefully curated metrics, this paper provides a deeper understanding of LLM's ability to plan and perform complex data science tasks. The experimental results also raise questions worth exploring because some results do not conform to traditional ideas.

[0208] Models are good at reasoning, but they don't necessarily perform better on complex data science tasks. While data science coding tasks do often involve scheduling and step-by-step execution similar to reasoning scenarios, the results show that even LLMs that are proficient in reasoning tasks still cannot complete complex data science tasks. For example, OpenAI's o1-mini model, which is generally considered one of the best reasoning models, unexpectedly failed in many DataSciBench tasks. The model's overall success rate was only 29.77%, significantly lagging behind the company's previously launched models such as GPT-4o and GPT-4-Turbo. After examining the outputs generated by o1-mini, the present invention found that the failure was mainly caused by non-compliance with instructions, incorrect calls, and forgetfulness. Although the task was successfully split into multiple subtasks, the model often forgot to export the required execution results, or only outputted unnecessary data. In other cases, the model may mistakenly call library functions or methods that sometimes do not even exist. These facts remind the present invention that real-life data science coding tasks often comprehensively challenge the model's ability to follow fine-grained instructions, utilize existing tools (libraries, APIs, ...), and plan. In order to perform and coordinate these tasks well, the model must be competitive in all relevant aspects.

[0209] Large models may sometimes fail to follow simple instructions more often. StarCoder2-15B performs worse than some smaller models, and Codellam-34B-Instruct performs even worse than its 13B and 7B versions. The main reason is that the large version lacks some other features, such as generating formatted text according to the question. Maybe a lot of data in a certain format was used to train a larger version, and this version did not generate another different format according to the question. In fact, in LiveCodeBench, CodeLlama's large-scale model also failed to outperform the small-scale version.

[0210] In summary, the present invention provides DataSciBench, a comprehensive benchmark to evaluate the capabilities of large language models (LLMs) in data science. Recent related benchmarks focus on single tasks, easily available standard answers, and direct evaluation metrics, which limits the scope of tasks that can be evaluated. In contrast, DataSciBench is built on a more comprehensive and natural and challenging set of problems. The present invention develops a semi-automatic pipeline for generating ground truth (GT) and verifying evaluation metrics. The pipeline exploits and implements a self-consistency strategy based on LLM to generate accurate GT by leveraging collected questions, predefined task types, and aggregated metrics. In addition, it adopts a comprehensive approach to filter high-quality task-function-code (TFC) lists and evaluate each code execution result in the TFC according to precisely defined metrics and programming rules. The experimental framework of the present invention includes testing 6 API-based models, 8 open source general models, and 9 open source code generation models, using a variety of problem sets collected by the present invention. With this approach, the present invention aims to provide a more comprehensive and rigorous evaluation of LLMs in the field of data science, revealing their strengths and weaknesses. Experimental results show that in addition to VLM-as-a-judge

[0211] In addition, the API-based model significantly outperforms the open source model in all metrics.

[0212] Deepseek-Coder-33B-Instruct scored the highest among the open source models.

[0213] The embodiment of the present invention further provides a large language model evaluation device; the specific device may include:

[0214] A data acquisition module is used to define typical data science task types and collect questions containing at least one typical data science task type to establish a question set;

[0215] A data processing module, used to generate and verify an answer for each question in the question set through a large language model to obtain a verified standard answer;

[0216] An evaluation function building module is used to define corresponding evaluation functions for different typical data science task types, aggregate the evaluation functions of each typical data science task type by function category, and obtain the aggregated evaluation function of each typical data science task type;

[0217] A test case set building module, used to generate a corresponding evaluation code for each question according to the aggregate evaluation function, and generate a test case set for large language model evaluation in combination with the standard answer corresponding to each question;

[0218] A target evaluation module is used to evaluate the target large language model based on the test case set.

[0219] The large language model evaluation device of this embodiment is used to implement the aforementioned large language model evaluation method. Therefore, the specific implementation method of the large language model evaluation device can be seen in the embodiment part of the large language model evaluation method above. For example, the data acquisition module, the data processing module, the evaluation function construction module, the test case set construction module, and the target evaluation module are respectively used to implement steps S101, S102, S103, S104 and S105 in the aforementioned large language model evaluation method. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part, which will not be repeated here.

[0220] A specific embodiment of the present invention further provides a large language model evaluation device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the above-mentioned large language model evaluation method when executing the computer program.

[0221] A specific embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned large language model evaluation method are implemented.

[0222] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0223] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0224] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0225] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0226] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A large language model evaluation method, characterized in that: include: Define typical data science task types and collect questions that contain at least one typical data science task type to create a question set; Generate and verify an answer for each question in the question set using a large language model to obtain a verified standard answer; Define corresponding evaluation functions for different typical data science task types, aggregate the evaluation functions of each typical data science task type by function category, and obtain the aggregated evaluation function of each typical data science task type; Generate a corresponding evaluation code for each question according to the aggregate evaluation function, and generate a test case set for large language model evaluation in combination with the standard answer corresponding to each question; The target large language model is evaluated based on the test case set.

2. The large language model evaluation method according to claim 1, characterized in that: The typical data science task types include, but are not limited to: data cleaning and preprocessing tasks, data exploration and statistical tasks, data visualization tasks, predictive modeling tasks, data mining and pattern recognition tasks, and interpretability and report generation tasks.

3. The large language model evaluation method according to claim 1, characterized in that: The collection contains at least one question of a typical data science task type, and establishing a question set includes: Collect the problems from preset channels, including but not limited to: data science task problems in actual programming problems, data science task problems in code bases, data science task problems generated by large language models, and data science task problems designed by domain experts; The problems are filtered using preset filtering rules, including but not limited to: keyword screening filtering, exclusion filtering requiring code update, error finding or concept explanation, and exclusion filtering requiring writing consistent with human preferences and large language models; Review the correctness, applicability and ease of assessment of the filtered questions; Format the reviewed questions and ensure data availability.

4. The large language model evaluation method according to claim 1, characterized in that: The large language model is used to generate and verify answers for each question in the question set, and the verified standard answers include: For each question, the large language model is sampled multiple times to generate multiple answer code snippets and corresponding answer outputs; Verify the answer output through preset test cases, or, A self-consistency strategy is used to cross-validate all generated answer code snippets; The verified answer is determined as the standard answer.

5. The large language model evaluation method according to claim 1, characterized in that: The definitions of corresponding evaluation functions for different typical data science task types include: Define corresponding evaluation functions for different typical data science task types; Generate a corresponding evaluation code for each problem according to the evaluation function; The typical data science task type, the evaluation function and the corresponding evaluation code of each problem are encapsulated into a triple; creating a directed acyclic graph in a hierarchy for each question based on the triples; The evaluation function is executed using a large language model to obtain expected results for each typical data science task type as a benchmark for evaluating the large language model.

6. The large language model evaluation method according to claim 1, characterized in that: The evaluation function of each typical data science task type is aggregated by function category to obtain the aggregated evaluation function of each typical data science task type, including: For each typical data science task type, select K optimal evaluation functions; Based on the K optimal evaluation functions, all evaluation functions of each typical data science task type are clustered into K clustered evaluation functions.

7. The large language model evaluation method according to claim 1, characterized in that: The generating of a corresponding evaluation code for each question according to the aggregate evaluation function, and combining the standard answer corresponding to each question to generate a test case set for large language model evaluation includes: Generate a corresponding evaluation code for each question according to the aggregate evaluation function; Determine the evaluation result by judging the consistency between the predicted output of the evaluation code corresponding to the aggregate evaluation function and the standard answer, and unify the evaluation result into a Boolean type or a decimal type between 0 and 1; The decimal evaluation result is converted into a Boolean value by setting a corresponding threshold.

8. A large language model evaluation device, characterized in that: include: A data acquisition module is used to define typical data science task types and collect questions containing at least one typical data science task type to establish a question set; A data processing module, used to generate and verify an answer for each question in the question set through a large language model to obtain a verified standard answer; An evaluation function building module is used to define corresponding evaluation functions for different typical data science task types, aggregate the evaluation functions of each typical data science task type by function category, and obtain the aggregated evaluation function of each typical data science task type; A test case set building module, used to generate a corresponding evaluation code for each question according to the aggregate evaluation function, and generate a test case set for large language model evaluation in combination with the standard answer corresponding to each question; A target evaluation module is used to evaluate the target large language model based on the test case set.

9. A large language model evaluation device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of a large language model evaluation method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the large language model evaluation method according to any one of claims 1 to 7 are implemented.

Images