A data science field large language model evaluation method and device and storage medium

Abstract

The present application relates to the technical field of data science, and in particular to a data science field large language model evaluation method, device, equipment and computer storage medium. The large language model evaluation method provided by the present application, from the perspective of coarse granularity, firstly aggregates the task type, function and corresponding code range, then, from the perspective of fine granularity, defines the programming rules for the output of each function according to the specific task, and compares the result with the standard answer, finally, the target large language model is evaluated through the generated test case set; the present application generates the standard answer and defines the specific evaluation index for each subtask from the comprehensive perspective, so as to be able to face the large language model evaluation of the more complex real world task composed of multiple subtasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data science, in particular to a large language model evaluation method, device and equipment in the field of data science and a computer storage medium. BACKGROUND

[0002] Large language models (LLM) are increasingly used in data science and scientific fields, such as data analysis, protein generation, and scientific reasoning. For data science tasks, LLMs provide the potential for (semi-) automated data analysis and data visualization by calling code interpreters and corresponding Python libraries for given publicly known problems. These works are benchmarked on relatively simple tasks and can accurately obtain true labels.

[0003] However, many real-world data analysis tasks require reasoning on more complex scenarios, such as evaluating the quality of images generated by data visualization tasks. Proper evaluation of these more complex data science tasks remains an open research direction. Although there are some existing benchmarks for evaluating LLMs on related challenges, these benchmarks usually focus on evaluating a narrow set of tasks with easily accessible standard answers and evaluation metrics. For example, MLAgentBench provides a machine learning research benchmark by building an LLM Agent pipeline. SWEBench benchmarks the ability of LLMs to solve real-world software problems from GitHub. InfiAgent–DABench generates labels with GPT-4 and calculates precision to complete data analysis tasks. The frontier of LLM evaluation is to face more complex real-world tasks composed of multiple subtasks. For these challenging problems, how to generate standard answers for each subtask from a comprehensive perspective and define specific evaluation metrics is a problem worth exploring. SUMMARY

[0004] To this end, the technical problem to be solved by the present application is to overcome the poor applicability of existing large language model evaluation benchmarks to complex real-world tasks.

[0005] To solve the above technical problems, the present application provides a large language model evaluation method, comprising:

[0006] defining a typical data science task type and collecting problems containing at least one typical data science task type to establish a problem set;

[0007] generating answers for each problem in the problem set by a large language model and verifying the answers to obtain standard answers that pass verification;

[0008] defining a corresponding evaluation function for different typical data science task types, performing function category aggregation on the evaluation function of each typical data science task type to obtain an aggregated evaluation function of each typical data science task type;

[0009] generating a corresponding evaluation code for each question according to the aggregated evaluation function, and combining the standard answer corresponding to each question to generate a test case set for large language model evaluation;

[0010] evaluating the target large language model based on the test case set.

[0011] Preferably, the typical data science task types include but are not limited to data cleaning and preprocessing tasks, data exploration and statistical tasks, data visualization tasks, predictive modeling tasks, data mining and pattern recognition tasks, and explainability and report generation tasks.

[0012] Preferably, the collection of questions containing at least one typical data science task type includes:

[0013] collecting the questions from preset channels, the preset channels including but not limited to data science task questions in actual programming questions, data science task questions in code libraries, data science task questions generated by a large language model, and data science task questions designed by domain experts;

[0014] filtering the questions through preset filtering rules, the preset filtering rules including but not limited to keyword screening filtering, exclusion filtering of questions requiring code updates, error finding or concept explanation, and exclusion filtering of questions requiring writing consistency with human preferences and large language models;

[0015] reviewing the correctness, applicability and evaluation ease of the filtered questions;

[0016] formatting the questions that pass the review and ensuring the availability of the data.

[0017] Preferably, the generation of answers by the large language model for each question in the question set and the verification of the answers include:

[0018] sampling the large language model multiple times for each question to generate multiple answer code segments and corresponding answer outputs;

[0019] verifying the answer outputs through preset test cases, or

[0020] cross-verifying all generated answer code segments using a self-consistency strategy;

[0021] determining the verified answers as standard answers.

[0022] Preferably, the defining of the corresponding evaluation functions for different typical data science task types comprises:

[0023] defining the corresponding evaluation functions for different typical data science task types;

[0024] generating the corresponding evaluation code for each question according to the evaluation function;

[0025] encapsulating the typical data science task type, the evaluation function and the corresponding evaluation code of each question as a triple;

[0026] creating a directed acyclic graph in a hierarchy for each question based on the triple;

[0027] using a large language model to execute the evaluation function to obtain the expected result of each typical data science task type as a benchmark for evaluating the large language model.

[0028] Preferably, the function category aggregation of the evaluation functions of each typical data science task type to obtain the aggregated evaluation function of each typical data science task type comprises:

[0029] selecting K optimal evaluation functions for each typical data science task type;

[0030] based on the K optimal evaluation functions, clustering all evaluation functions of each typical data science task type into K cluster evaluation functions.

[0031] Preferably, the generating of the corresponding evaluation code for each question according to the aggregated evaluation function, and the generating of the test case set for the evaluation of the large language model in combination with the standard answer corresponding to each question comprises:

[0032] generating the corresponding evaluation code for each question according to the aggregated evaluation function;

[0033] determining the evaluation result by judging the consistency between the predicted output of the evaluation code corresponding to the aggregated evaluation function and the standard answer, and unifying the evaluation result into a Boolean type or a decimal type between 0 and 1;

[0034] for the evaluation result of the decimal type, converting it into a Boolean value by setting a corresponding threshold.

[0035] The present application also provides a large language model evaluation device, comprising:

[0036] a data acquisition module for defining typical data science task types and collecting questions containing at least one typical data science task type to establish a question set;

[0037] A data processing module is used to generate and verify an answer for each question in the question set using a large language model to obtain a verified standard answer;

[0038] An evaluation function building module is used to define corresponding evaluation functions for different typical data science task types, aggregate the evaluation functions of each typical data science task type into function categories, and obtain the aggregated evaluation function for each typical data science task type;

[0039] A test case set construction module is used to generate corresponding evaluation code for each question according to the aggregate evaluation function, and generate a test case set for large language model evaluation in combination with the standard answer corresponding to each question;

[0040] A target evaluation module is used to evaluate the target large language model based on the test case set.

[0041] The present invention also provides a large language model evaluation device, comprising:

[0042] memory for storing computer programs;

[0043] A processor is configured to implement the steps of the aforementioned large language model evaluation method when executing the computer program.

[0044] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned large language model evaluation method are implemented.

[0045] The above technical solution of the present invention has the following advantages over the prior art:

[0046] The large language model evaluation method described in the present invention first summarizes the scope of task types, functions, and corresponding codes from a coarse-grained perspective. Then, from a fine-grained perspective, programming rules are defined for the output of each function according to the specific task, and the results are compared with the standard answers. Finally, the target large language model is evaluated using the generated test case set. From a comprehensive perspective, the present invention generates standard answers for each subtask and defines specific evaluation indicators, thereby enabling large language model evaluation for more complex real-world tasks composed of multiple subtasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0048] Figure 1 This is a flow chart of an implementation of a large language model evaluation method provided by the present invention;

[0049] Figure 2 is a framework diagram of the large language model evaluation benchmark DataSciBench provided by the present application;

[0050] Figure 3 is an all-result and average result diagram of all evaluation models;

[0051] Figure 4 is an average completion rate diagram of all evaluation models on different difficulty problems;

[0052] Figure 5 is a Pass@1 comparison diagram of all test LLMs between DataSciBench and HumanEval. DETAILED DESCRIPTION

[0053] The core of the present application is to provide a large language model evaluation method, device, equipment and computer storage medium in the field of data science, which effectively provides a large language model evaluation benchmark suitable for complex real-world tasks.

[0054] In order to enable personnel in the technical field to better understand the present application scheme, the present application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0055] Please refer to Figure 1 , Figure 1 is an implementation flowchart of a large language model evaluation method provided by the present application; the specific operation steps are as follows:

[0056] S101: defining typical data science task types and collecting problems containing at least one typical data science task type to establish a problem set;

[0057] S102: generating answers for each problem in the problem set by a large language model and verifying to obtain standard answers that pass verification;

[0058] S103: defining corresponding evaluation functions for different typical data science task types, and aggregating the evaluation functions of each typical data science task type by function category to obtain an aggregated evaluation function for each typical data science task type;

[0059] S104: generating corresponding evaluation codes for each problem according to the aggregated evaluation function, and combining the standard answers corresponding to each problem to generate a test case set for large language model evaluation;

[0060] S105: evaluating the target large language model based on the test case set.

[0061] Please refer to Figure 2 , Figure 2 The framework diagram of the large language model evaluation benchmark DataSciBench provided by the present application is as follows:

[0062] DataSciBench consists of three important components:

[0063] 1. Question definition and collection, which defines 6 task types and collects 222 real, challenging and high-quality question sets through question filtering and expert review.

[0064] 2. Response integration and verification, which proposes a novel task-function-code (TFC) to evaluate the key tasks of each question, and defines aggregation functions and programming rules to effectively evaluate specific task descriptions and compare predictions with standard answers.

[0065] 3. LLM evaluation, which evaluates 6 API-based models, 8 open-source general models and 9 open-source code generation models from a coarse-grained (e.g., success rate, completion rate) and fine-grained (e.g., VLM-as-a-judge, aggregation function) perspective.

[0066] The present application introduces a new benchmark, called DataSciBench, for evaluating the data science capabilities of LLMs, helping LLMs improve their data analysis and data visualization capabilities. For the collected questions and their corresponding responses and evaluation indicators, the present application hopes to meet the following characteristics: (1) more natural, challenging and high-quality questions are needed to promote the development of LLMs. (2) Each task requires multi-hop reasoning, and sequential tasks require strong correlation to distinguish models well. (3) Multiple types of results are needed for comprehensive evaluation.

[0067] Based on the above embodiment, step S101 is described in detail:

[0068] The typical data science task types include but are not limited to data cleaning and preprocessing tasks, data exploration and statistical tasks, data visualization tasks, predictive modeling tasks, data mining and pattern recognition tasks, and explainability and report generation tasks, wherein:

[0069] 1. Data cleaning and preprocessing tasks. This task detects and handles missing values, outliers and duplicate data; and standardizes data formats, such as uniform formats for dates and times.

[0070] 2. Data exploration and statistical tasks. This task is to calculate the basic statistical indicators of the data (mean, median, standard deviation, etc.), generate data distribution graphs (histogram, box plot, etc.), calculate the correlation between variables, and draw a correlation matrix or correlation graph.

[0071] 3. Data visualization tasks. The goal of this task is to visualize and analyze data and create interactive charts so that users can freely explore the data.

[0072] 4. Predictive modeling tasks. This task aims to select appropriate machine learning algorithms such as linear regression, decision trees, random forests, etc.; perform feature engineering such as feature selection, feature transformation, feature combination, etc.; divide the dataset into training and test sets, train and evaluate the model; and select the appropriate evaluation metrics for different prediction problems such as classification, regression, or clustering.

[0073] 5. Data mining and pattern recognition tasks. This task uses association rule mining, frequent item set mining, etc. to find interesting patterns in data; text mining techniques are used to extract keywords, topics, etc. from text data; and clustering analysis, classification algorithms, etc. are applied to identify underlying patterns and structures. Pattern recognition tasks can complete the following functions: image recognition, text clustering, time series detection.

[0074] 6. Explainability and report generation tasks. This task aims to provide explanations for model results such as feature importance, model parameters, etc., and automatically generate reports and summaries to present analysis results in an easy-to-understand and shareable manner.

[0075] To increase the difficulty of the evaluation problem, this embodiment selects a more complex problem that includes multiple tasks. These sequential tasks can be any combination of tasks.

[0076] The collection includes at least one typical data science task type problem, and the problem set is established, including:

[0077] The problems are collected from preset channels, including but not limited to: data science task problems in actual programming problems, data science task problems in code libraries, data science task problems generated by large language models, and data science task problems designed by domain experts, wherein in a specific embodiment:

[0078] · Collect real problems from users on an online code generation platform CodeGeeX to achieve coarse-grained collection from real-world online platforms.

[0079] · Select high-quality data science-related problems from BigCodeBench and then rewrite them to meet the requirements of the present invention.

[0080] • Using a few-shot problem approach to ask LLM to generate similar problems.

[0081] • Detailed questions were written with reference to relevant websites to increase the difficulty and robustness of the evaluation benchmark.

[0082] The questions are filtered by pre-set filtering rules, including but not limited to: keyword screening filtering (such as task name, deep learning and machine learning), exclusion filtering of code update, error finding or concept explanation, and exclusion filtering of writing consistency with human preference and large language model; that is:

[0083] Low-quality questions are filtered by the following three principles: (1) Select questions containing keywords such as task name, deep learning and machine learning. (2) Filter questions that require code updates, error finding and concept explanation. (3) Filter questions that require rewriting consistency with human preference and LLM.

[0084] The correctness, applicability and evaluation ease of the filtered questions are reviewed, the questions that pass the review are formatted, and the availability of the data is ensured; that is:

[0085] To ensure the quality of the collected questions, the present invention is reviewed by experts in computer science and data analysis. The review process consists of three stages: (1) In the first stage, the expert verifies the correctness of the question and adjusts the applicability of the question. In addition, the expert ensures that the answer to the question is easy to evaluate. (2) In the second stage, the expert formats the input, output and instruction problem of the question, which makes the LLM follow uniform instructions. (3) In the third stage, the expert ensures the availability of the input question dataset, including generating random datasets and collecting public datasets.

[0086] Based on the above embodiment, the present embodiment details step S102:

[0087] For each question, the large language model is sampled multiple times to generate multiple answer code snippets and corresponding answer outputs; for each collected question, the LLM is required to generate multiple independent responses, which may include code snippets, data analysis results or other forms of answers, and the LLM-generated code is executed in a safe sandbox environment, and its running state and output results are monitored.

[0088] The present invention increases the diversity and reliability of the results by multiple sampling, reduces the bias brought by randomness, and verifies whether the code snippets generated by the LLM can be correctly executed to produce the expected output.

[0089] To ensure the reasonableness and reliability of the answers generated by the LLM, the present invention uses two different methods for verification:

[0090] • Verify answer output by pre-set test cases; for problems from BigCodeBench, these problems provide reliable test cases, and by executing all test cases, the generated answers are verified, and then experts review the answers that pass the test cases to confirm whether they meet the expected solutions.

[0091] • Cross-verify all generated answer code snippets using self-consistency strategy; for other problems that do not provide test cases, cross-verify all generated code using the self-consistency strategy, i.e. compare multiple answers generated for the same problem, check if they are consistent, and if they can all reasonably solve the given problem.

[0092] Determine the verified answers as standard answers.

[0093] Through the above strategies, it can be ensured that the answers generated by LLM are not only technically feasible, but also reasonable and reliable in practical applications. Multiple sampling and cross-validation help to reduce the bias of single output and improve the stability of evaluation results. Through explicit verification steps, the transparency of the evaluation process is improved, making the evaluation results easier to understand and accept.

[0094] In order to generate standard answers and obtain evaluation indicators for each sub-task, the present application proposes a semi-automatic framework called Task-Function-Code (TFC) generation and evaluation. Based on the above embodiment, this embodiment describes step S103 in detail:

[0095] Define corresponding evaluation functions for different typical data science task types; in one embodiment:

[0096] Identify all relevant data science task types, such as data cleaning, data exploration, predictive modeling, etc., and design and predefine evaluation functions for each task type, which can quantify the completion and quality of the task;

[0097] Using the capabilities of GPT-4o-mini, select task types according to predefined criteria (such as task complexity, commonality, importance, etc.), and return the corresponding evaluation functions for these selected task types.

[0098] Generate corresponding evaluation code for each problem according to the evaluation function; that is, generate evaluation code for each task problem according to the selected task type and pre-defined evaluation function, which will be used to actually perform evaluation and generate evaluation results.

[0099] Encapsulate the typical data science task type, the evaluation function and the corresponding evaluation code for each problem as a triple wherein:

[0100] T (Task Type): Task type;

[0101] F (Evaluation Function): Evaluation function;

[0102] C (Evaluation Code): Evaluation code;

[0103] N: The number of valuable task types in each question;

[0104] For each task question, determine its corresponding task type, select the corresponding evaluation function, and generate the evaluation code, then encapsulate these information as a tuple.

[0105] Based on the triplets, create a directed acyclic graph (DAG) in a hierarchy for each question; analyze the task question using a data interpreter (DI), identify the task type and its dependencies, and generate a DAG for each task question, where each task type is defined as a node in the DAG.

[0106] Using a large language model to execute the evaluation function, obtain the expected results of each typical data science task type as a benchmark for evaluating the large language model. Based on the generated DAG, use a powerful LLM as the backbone to run all evaluation functions. Obtain the expected results (ground truth) of each task type, which will be used as a benchmark for evaluating the LLM output.

[0107] Based on the above embodiment, this embodiment continues to describe step S103 in detail:

[0108] In order to unify the key functions and improve the scalability of the evaluation, the present application selects top-K functions for each task type, and aggregates all generated functions into top-K function categories:

[0109] For each typical data science task type, select K optimal evaluation functions;

[0110] Based on the K optimal evaluation functions, cluster all evaluation functions of each typical data science task type into K cluster evaluation functions.

[0111] In a specific embodiment, the categories of typical data science task types can be specifically divided as follows:

[0112] Data cleaning and preprocessing

[0113] Data cleaning integrity: Ensure that the records in the dataset remain complete after cleaning without unnecessary loss.

[0114] Data frame shape validation: Verify the dimensions of a DataFrame, such as the number of rows and columns, against expectations.

[0115] Data integrity: Check that missing values in the data are handled appropriately, ensuring the integrity of the data.

[0116] Normalization range check: Verify that data is within the expected range after normalization or standardization, such as 0 to 1 or -1 to 1.

[0117] Data quality score: Assign a score to the overall quality of the data, potentially including consistency, accuracy, and reliability.

[0118] Data exploration and statistical analysis

[0119] Descriptive statistics: Calculate central tendencies (mean, median) and dispersion measures (variance, standard deviation) of the data.

[0120] Correlation analysis: Assess the correlation between variables, such as Pearson's correlation coefficient or Spearman's rank correlation coefficient.

[0121] Distribution analysis: Analyze the distribution characteristics of the data, such as normality tests.

[0122] Outlier detection: Identify and handle outliers in the data.

[0123] Statistical visualization: Generate statistical plots, such as histograms and box plots, to visually represent data characteristics.

[0124] Data visualization

[0125] Interactive charts: Create charts that users can interact with, such as scalable scatter plots or filterable bar charts.

[0126] Informational chart design: Design informative charts to convey key data points.

[0127] Multidimensional data visualization: Use techniques like heatmaps and parallel coordinates to visualize multidimensional data.

[0128] Geospatial data visualization: Visualize geographic data, such as maps and heatmaps.

[0129] Dynamic data visualization: Create dynamic charts that show data changes over time.

[0130] Predictive modeling

[0131] Model selection: Choose appropriate predictive models, such as linear regression, decision trees, random forests, and others.

[0132] Feature engineering: Perform feature selection, feature transformation, and feature combination.

[0133] Model training and evaluation: Train the model and evaluate its performance using techniques such as cross-validation.

[0134] Hyperparameter optimization: Adjust the hyperparameters of the model to optimize performance.

[0135] Prediction accuracy: Evaluate the accuracy of the model's predictions, such as mean squared error (MSE) or accuracy rate.

[0136] Data mining and pattern recognition

[0137] Association rule mining: Discover frequent itemsets and association rules in the data.

[0138] Clustering analysis: Cluster data to identify natural groupings in the data.

[0139] Classification and classifier learning: Train classifiers to classify data.

[0140] Anomaly detection: Identify unusual patterns or outliers in the data.

[0141] Trend and seasonality analysis: Identify trends and seasonal patterns in time series data.

[0142] Interpretability and report generation

[0143] Model interpretability: Provide explanations for the model's predictions, such as feature importance analysis.

[0144] Result summarization: Generate summaries of key results for quick understanding.

[0145] Report automation: Automatically generate detailed analysis reports.

[0146] Data storytelling: Tell the story behind the data through data visualization and narrative.

[0147] Decision support: Provide analysis and recommendations to support decision-making.

[0148] Based on the above embodiments, this embodiment details step S104:

[0149] Generate the corresponding evaluation code for each question according to the aggregation evaluation function;

[0150] Determine the evaluation result by judging the consistency between the prediction output of the evaluation code corresponding to the aggregation evaluation function and the standard answer, and unify the evaluation result into a Boolean type or a decimal type between 0 and 1;

[0151] For the evaluation results of the decimal type, set appropriate thresholds for different aggregation functions according to the specific requirements and expected results of the task, and convert them to Boolean values; the selection of thresholds should be based on domain knowledge and experience to ensure the reasonableness and accuracy of the evaluation results.

[0152] In a specific embodiment, all outputs are unified into Boolean or decimal values between 0 and 1, where Boolean values represent whether the task is successfully completed (1 for success, 0 for failure), and decimal values provide more detailed performance metrics such as accuracy, similarity, etc.

[0153] The following are two specific examples to illustrate the above method:

[0154] Data cleaning integrity

[0155] Task description: Verify the data integrity of the data after preprocessing, ensuring that there is no unnecessary data loss.

[0156] Verification method:

[0157] Calculate the final number of rows or columns after preprocessing.

[0158] Compare the calculation result with the standard answer (expected number of rows or columns).

[0159] If they are the same, output 1 (success); if they are different, output 0 (failure).

[0160] Data mining and pattern recognition

[0161] Task description: Evaluate the performance of data mining tasks such as the silhouette coefficient of clustering analysis.

[0162] Verification method:

[0163] Calculate the silhouette coefficient, which is an indicator of clustering effectiveness, with a value range usually between -1 and 1.

[0164] Set a threshold (e.g. 0.5) to convert the value of the silhouette coefficient to a Boolean value:

[0165] If the silhouette coefficient is greater than or equal to 0.5, output 1 (indicating good clustering effect).

[0166] If the silhouette coefficient is less than 0.5, output 0 (indicating poor clustering effect).

[0167] Suppose there is a data cleaning task where the data is expected to have 100 rows after preprocessing. The result obtained after executing the code generated by the LLM is 98 rows. According to the defined rules:

[0168] Calculate the difference in the number of rows: |100-98|=2

[0169] Due to the mismatch in the number of rows, the final output is 0 (failure).

[0170] For a clustering task, the silhouette coefficient obtained after the LLM-generated code is executed is 0.55. According to the defined rule:

[0171] Since the silhouette coefficient is greater than the threshold value of 0.5, the final output is 1 (success).

[0172] Through these unified rules and threshold settings, the DataSciBench of the present application can provide a clear, quantifiable evaluation standard for the performance of LLMs in data science tasks. This method not only improves the consistency and comparability of evaluation, but also simplifies the evaluation process, making the results easier to understand and operate.

[0173] Based on the above embodiments, this embodiment explains step S105 in detail:

[0174] In one specific application, we obtained 222 valid questions and corresponding test cases, which helped to evaluate API-based and open-source models as follows:

[0175] In order to evaluate the performance of different models (for example, API-based models and open-source general / code generation models), this embodiment builds a comprehensive benchmark on the collected questions.

[0176] (1) Experimental setup

[0177] • Six API-based models include o1-mini / GPT-4o-mini / GPT-4o-2024-05-13 / GPT-4-Turbo, Claude-3.5-Sonnet and GLM-4-Flash.

[0178] • 8 open-source general models include Llama3.1-8B-Instruct, Llama3-8B-Instruct, Qwen2.5-7B-Instruct, Qwen2-1.5 / 7B-Instruct, Gemma2-9B-it, GLM-4-9B-Chat, Yi-1.5-9B-Chat-16k.

[0179] • 9 open-source code generation models include Deepseek-Coder-1.3 / 6.7 / 33B-Instruct, Codellama-7 / 13 / 34B-Instruct, Qwen2.5-Coder-1.5 / 7B-Instruct, StarCoder2-15B.

[0180] (2) Evaluation metrics

[0181] Coarse-grained metrics. The present invention defines coarse-grained metrics (CR and SR) for evaluating LLMs.

[0182] • Completion Rate (CR). According to the Data Interpreter, the present invention computes the completion rate for a given TFC. For each TFC in the list of TFCs, the present invention gives it a completion score, with the lowest score being 0 and the highest score being 2. The given step completion scores are as follows: skip (score of 0), fail (score of 0), success-non-compliant (1 score), and success-compliant (2 score). The final completion rate is then computed as follows:

[0183]

[0184] The numerator is the sum of the completion scores for each step, while the denominator is the sum of the maximum possible scores for all steps, which is 2 x T, T being the number of TFCs.

[0185] • Success Rate (SR). Similar to Codex, the success rate of the present invention is defined as the estimated single-problem complete success rate under 10 runs. Specifically, if all TFCs pass in a single run of a problem symbol, it will be considered a success. Otherwise, it will be considered a failure. Note that for problems obtained from BigCodeBench, the present invention compares the output of the completion function with the output of the ground truth completion function to determine whether a single run passes, as the TFCs are derived from the required function output in this case.

[0186] Fine-grained aggregate metrics. The present invention also defines fine-grained aggregate metrics for evaluating all LLMs in detail.

[0187] • Visual Language Model (VLM)-as-a-judge evaluation based on pre-defined criteria for the overall score of two inputs, providing a step-by-step rationale for its evaluation.

[0188] • Data Quality Score (F1) in data cleaning and pre-processing, aiming to evaluate the cleanliness of the data after pre-processing. It produces a Boolean output of 1 if it matches the standard answer, otherwise 0.

[0189] • Plot Validity (F2) in data visualization is related to the accuracy of the visual representation, for example, checking whether the shape of the correlation matrix matches the standard answer. If it matches, the final value is 1, otherwise 0.

[0190] • Data accuracy in data exploration and statistical understanding (F3) focuses on understanding data quality, which can be quantified using mean squared error (MSE). By comparing it with the true value with a predefined threshold, the final value is obtained.

[0191] • Visualization integrity in data visualization (F4) assesses the comprehensiveness of generated images (e.g., PNG, jpeg, PDF) by checking their existence against standard answers. If the file exists, it scores 1, otherwise 0.

[0192] • Model accuracy in predictive modeling (F5) is used to measure the predictive performance of the model, providing a Boolean accuracy value or a decimal between 0 and 1.

[0193] (3) Overall experimental results

[0194] The overall experimental results are shown in Table 1 and Figure 3 Table 1 Overall experimental results

[0195] Table 1 Overall experimental results

[0196]

[0197] (1) In terms of average performance, API-based models are significantly better than open-source models. Among API-based models, GPT-4o has the highest total score of 64.43%, which is 9.84% higher than GPT-4-Turbo with a total score of 54.59%. Notably, GPT-4o also outperforms all other models in all metrics except VLM, indicating its comprehensive capabilities in various aspects.(2) For open-source models, the performance gap between general-purpose models and code generation models is not significant. Among them, Deepseek-Coder-33B-Instruct scored the highest at 56.74%, even exceeding various closed-source models such as o1-mini and GPT-4-Turbo. Other models such as Qwen2.5-Coder-7B-Instruct and Qwen2.5-7b-Instruct also showed quite good capabilities, with total scores of 47.67% and 45.99%, respectively. In contrast, there are also some models that only passed a few tasks, with total scores even lower than 5.0%. Among them, CodeLlama-34B-Instruct surprisingly scored 1.33%, even falling behind its small version CodeLlama-7B-Instruct. This anomaly is analyzed by the present invention.(3) In addition, the present invention analyzes the performance of each model in each task, and the results are shown in Table 2. Figure 3The overall and average scores are shown in Figure 6. It can be seen that the average performance of the API-based models is better than that of the open-source models, with the average scores of the open-source models being 49.15%~23.01%. In contrast, the performance difference between the API-based models is smaller than that of the open-source models.

[0198] (4) Experimental results of problems of different difficulties

[0199] To evaluate the ability of multiple LLMs to complete problems of different difficulties, the present application classifies tasks using BCB and CSV format data, human handwritten problems, and deep learning tasks related to data science into simple, medium, and difficult levels, respectively. The present application evaluates multiple LLMs by combining the overall average CR and the average CR of each difficulty level. From Figure 4 It can be observed from Table 6 that: (1) Consistency across difficulties: Some LLMs, such as GPT-4o, GPT-4o-mini, GPT-4-Turbo, and Deepseek-Coder-33B-Instruct, exhibit consistent performance across all difficulty levels, indicating robustness. (2) Models that perform best at the difficult level: Models such as the GPT-4 series and the Deepseek-Coder-Instruct series are among the best-performing models, achieving high average CRs, especially in complex, data-driven tasks. (3) Performance variance: There is a significant difference in average CR between general-purpose models and small-scale models, with some models having low overall scores, indicating that general-purpose models are less efficient or accurate in data science tasks.

[0200] (5) Comparison with other benchmarks

[0201] Comparison with HumanEval in Figure 5 The present application compares DataSciBench with HumanEval. As shown in Figure 7, the present application observes that most LLMs are located in the upper triangular region of the graph, and all tested models are divided into two groups, where circle markers represent API-based models, and other markers represent various open-source LLMs, the dashed area LLMs perform well on both benchmarks, and the solid area has relatively mismatched performance on the two datasets. Figure 5 Correlation analysis with other benchmarks in Table 2:

[0202] Table 2 Correlation analysis

[0203]

[0204]

[0205] ​The present invention performs correlation analysis to assess the benchmarks of the present invention and the consistency between the benchmarks and established code evaluation benchmarks (e.g., BigCodeBench and LiveCodeBench). To achieve this, the present invention computes the Pearson’s r and Spearman’s p correlation coefficients, which provide insights into the strength and direction of the relationship between the benchmarks and these established metrics. This analysis not only validates the results of the present invention but also ensures robustness across different evaluation dimensions. The results of the present invention’s study show strong positive correlations, indicating that the benchmarks of the present invention are well-aligned with these established code evaluation metrics.

[0206] (6) Research or engineering observations

[0207] With carefully curated metrics, the present invention is able to gain deeper insights into the capabilities of LLMs in planning and executing complex data science tasks. The experimental results also raise questions worth exploring as some results do not align with conventional wisdom.

[0208] Models are good at reasoning but do not necessarily perform better on complex data science tasks. While data science coding tasks do often involve scheduling and step-by-step execution similar to reasoning scenarios, the results show that even LLMs that excel at reasoning tasks still struggle with complex data science tasks. For example, OpenAI’s o1-mini model, which is generally considered one of the best reasoning models, surprisingly fails on many DataSciBench tasks. The overall success rate of this model is only 29.77%, significantly trailing behind models previously released by the company, such as GPT-4o and GPT-4-Turbo. Upon examining the outputs generated by o1-mini, the present invention found that failures were mainly due to non-compliance with instructions, incorrect calls, and forgetting. While successfully breaking down tasks into multiple sub-tasks, the model often forgot to export the required execution results or only outputted unnecessary data. In other cases, the model might incorrectly call library functions or methods that sometimes do not even exist. These facts remind the present invention that real-life data science coding tasks often comprehensively challenge the ability of models to follow fine-grained instructions, leverage existing tools (libraries, APIs, …), and plan. To perform and coordinate these tasks well, models must be competitive in all relevant aspects.

[0209] Large models can sometimes fail to follow simple instructions more frequently. StarCoder 2-15B underperforms some smaller models, and Codellam-34B-Instruct even underperforms its 13B and 7B versions. The main reason is that the large versions lack some other capabilities, such as generating formatted text according to the question. Perhaps a large amount of data in a certain format was used to train a larger version, and this version did not generate another different format according to the question. In fact, in LiveCodeBench, large-scale models of CodeLlama also cannot outperform small-scale versions.

[0210] In summary, the present application provides DataSciBench, a comprehensive benchmark to evaluate the capabilities of large language models (LLMs) in data science. Recent related benchmarks mainly focus on single tasks, easily accessible standard answers, and direct evaluation metrics, which limit the range of tasks that can be evaluated. In contrast, DataSciBench is built on a more comprehensive and natural set of challenging questions. The present application develops a semi-automatic pipeline for generating ground truth (GT) and validating evaluation metrics. The pipeline leverages and implements a self-consistency strategy based on LLMs, generating accurate GT by utilizing collected questions, pre-defined task types, and aggregated metrics. Additionally, it employs a comprehensive approach to filter a high-quality list of task-function-code (TFC) and evaluate each code execution result in TFC according to precisely defined metrics and programming rules. The experimental framework of the present application includes testing 6 API-based models, 8 open-source general-purpose models, and 9 open-source code generation models using various question sets collected by the present application. Through this approach, the present application aims to provide a more comprehensive and rigorous evaluation of LLMs in the field of data science, revealing their strengths and weaknesses. Experimental results show that, except for VLM-as-a-judge

[0211] API-based models significantly outperform open-source models on all metrics.

[0212] Deepseek-Coder-33B-Instruct scores the highest among open-source models.

[0213] The embodiments of the present application also provide a large language model evaluation device; the specific device can include:

[0214] a data acquisition module configured to define typical data science task types and collect questions containing at least one typical data science task type to establish a question set;

[0215] a data processing module configured to generate answers for each question in the question set by a large language model and perform verification to obtain standard answers that pass the verification; and

[0216] an evaluation function construction module configured to define a corresponding evaluation function for each of the different typical data science task types, and to aggregate the evaluation functions for each of the typical data science task types by function category to obtain an aggregated evaluation function for each of the typical data science task types;

[0217] a test case set construction module configured to generate a corresponding evaluation code for each question according to the aggregated evaluation function, and to generate a test case set for evaluating the large language model in combination with the standard answer corresponding to each question;

[0218] a target evaluation module configured to evaluate the target large language model based on the test case set.

[0219] The large language model evaluation device of the embodiment is used to implement the large language model evaluation method described above, and thus the specific embodiments of the large language model evaluation device can be seen from the embodiment part of the large language model evaluation method, for example, the data acquisition module, the data processing module, the evaluation function construction module, the test case set construction module, and the target evaluation module are respectively used to implement steps S101, S102, S103, S104, and S105 in the large language model evaluation method described above, and thus the specific embodiments can be referred to the description of the respective embodiment parts, which will not be described herein again.

[0220] The embodiment of the present application also provides a large language model evaluation device, which comprises a memory configured to store a computer program, and a processor configured to implement the steps of the large language model evaluation method described above when executing the computer program.

[0221] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is configured to implement the steps of the large language model evaluation method described above when executed by a processor.

[0222] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0223] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0224] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0225] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0226] Obviously, the above-described embodiments are only examples for clarity of description and are not limiting on the embodiments. Based on the above description, one of ordinary skill in the art can further make other different forms of changes or modifications. Here, all the embodiments are not required to be enumerated, and the obvious changes or modifications derived therefrom are still within the protection scope of the present application.

Claims

1. A large language model evaluation method, characterized in that, The method comprises the following steps: Defining typical data science task types and collecting questions containing at least one typical data science task type to establish a question set; Generating answers for each question in the question set by a large language model and verifying them to obtain standard answers that pass the verification; Defining corresponding evaluation functions for different typical data science task types, and aggregating the evaluation functions of each typical data science task type by function category to obtain an aggregated evaluation function for each typical data science task type: Defining corresponding evaluation functions for different typical data science task types; Generating corresponding evaluation codes for each question according to the evaluation functions; Packaging the typical data science task type, the evaluation function and the corresponding evaluation code of each question into a triple; Creating a directed acyclic graph in a hierarchy for each question based on the triple; Using a large language model to execute the evaluation function to obtain the expected result of each typical data science task type as a benchmark for evaluating the large language model; Generating corresponding evaluation codes for each question according to the aggregated evaluation function, and combining the corresponding standard answers to generate a test case set for evaluating the large language model: Generating corresponding evaluation codes for each question according to the aggregated evaluation function; Determining the evaluation result by judging the consistency between the prediction output of the corresponding evaluation code of the aggregated evaluation function and the standard answer, and unifying the evaluation result into a Boolean type or a decimal type between 0 and 1; For the evaluation result of the decimal type, convert it to a Boolean value by setting a corresponding threshold; Evaluating the target large language model based on the test case set.

2. The method of claim 1, wherein, The typical data science task types include but are not limited to data cleaning and preprocessing tasks, data exploration and statistical tasks, data visualization tasks, predictive modeling tasks, data mining and pattern recognition tasks, and explainability and report generation tasks.

3. The method of claim 1, wherein, The collection of questions containing at least one typical data science task type to establish a question set comprises: Collecting the questions from preset channels, which include but are not limited to data science task questions in actual programming problems, data science task questions in code libraries, data science task questions generated by a large language model, and data science task questions designed by domain experts; Filtering the questions by preset filtering rules, which include but are not limited to keyword screening filtering, exclusion filtering of questions that require code updates, error finding or concept explanation, and exclusion filtering of questions that require writing consistency with human preferences and large language models; Reviewing the correctness, applicability and evaluation ease of the filtered questions; Formatting the questions that pass the review and ensuring the availability of the data.

4. The method of claim 1, wherein, The method of generating answers for each question in the question set by a large language model and verifying them to obtain standard answers that pass the verification comprises: For each question, multiple sampling of the large language model is performed to generate multiple answer code segments and corresponding answer outputs; Verifying the answer output by a preset test case, or Cross-validation is performed on all generated answer code snippets using a self-consistency strategy; The validated answers are determined as standard answers.

5. The method of claim 1, wherein, The evaluation function for each typical data science task type is aggregated by function category to obtain an aggregated evaluation function for each typical data science task type, which includes: For each typical data science task type, select K optimal evaluation functions; Based on the K optimal evaluation functions, all evaluation functions for each typical data science task type are clustered into K cluster evaluation functions.

6. A large language model evaluation apparatus, characterized by comprising: It includes: The data acquisition module is used to define typical data science task types and collect questions containing at least one typical data science task type to establish a question set; The data processing module is used to generate answers for each question in the question set through a large language model and verify them to obtain standard answers that pass verification; The evaluation function construction module is used to define corresponding evaluation functions for different typical data science task types, aggregate the evaluation functions for each typical data science task type by function category, and obtain an aggregated evaluation function for each typical data science task type: Define corresponding evaluation functions for different typical data science task types; Generate corresponding evaluation code for each question according to the evaluation function; The typical data science task type, the evaluation function, and the corresponding evaluation code for each question are encapsulated into a triple; Based on the triple, a directed acyclic graph in a hierarchical structure is created for each question; Use a large language model to execute the evaluation function to obtain the expected result of each typical data science task type as a benchmark for evaluating the large language model; The test case set construction module is used to generate corresponding evaluation code for each question according to the aggregated evaluation function, and combine the corresponding standard answers for each question to generate a test case set for large language model evaluation: Generate corresponding evaluation code for each question according to the aggregated evaluation function; Determine the evaluation result by judging the consistency between the predicted output of the corresponding evaluation code of the aggregated evaluation function and the standard answer, and unify the evaluation result into a Boolean type or a decimal type between 0 and 1; For the evaluation result of the decimal type, convert it to a Boolean value by setting a corresponding threshold; The target evaluation module is used to evaluate the target large language model based on the test case set.

7. A large language model evaluation device, characterized by, It includes: Memory for storing computer programs; The processor is used to execute the computer program to realize the steps of the large language model evaluation method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to realize the steps of the large language model evaluation method according to any one of claims 1 to 5.

Images