Data set evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides a data set evaluation method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the preprocessing of a to-be-evaluated data set, obtaining a standard data set, and enabling the preprocessing at least to comprise a classification mark, and enabling the classification mark to be used for marking the data type of the to-be-evaluated data set; constructing a target data set according to the standard data set; performing zero-sample reasoning or few-sample reasoning on the reference model according to the target data set to obtain an initial score of the target data set; performing intervention training based on parameter efficient fine tuning on the reference model according to the standard data set to obtain a target model; performing zero sample reasoning or few sample reasoning on the target model according to the target data set to obtain an update score of the target data set; and calculating a performance gain rate of the reference model according to the initial score and the updated score, and evaluating the to-be-evaluated data set according to the performance gain rate. According to the method and the device, the accuracy of data set quality evaluation can be improved, and the scene adaptability of data set evaluation can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a dataset evaluation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, data has become a core element driving the leap in the capabilities of large models. According to the "Guidelines for Building High-Quality Datasets" and the "Practical Guidelines for High-Quality Datasets (1.0)," a high-quality dataset is a collection of data that has been collected, processed, and other measures that can effectively improve model performance.

[0003] Existing mainstream dataset evaluation technologies mainly focus on the "static evaluation" dimension: document integrity and format standardization: checking whether the data has clear metadata (such as id, source, label, etc. fields specified in the "Requirements for High-Quality Dataset Format") and whether the file structure conforms to the JSON / Parquet standard; content compliance and basic quality: using rule engines or regular expressions to detect whether the data contains privacy information, garbled characters, duplicate data, or content that violates laws and regulations.

[0004] However, existing technologies have significant limitations: they cannot quantify the "value of knowledge," and static metrics cannot identify data that is "correct in content but lacks substance." For example, a large amount of repetitive, irrelevant information or synthetic data generated by low-level models may be perfectly compliant in terms of format and syntax, but it cannot improve the model's logical reasoning or professional capabilities. There is a lack of objective verification for scenario adaptability: the "High-Quality Dataset Classification Guide" categorizes data into general knowledge, industry general knowledge, and industry specialized knowledge. Static labels alone cannot determine whether a particular "medical dataset" can truly improve a model's performance in a "medical diagnosis" scenario.

[0005] Therefore, a new dataset evaluation method is urgently needed to solve the above problems. Summary of the Invention

[0006] In view of this, this application provides a dataset evaluation method, apparatus, electronic device, and storage medium, which can improve the accuracy of dataset quality evaluation and enhance the scenario adaptability of dataset evaluation.

[0007] A first aspect of this application provides a dataset evaluation method, comprising: preprocessing a dataset to be evaluated to obtain a standard dataset, wherein the preprocessing includes at least classification labels used to label the data type of the dataset to be evaluated; constructing a target dataset based on the standard dataset, wherein the data type of the target dataset is the same as that of the dataset to be evaluated, and the data content is isolated from that of the dataset to be evaluated; performing zero-shot or few-shot inference on a benchmark model based on the target dataset to obtain an initial score for the target dataset, wherein the benchmark model is an AI model that has not been trained on the dataset to be evaluated; performing parameter-efficient fine-tuning intervention training on the benchmark model based on the standard dataset to obtain a target model; performing zero-shot or few-shot inference on the target model based on the target dataset to obtain an updated score for the target dataset; calculating the performance gain ratio of the benchmark model based on the initial score and the updated score, and evaluating the dataset to be evaluated based on the performance gain ratio.

[0008] In one possible implementation, calculating the performance gain ratio of the benchmark model based on the initial score and the updated score includes: calculating the performance gain ratio according to the following formula:

[0009] ; Where Gain is the performance gain rate, and Score is... Tuned For the updated score, Score Base The initial score is given.

[0010] In one possible implementation, evaluating the dataset to be evaluated based on the performance gain ratio includes: evaluating the dataset to be evaluated as a high-quality dataset when the performance gain ratio is greater than a preset threshold; evaluating the dataset to be evaluated as a normal dataset when the performance gain is greater than or equal to 0 and less than or equal to the preset threshold; and evaluating the dataset to be evaluated as a low-quality dataset when the performance gain is less than 0.

[0011] In one possible implementation, the intervention training of the benchmark model based on the standard dataset for parameter-efficient fine-tuning includes: loading the weights of the benchmark model and freezing the main parameters of the benchmark model; injecting a low-rank matrix into the transport layer bypass of the benchmark model; inputting the standard dataset into the benchmark model and updating the parameters of the low-rank matrix according to the output of the benchmark model; and updating the weights according to the gradient-updated low-rank matrix.

[0012] In one possible implementation, constructing the target dataset based on the standard dataset includes: randomly selecting a preset number of input data from the standard dataset; inputting the input data into a preset large language model, wherein the large language model outputs target data corresponding to the input data according to preset execution instructions; and constructing the target dataset based on the target data.

[0013] In one possible implementation, after constructing the target dataset based on the target data, the method further includes: removing the input data from the standard dataset to obtain the final dataset; the step of performing intervention training on the benchmark model based on the standard dataset using parameter-efficient fine-tuning includes: performing intervention training on the benchmark model based on the final dataset using parameter-efficient fine-tuning.

[0014] In one possible implementation, the preprocessing further includes metadata verification and noise removal cleaning; the metadata verification is used to detect whether there are missing target fields or incorrectly formatted data in the dataset to be evaluated; the noise removal cleaning is used to remove the data to be cleaned.

[0015] Secondly, embodiments of this application also provide a dataset evaluation device, comprising: a preprocessing module, a construction module, an inference module, an intervention training module, a computation module, and an evaluation module; the preprocessing module is used to preprocess the dataset to be evaluated to obtain a standard dataset, wherein the preprocessing includes at least classification labels, the classification labels being used to label the data type of the dataset to be evaluated; the construction module is used to construct a target dataset based on the standard dataset, wherein the data type of the target dataset is the same as that of the dataset to be evaluated, and the data content is isolated from that of the dataset to be evaluated; the inference module is used to perform inference on a benchmark model based on the target dataset. The initial score of the target dataset is obtained through zero-shot or few-shot inference, wherein the benchmark model is an AI model that has not been trained on the dataset to be evaluated; the intervention training module is used to perform parameter-efficient fine-tuning intervention training on the benchmark model based on the standard dataset to obtain the target model; the inference module is also used to perform zero-shot or few-shot inference on the target model based on the target dataset to obtain an updated score of the target dataset; the calculation module is used to calculate the performance gain ratio of the benchmark model based on the initial score and the updated score; the evaluation module is used to evaluate the dataset to be evaluated based on the performance gain ratio.

[0016] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, causing the electronic device to execute the dataset evaluation method as described in the first aspect.

[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the dataset evaluation method as described in the first aspect.

[0018] Compared with related technologies, the embodiments of this application have at least the following advantages: By classifying and labeling the dataset to be evaluated, the data type of the dataset to be evaluated can be accurately determined. Then, by constructing a target dataset based on a standard dataset, since the data type of the target dataset is the same as that of the dataset to be evaluated and the data content is isolated from the dataset to be evaluated, the target dataset can be used as a test dataset to perform zero-shot or few-shot inference on the benchmark model, thereby obtaining the initial score of the target dataset output by the benchmark model. Furthermore, by performing intervention training on the benchmark model based on parameter-efficient fine-tuning using the standard dataset, on the one hand, the performance of the benchmark model can be changed through the standard dataset to obtain the target model; on the other hand, the use of parameter-efficient fine-tuning technology to replace full fine-tuning makes it possible to perform dynamic evaluation of massive datasets one by one in terms of computing power cost, reducing the construction cost of the target model. Finally, zero-shot or few-shot inference is performed on the target model based on the target dataset to obtain the updated score of the target dataset. Then, the performance gain ratio of the benchmark model is calculated based on the initial score and the updated score, and the dataset to be evaluated is evaluated based on the performance gain ratio. In other words, the abstract concept of data quality is concretized into the rise and fall of model scores, which solves the problem of difficulty in quantifying the "scenario applicability" of the dataset, improves the scenario adaptability of the dataset evaluation, and improves the accuracy of the dataset quality evaluation.

[0019] The technical effects achieved by the second, third, and fourth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the steps of a dataset evaluation method provided in an embodiment of this application; Figure 2 A functional block diagram of a dataset evaluation device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0022] The following description sets forth many specific details to provide a full understanding of this application. The described embodiments are only some, not all, of the embodiments of this application.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0024] It should be further noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0025] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0026] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0027] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example.

[0028] Zero-shot reasoning: Enables AI models to perform reasoning, such as answering questions or solving problems, based solely on instructions and general knowledge, without having seen any specific task examples.

[0029] Baseline model: The core infrastructure in the field of AI. Simply put, it is a general-purpose model that has been pre-trained on massive amounts of data. It is like a "universal toolbox" that allows you to quickly develop various specialized AI applications without having to train each model from scratch.

[0030] Efficient parameter fine-tuning: During the training of deep learning models, certain parameters of the model are adjusted more precisely or efficiently to improve model performance or accelerate convergence. Common methods include using different optimizers (such as Adam, SGD, etc.), adjusting the learning rate, freezing some layers, and using gradient clipping.

[0031] Large language models refer to deep learning models trained on large amounts of text data, which enable the model to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics by being trained on large datasets [1]. The core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, and to a certain extent simulate the human language cognition and generation process.

[0032] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of an embodiment of the dataset evaluation method provided in this application. Depending on different requirements, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0033] It should be noted that the dataset evaluation method of this application embodiment can be applied to dataset quality evaluation scenarios, and its execution entity can be a dataset evaluation device. For example, after obtaining the dataset to be evaluated, the dataset to be evaluated can be performed on the dataset through the dataset evaluation device. Of course, the dataset evaluation method of this application embodiment can also be applied to other scenarios that require dataset quality evaluation, and this application does not specifically limit it in this regard.

[0034] The specific process of this embodiment is as follows: Figure 1 As shown, it includes the following steps: S101, preprocess the dataset to be evaluated to obtain the standard dataset.

[0035] Specifically, the preprocessing in this embodiment includes at least classification labels, which are used to label the data type of the dataset to be evaluated.

[0036] In some embodiments, a hybrid judgment method combining "keyword feature extraction" and "lightweight classification model" can be used to classify and label the dataset to be evaluated.

[0037] Specifically, for "keyword feature extraction," a high-frequency keyword database containing fields such as "medical, legal, code, and finance" can be established. The hit density of these words in the dataset to be evaluated is calculated. If the frequency of words such as "diagnosis" and "prescription" exceeds a threshold (e.g., 5%), the dataset to be evaluated is labeled as "medical."

[0038] For "lightweight classification models," a lightweight pre-trained classification model (such as BERT or FastText) can be used, or a large language model with a small number of parameters can be called for zero-shot classification. For example, the large language model can be fed the input "Please analyze the domain attributes of the following text fragments and select only one output from [General Knowledge, Medical, Legal, Programming]". The large language model randomly selects 100 samples from the dataset to be evaluated and performs the above inference, taking the mode as the final classification label for the dataset to be evaluated.

[0039] In some embodiments, preprocessing further includes metadata verification and noise removal cleaning; metadata verification is used to detect whether there are missing target fields or incorrectly formatted data in the dataset to be evaluated; noise removal cleaning is used to remove data to be cleaned.

[0040] Specifically, metadata validation can be performed as follows: The system predefines a standard JSON Schema rule file (e.g., a .schema file), which specifies the required field names (keys), field types (String / Int), and field value formats (e.g., source must be in URL format). The validation module loads the dataset to be tested (usually in JSONL or Parquet format) and scans the data line by line. A validator (e.g., Python's jsonschema library) compares the data with the rules. If a piece of data is missing a key field (e.g., missing answer) or has an incorrect format, the script automatically marks it as "Invalid" and removes it, or outputs an error log.

[0041] S102, construct the target dataset based on the standard dataset, wherein the data type of the target dataset is the same as that of the dataset to be evaluated, and the data content is isolated from that of the dataset to be evaluated.

[0042] In some embodiments, constructing a target dataset based on a standard dataset includes: randomly selecting a preset number of input data from the standard dataset; inputting the input data into a preset large language model, wherein the large language model outputs target data corresponding to the input data according to preset execution instructions; and constructing a target dataset based on the target data.

[0043] Specifically, 10% of the sample data is randomly selected from the standard dataset as a seed. This data is then read using a large language model (such as GPT-4 or Qwen-Max), and the instruction is executed: "Give 5 multiple-choice questions and their standard answers based on the content of this text." The generated questions and answers are then formatted and stored in a separate database as the target dataset for this evaluation.

[0044] In some embodiments, if the data type of the dataset to be evaluated is general and common industry, the target dataset can be obtained by automatically attaching publicly available authoritative test sets (such as C-Eval, GSM8K, MedQA) through the system's built-in mapping table.

[0045] In some embodiments, after constructing the target dataset based on the target data, the method further includes: removing the input data from the standard dataset to obtain the final dataset; and performing intervention training on the benchmark model based on the standard dataset using parameter-efficient fine-tuning, including: performing intervention training on the benchmark model based on the final dataset using parameter-efficient fine-tuning. This approach ensures data isolation between the target dataset and the final dataset, thereby ensuring the effectiveness of training the benchmark model using the final dataset.

[0046] S103, perform zero-shot or few-shot inference on the benchmark model based on the target dataset to obtain the initial score of the target dataset.

[0047] Specifically, the baseline model is an AI model that has not been trained on the dataset to be evaluated.

[0048] In some embodiments, zero-shot or few-shot inference is achieved by constructing a standardized Prompt template and inputting it into a baseline model.

[0049] S104. Based on the standard dataset, the baseline model is trained using intervention training with efficient parameter fine-tuning to obtain the target model.

[0050] In some embodiments, intervention training of a benchmark model based on a standard dataset for parameter-efficient fine-tuning includes: loading the weights of the benchmark model and freezing the main parameters of the benchmark model; injecting a low-rank matrix into the transport layer bypass of the benchmark model; inputting the standard dataset into the benchmark model and updating the parameters of the low-rank matrix using gradients based on the output of the benchmark model; and updating the weights based on the gradient-updated low-rank matrix.

[0051] Specifically, in order to reduce the computational cost of evaluation, this embodiment adopts efficient fine-tuning techniques such as low-rank adaptation (LoRA) or P-Tuning. By setting uniform hyperparameters (Learning Rate, Epochs, Batch Size), the training conditions are consistent when evaluating different datasets.

[0052] The function of each hyperparameter in this embodiment will be explained in detail below: Learning Rate: Determines the magnitude of each parameter update. Too large a rate leads to non-convergence of the loss function, while too small a rate results in slow training or an inability to escape local optima. In LoRA (Low-Rank Adaptation), it is typically set to 2e. 4 to 5e 4.

[0053] Epochs (training rounds): The number of times the dataset is fully learned by the model. One epoch may only give the model a rough overview; too many epochs can lead to overfitting and loss of generalization ability. Evaluation is typically set to 3 epochs.

[0054] Batch Size: The number of data samples included in each weight update. It determines the stability of the gradient. A larger batch size results in more stable training but consumes more GPU memory.

[0055] S105, perform zero-shot or few-shot inference on the target model based on the target dataset to obtain an updated score for the target dataset.

[0056] In some embodiments, the target model's built-in scoring script automatically compares the model output (Prediction) with the standard answer (Ground Truth). If they match, 1 point is awarded; otherwise, 0 points are awarded. The updated score is obtained by calculating the total score and the average accuracy.

[0057] S106, calculate the performance gain rate of the baseline model based on the initial score and the updated score.

[0058] In some embodiments, the performance gain rate is calculated according to the following formula: Where Gain is the performance gain rate, and Score is the performance gain rate. Tuned To update the score, Score Base This is the initial score.

[0059] Specifically, Score Base Score represents the average accuracy of the untrained baseline model on the test set (i.e., the target dataset).Tuned This represents the average accuracy of the model on the same test set after fine-tuning using the standard LoRA dataset.

[0060] S107, evaluate the dataset to be evaluated based on performance gain rate.

[0061] In some embodiments, if the performance gain rate is greater than a preset threshold, the dataset to be evaluated is a high-quality dataset; if the performance gain is greater than or equal to 0 and less than or equal to the preset threshold, the dataset to be evaluated is a normal dataset; if the performance gain is less than 0, the dataset to be evaluated is a low-quality dataset.

[0062] Compared with related technologies, the embodiments of this application have at least the following advantages: By classifying and labeling the dataset to be evaluated, the data type of the dataset to be evaluated can be accurately determined. Then, by constructing a target dataset based on a standard dataset, since the data type of the target dataset is the same as that of the dataset to be evaluated and the data content is isolated from the dataset to be evaluated, the target dataset can be used as a test dataset to perform zero-shot or few-shot inference on the benchmark model, thereby obtaining the initial score of the target dataset output by the benchmark model. Furthermore, by performing intervention training on the benchmark model based on parameter-efficient fine-tuning using the standard dataset, on the one hand, the performance of the benchmark model can be changed through the standard dataset to obtain the target model; on the other hand, the use of parameter-efficient fine-tuning technology to replace full fine-tuning makes it possible to perform dynamic evaluation of massive datasets one by one in terms of computing power cost, reducing the construction cost of the target model. Finally, zero-shot or few-shot inference is performed on the target model based on the target dataset to obtain the updated score of the target dataset. Then, the performance gain ratio of the benchmark model is calculated based on the initial score and the updated score, and the dataset to be evaluated is evaluated based on the performance gain ratio. In other words, the abstract concept of data quality is concretized into the rise and fall of model scores, which solves the problem of difficulty in quantifying the "scenario applicability" of the dataset, improves the scenario adaptability of the dataset evaluation, and improves the accuracy of the dataset quality evaluation.

[0063] Based on the same idea as the dataset evaluation method in the above embodiments, this application also provides a dataset evaluation apparatus, which can be used to execute the above dataset evaluation method. For ease of explanation, the structural schematic diagram of the dataset evaluation apparatus embodiment only shows the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the apparatus, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0064] like Figure 2As shown, the dataset evaluation device 20 includes a preprocessing module 201, a construction module 202, an inference module 203, an intervention training module 204, a calculation module 205, and an evaluation module 206. In some embodiments, the above modules can be programmable software instructions stored in memory and executable by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware embedded in the processor.

[0065] The preprocessing module 201 is used to preprocess the dataset to be evaluated to obtain a standard dataset, wherein the preprocessing includes at least classification labels, which are used to label the data type of the dataset to be evaluated; The construction module 202 is used to construct a target dataset based on the standard dataset, wherein the data type of the target dataset is the same as that of the dataset to be evaluated, and the data content is isolated from that of the dataset to be evaluated; The inference module 203 is used to perform zero-shot or few-shot inference on the benchmark model based on the target dataset to obtain an initial score for the target dataset, wherein the benchmark model is an AI model that has not been trained on the dataset to be evaluated. Intervention training module 204 is used to perform intervention training on the benchmark model based on the standard dataset to efficiently fine-tune the parameters and obtain the target model; The inference module 203 is also used to perform zero-shot inference or few-shot inference on the target model based on the target dataset to obtain an updated score for the target dataset; Calculation module 205 is used to calculate the performance gain rate of the benchmark model based on the initial score and the updated score; Evaluation module 206 is used to evaluate the dataset to be evaluated based on the performance gain rate.

[0066] The dataset evaluation device 20 provided in the above embodiments can implement the technical solutions described in the above dataset evaluation method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above dataset evaluation method embodiments, and will not be repeated here.

[0067] Please refer to Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the electronic device of this application. In this embodiment of the invention, the electronic device 300 includes a processor 301, a memory 302, and a display 303. Figure 3 Only some components of the electronic device 300 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0068] In some embodiments, processor 301 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 302 or process data, such as the dataset evaluation method of the present invention.

[0069] In some embodiments, processor 301 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 301 may be local or remote. In some embodiments, processor 301 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0070] In some embodiments, memory 302 may be an internal storage unit of electronic device 300, such as a hard disk or memory of electronic device 300. In other embodiments, memory 302 may also be an external storage device of electronic device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 300.

[0071] Furthermore, the memory 302 may include both internal storage units of the electronic device 300 and external storage devices. The memory 302 is used to store application software and various types of data installed on the electronic device 300.

[0072] In some embodiments, display 303 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 303 is used to display information from electronic device 300 and to display visual user applications. Components 301-303 of electronic device 300 communicate with each other via a system bus.

[0073] In one embodiment, when processor 301 executes the dataset evaluation program in memory 302, the following steps can be performed: The dataset to be evaluated is preprocessed to obtain a standard dataset, wherein the preprocessing includes at least classification labels, which are used to label the data type of the dataset to be evaluated; A target dataset is constructed based on the standard dataset, wherein the data type of the target dataset is the same as that of the dataset to be evaluated, and the data content of the target dataset is isolated from that of the dataset to be evaluated; Based on the target dataset, zero-shot or few-shot inference is performed on the benchmark model to obtain an initial score for the target dataset, wherein the benchmark model is an AI model that has not been trained on the dataset to be evaluated. The benchmark model is trained using parameter-efficient fine-tuning based on the standard dataset to obtain the target model; Based on the target dataset, zero-shot or few-shot inference is performed on the target model to obtain an updated score for the target dataset; The performance gain ratio of the benchmark model is calculated based on the initial score and the updated score, and the dataset to be evaluated is evaluated based on the performance gain ratio.

[0074] It should be understood that when the processor 301 executes the dataset evaluation program in the memory 302, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0075] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 300 mentioned. Electronic device 300 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 300 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0076] Accordingly, this application also provides a computer-readable storage medium for storing a computer-readable program or instruction. When the program or instruction is executed by a processor, it can implement the steps or functions of the dataset evaluation method provided in the above-described method embodiments.

[0077] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0078] The above provides a detailed description of the dataset evaluation method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A dataset evaluation method, characterized in that, include: The dataset to be evaluated is preprocessed to obtain a standard dataset, wherein the preprocessing includes at least classification labels, which are used to label the data type of the dataset to be evaluated; A target dataset is constructed based on the standard dataset, wherein the data type of the target dataset is the same as that of the dataset to be evaluated, and the data content of the target dataset is isolated from that of the dataset to be evaluated; Based on the target dataset, zero-shot or few-shot inference is performed on the benchmark model to obtain an initial score for the target dataset, wherein the benchmark model is an AI model that has not been trained on the dataset to be evaluated. The benchmark model is trained using parameter-efficient fine-tuning based on the standard dataset to obtain the target model; Based on the target dataset, zero-shot or few-shot inference is performed on the target model to obtain an updated score for the target dataset; The performance gain ratio of the benchmark model is calculated based on the initial score and the updated score, and the dataset to be evaluated is evaluated based on the performance gain ratio.

2. The dataset evaluation method according to claim 1, characterized in that, The step of calculating the performance gain ratio of the benchmark model based on the initial score and the updated score includes: The performance gain rate is calculated using the following formula: ； Where Gain is the performance gain rate, and Score is... Tuned For the updated score, Score Base The initial score is given.

3. The dataset evaluation method according to claim 2, characterized in that, The evaluation of the dataset to be evaluated based on the performance gain ratio includes: If the performance gain rate is greater than a preset threshold, the dataset to be evaluated is deemed to be a high-quality dataset. If the performance gain is greater than or equal to 0 and less than or equal to the preset threshold, the dataset to be evaluated is considered a normal dataset. If the performance gain is less than 0, the dataset to be evaluated is considered a low-quality dataset.

4. The dataset evaluation method according to claim 1, characterized in that, The intervention training of the benchmark model based on the standard dataset for efficient parameter fine-tuning includes: Load the weights of the baseline model and freeze the main parameters of the baseline model; A low-rank matrix is ​​injected into the transport layer bypass of the baseline model; The standard dataset is input into the benchmark model, and the parameters of the low-rank matrix are updated by gradient based on the output of the benchmark model. The weights are updated based on the low-rank matrix after gradient update.

5. The dataset evaluation method according to claim 1, characterized in that, The construction of the target dataset based on the standard dataset includes: A preset number of input data are randomly selected from the standard dataset; The input data is input into a preset large language model, and the large language model outputs target data corresponding to the input data according to preset execution instructions; The target dataset is constructed based on the target data.

6. The dataset evaluation method according to claim 5, characterized in that, After constructing the target dataset based on the target data, the method further includes: The input data is removed from the standard dataset to obtain the final dataset; The intervention training of the benchmark model based on the standard dataset for efficient parameter fine-tuning includes: The baseline model is trained using parameter-efficient fine-tuning based on the final dataset.

7. The dataset evaluation method according to any one of claims 1 to 6, characterized in that, The preprocessing also includes metadata verification and noise removal / cleaning; The metadata verification is used to detect whether there is missing target fields or incorrectly formatted data in the dataset to be evaluated; The noise removal cleaning is used to remove the data to be cleaned.

8. A dataset evaluation device, characterized in that, include: The module includes a preprocessing module, a construction module, an inference module, an intervention training module, a computation module, and an evaluation module. The preprocessing module is used to preprocess the dataset to be evaluated to obtain a standard dataset. The preprocessing includes at least classification labels, which are used to label the data type of the dataset to be evaluated. The construction module is used to construct a target dataset based on the standard dataset, wherein the data type of the target dataset is the same as that of the dataset to be evaluated, and the data content is isolated from that of the dataset to be evaluated; The inference module is used to perform zero-shot or few-shot inference on the benchmark model based on the target dataset to obtain an initial score for the target dataset, wherein the benchmark model is an AI model that has not been trained on the dataset to be evaluated. The intervention training module is used to perform intervention training on the benchmark model based on the standard dataset to efficiently fine-tune the parameters, thereby obtaining the target model; The inference module is also used to perform zero-shot or few-shot inference on the target model based on the target dataset to obtain an updated score for the target dataset; The calculation module is used to calculate the performance gain rate of the benchmark model based on the initial score and the updated score; The evaluation module is used to evaluate the dataset to be evaluated based on the performance gain rate.

9. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the electronic device to execute the dataset evaluation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the dataset evaluation method as described in any one of claims 1 to 7.

Images