Quality evaluation method and system for AI large model text training data set

Abstract

The invention discloses a quality evaluation method and system for an AI large model text training data set. The method comprises the steps that the large model text training data set is preprocessed; setting an evaluation index system, and defining a threshold value and a weight of each index; setting a data set sampling rule, and extracting a data set; counting preset data indexes; performing prejudice and compliance evaluation on the data set by using a large model through cue words and evaluation specifications; identifying illegal toxic contents in the data set according to evaluation specifications by utilizing cooperation of a text mining technology and a large model; and calculating a text confusion degree of the data set based on the BERT model, and calculating a confusion degree evaluation result according to the evaluation specification. Through the technical scheme of the invention, invalid training and influence on large model capability output caused by a data set problem are avoided, the AI large model training efficiency and the model effect are effectively improved, meanwhile, the large model training cost is greatly reduced, and the evaluation effect is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition analysis, and in particular to a quality assessment method for an AI large model text training data set and a quality assessment system for an AI large model text training data set. Background Art

[0002] With the rapid development of AI big model technology, the training of AI big models is increasingly dependent on large-scale, high-quality data sets. However, defects such as noise, duplicate records, missing values, and data annotation errors in the data set will seriously affect the training effect and evaluation accuracy of the model. These problems will cause various problems in the model training process, such as overfitting and underfitting. High-quality data sets can improve the accuracy and interpretability of the model and reduce the training time.

[0003] Different from traditional formatted data, AI large model training data sets are mainly unformatted data. In the process of data quality assessment, in addition to the requirements for data accuracy, there are also requirements for data diversity, data balance, data redundancy, and data labeling quality, which makes data set quality assessment more difficult.

[0004] In addition, many current methods involve conducting post-validation evaluation after training. This approach will incur more training resource overhead and more additional workload. How to place more quality assessments in advance has become a key difficulty in the model training process.

[0005] Therefore, it is very important to evaluate the quality of the training data set. Evaluating the quality of the training data set for large AI models mainly faces the following difficulties:

[0006] (1) The scale of data set evaluation is huge. The training data set of the model often contains hundreds of millions of data samples. Such a huge amount of data makes it difficult to evaluate the data quality manually.

[0007] (2) Diversity of dataset formats: datasets may come from a variety of different sources and channels, and the formats, standards, and quality of data vary, increasing the difficulty of evaluation;

[0008] (3) AI large model training datasets contain a large amount of unformatted text datasets, which may contain erroneous, repeated, or contradictory information. These noise data will interfere with the model training process, and it is difficult to identify errors and repetitions in text datasets.

[0009] (4) There is often some illegal information in the text data set. Checking for illegal information is also a key part of the quality assessment of AI large model data sets.

[0010] (5) In the application scenarios of large AI models, in addition to traditional requirements such as accuracy and consistency, data quality requirements also include data bias and fairness requirements. Biases in the data set may cause the model to produce unfair results when processing certain tasks, and detecting and correcting these biases is a challenge;

[0011] (6) Dynamic changes: Data sets may change over time, and new data may be inconsistent with the original data distribution. Quality assessment methods need to adapt to such changes. Summary of the invention

[0012] In response to the above problems, the present invention provides a quality assessment method and system for an AI large model text training data set. By providing a comprehensive data set quality assessment system and establishing a pre-data set assessment method, it is possible to comprehensively and comprehensively evaluate key data quality issues in the large model training process. By providing large models and text mining to evaluate data set quality capabilities, it is possible to effectively improve the efficiency of AI large model training and model effects, while significantly reducing the cost of large model training. Through data set statistical feature analysis, data bias and compliance assessment, and text perplexity assessment, the assessment cost is controllable, and the assessment effect is also relatively significant, which can quickly guide data set developers to optimize the data set.

[0013] To achieve the above object, the present invention provides a quality assessment method for an AI large model text training data set, comprising:

[0014] Preprocess the large model text training dataset to be evaluated;

[0015] Set up an evaluation indicator system and define the threshold and weight of each indicator to form evaluation standards;

[0016] Setting a data set sampling rule, and sampling and extracting a data set according to the sampling rule;

[0017] Counting the preset data indicators of the data set and recording the indicator statistical results;

[0018] Using the pre-written prompt words and the evaluation specifications, the existing large model is used to evaluate the bias and compliance of the data set, and the bias and compliance evaluation results are recorded;

[0019] Using text mining technology in collaboration with the big model, identifying illegal toxic content in the data set according to the evaluation specification, and recording toxicity evaluation results;

[0020] The text perplexity of the data set is calculated based on the BERT model, and the perplexity evaluation result is calculated according to the evaluation specification.

[0021] In the above technical solution, preferably, the quality assessment method of the AI ​​large model text training data set also includes:

[0022] Determine whether the quality assessment system is a comprehensive assessment or a focused assessment based on the type of data set;

[0023] A comprehensive evaluation is conducted on Internet datasets, document datasets, and book datasets, and data indicator statistics and text perplexity calculations are performed on internal business system datasets.

[0024] In the above technical solution, preferably, the quality assessment method of the AI ​​large model text training data set also includes:

[0025] Providing corresponding quality improvement suggestions for each of the data sets based on the indicator statistical results, bias and compliance assessment results, illegal toxicity assessment results, and perplexity assessment results of the data sets;

[0026] For data sets whose word repetition rate, special character ratio, and text similarity exceed the preset threshold, direct feedback on improvement points is provided;

[0027] For datasets with bias and toxic data, provide a method to detoxify the dataset;

[0028] For data sets whose perplexity exceeds a preset threshold, a method to reduce the perplexity is provided.

[0029] In the above technical solution, preferably, the indicators of the evaluation indicator system include statistical data set indicator thresholds, data set bias and compliance assessment thresholds, illegal toxicity assessment thresholds and text perplexity mean indicator thresholds.

[0030] In the above technical solution, preferably, the preset data indicators of the data set include word repetition rate, special character ratio and text similarity.

[0031] The present invention also proposes a quality assessment system for an AI large model text training data set, which applies the quality assessment method for an AI large model text training data set disclosed in any one of the above technical solutions, including:

[0032] The dataset preprocessing module is used to preprocess the large model text training dataset to be evaluated;

[0033] The evaluation specification definition module is used to set up the evaluation indicator system and define the threshold and weight of each indicator to form the evaluation specification;

[0034] A data sampling and extraction module, used to set a data set sampling rule and sample and extract a data set according to the sampling rule;

[0035] A data indicator statistics module is used to count the preset data indicators of the data set and record the indicator statistics results;

[0036] A data bias assessment module, for performing bias and compliance assessment on the data set using an existing large model using pre-written prompt words and the assessment specification, and recording bias and compliance assessment results;

[0037] A data toxicity assessment module, used to utilize text mining technology in collaboration with a large model to identify illegal toxic content in the data set according to the assessment specification, and record toxicity assessment results;

[0038] A text perplexity evaluation module is used to calculate the text perplexity of the data set based on the BERT model, and calculate the perplexity evaluation result according to the evaluation specification.

[0039] In the above technical solution, preferably, the quality assessment system of the AI ​​large model text training data set also includes an assessment system determination module, which is specifically used to:

[0040] Determine whether the quality assessment system is a comprehensive assessment or a focused assessment based on the type of data set;

[0041] A comprehensive evaluation is conducted on Internet datasets, document datasets, and book datasets, and data indicator statistics and text perplexity calculations are performed on internal business system datasets.

[0042] In the above technical solution, preferably, the quality assessment system of the AI ​​large model text training data set also includes a data improvement suggestion module, which is specifically used to:

[0043] Providing corresponding quality improvement suggestions for each of the data sets based on the indicator statistical results, bias and compliance assessment results, illegal toxicity assessment results, and perplexity assessment results of the data sets;

[0044] For data sets whose word repetition rate, special character ratio, and text similarity exceed the preset threshold, direct feedback on improvement points is provided;

[0045] For datasets with bias and toxic data, provide a method to detoxify the dataset;

[0046] For data sets whose perplexity exceeds a preset threshold, a method to reduce the perplexity is provided.

[0047] In the above technical solution, preferably, the indicators of the evaluation indicator system include statistical data set indicator thresholds, data set bias and compliance assessment thresholds, illegal toxicity assessment thresholds and text perplexity mean indicator thresholds.

[0048] In the above technical solution, preferably, the preset data indicators of the data set include word repetition rate, special character ratio and text similarity.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows: by providing a comprehensive data set quality assessment system and establishing a pre-data set assessment method, the key data quality issues in the large model training process are comprehensively and comprehensively evaluated; by providing large models and text mining to evaluate data set quality capabilities, the efficiency of AI large model training and model effect are effectively improved, while the cost of large model training is greatly reduced; through data set statistical feature analysis, data bias and compliance assessment, and text perplexity assessment, the assessment cost is controllable, the assessment effect is significantly improved, and the data set developers can be quickly guided to optimize the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a quality assessment method for a large AI model text training data set disclosed in an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the modules of a quality assessment system for an AI large model text training data set disclosed in an embodiment of the present invention.

[0052] In the figure, the corresponding relationship between each component and the reference numeral is as follows:

[0053] 1. Data set preprocessing module, 2. Evaluation specification definition module, 3. Data sampling and extraction module, 4. Data indicator statistics module, 5. Data bias assessment module, 6. Data toxicity assessment module, 7. Text confusion assessment module, 8. Evaluation system determination module, 9. Data improvement suggestion module. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0056] like Figure 1 As shown, a quality assessment method for a large AI model text training data set provided by the present invention includes:

[0057] Preprocess the large model text training dataset to be evaluated;

[0058] Set up an evaluation indicator system and define the threshold and weight of each indicator to form evaluation standards;

[0059] Set the data set sampling rules and extract the data set by sampling according to the sampling rules;

[0060] Count the preset data indicators of the statistical data set and record the statistical results of the indicators;

[0061] Use pre-written prompts and evaluation specifications to evaluate the bias and compliance of the data set using existing large models, and record the bias and compliance evaluation results;

[0062] Use text mining technology to collaborate with large models to identify illegal toxic content in the data set according to the assessment specifications and record the toxicity assessment results;

[0063] The text perplexity of the dataset is calculated based on the BERT model, and the perplexity evaluation results are calculated according to the evaluation specifications.

[0064] In this implementation, by providing a comprehensive data set quality assessment system and establishing a pre-data set assessment method, the key data quality issues in the large model training process are comprehensively and comprehensively evaluated, thereby guiding data set developers to improve and optimize the data set before model training, avoiding invalid training due to data set problems and affecting the output of large model capabilities. By providing large models and text mining to evaluate data set quality capabilities, the efficiency of AI large model training and model effects are effectively improved, while the cost of large model training is greatly reduced. Through data set statistical feature analysis, data bias and compliance assessment, and text perplexity assessment, the assessment cost is controllable, the assessment effect is significantly improved, and the data set developers can quickly guide data set optimization.

[0065] Specifically, statistical characteristic analysis includes word frequency statistics, sentence length statistics, etc. Through statistical characteristic analysis, we can understand the distribution and diversity of the data set and evaluate the richness and balance of the data set, including:

[0066] Character repetition rate: Count the character repetition rate in the text dataset. If the character repetition rate is too high, it means that the dataset is not diverse enough.

[0067] Special character ratio: Special characters often do not contain substantive knowledge content. If there are too many special characters in the training data set, it also means that the quality of the data set is not high.

[0068] Text similarity: Count the text similarities in the training data set. If the similarity is too high, it means that the data set is not diverse enough. Data sets with too high similarity may lead to overfitting problems in model training.

[0069] Bias and compliance assessments include representativeness analysis, sensitive word detection, etc., including: data set gender, race, occupation, age, social status, physical ability, regional bias assessment and toxic content assessment. Through bias and fairness assessments, the model can avoid bias and discrimination against specific groups and improve the fairness and reliability of the model; through compliance checks, illegal information can be prevented from participating in training and serious impacts on the performance, reliability and ethics of the model can be avoided. Specifically include:

[0070] Dataset bias assessment: Use AI big models to assess the bias of data sets, including gender, race, occupation, age, social status, physical ability, regional bias, etc.

[0071] Compliance assessment: Once illegal, toxic or false data sets are involved in model training, large language models may produce offensive, harmful or misleading content. The toxicity of data sets can be identified and assessed through text analysis and mining combined with AI large model capabilities.

[0072] Text perplexity evaluation is an indicator used to evaluate the quality of language models in natural language processing. It measures the difficulty of the model in predicting a given text. The lower the perplexity, the more accurate the model's prediction of the text, that is, the better the model's adaptability to the text; the higher the perplexity, the more difficult it is for the model to predict the text, and the worse the model's adaptability.

[0073] The perplexity calculation based on BERT includes: converting text into a digital representation that the model can understand, namely embeddings. BERT uses word embeddings, position embeddings, and segment embeddings to represent the input text. The encoded text is input into the BERT model to calculate the perplexity.

[0074] In the above implementation, preferably, the quality assessment method of the AI ​​large model text training data set also includes:

[0075] Determine whether the quality assessment system is a comprehensive assessment or a focused assessment based on the type of data set;

[0076] A comprehensive evaluation is conducted on Internet datasets, document datasets, and book datasets, and data indicator statistics and text perplexity calculations are performed on internal business system datasets.

[0077] In the above implementation, preferably, the quality assessment method of the AI ​​large model text training data set also includes:

[0078] Provide corresponding quality improvement suggestions for each data set based on the indicator statistics, bias and compliance assessment results, illegal toxicity assessment results, and perplexity assessment results of the data set;

[0079] For data sets whose word repetition rate, special character ratio, and text similarity exceed the preset threshold, direct feedback on improvement points is provided;

[0080] For datasets with bias and toxic data, provide a method to detoxify the dataset;

[0081] For data sets whose perplexity exceeds a preset threshold, a method to reduce the perplexity is provided.

[0082] like Figure 2 As shown, the present invention also proposes a quality assessment system for an AI large model text training data set, which applies the quality assessment method for an AI large model text training data set disclosed in any one of the above embodiments, including:

[0083] Dataset preprocessing module 1, used to preprocess the large model text training dataset to be evaluated;

[0084] Evaluation specification definition module 2 is used to set the evaluation indicator system and define the threshold and weight of each indicator to form the evaluation specification;

[0085] The data sampling and extraction module 3 is used to set the data set sampling rules and sample and extract the data set according to the sampling rules;

[0086] The data indicator statistics module 4 is used to count the preset data indicators of the statistical data set and record the indicator statistical results;

[0087] Data bias assessment module 5, used to use pre-written prompt words and assessment specifications to perform bias and compliance assessment on the data set using the existing large model, and record the bias and compliance assessment results;

[0088] Data toxicity assessment module 6, used to use text mining technology to collaborate with the big model, identify illegal toxic content in the data set according to the assessment specifications, and record the toxicity assessment results;

[0089] The text perplexity evaluation module 7 is used to calculate the text perplexity of the data set based on the BERT model, and calculate the perplexity evaluation result according to the evaluation specification.

[0090] In the above implementation, preferably, the quality assessment system of the AI ​​large model text training data set also includes an assessment system determination module 8, which is specifically used to:

[0091] Determine whether the quality assessment system is a comprehensive assessment or a focused assessment based on the type of data set;

[0092] A comprehensive evaluation is conducted on Internet datasets, document datasets, and book datasets, and data indicator statistics and text perplexity calculations are performed on internal business system datasets.

[0093] In the above implementation, preferably, the quality assessment system of the AI ​​large model text training data set also includes a data improvement suggestion module 9, which is specifically used to:

[0094] Provide corresponding quality improvement suggestions for each data set based on the indicator statistics, bias and compliance assessment results, illegal toxicity assessment results, and perplexity assessment results of the data set;

[0095] For data sets whose word repetition rate, special character ratio, and text similarity exceed the preset threshold, direct feedback on improvement points is provided;

[0096] For datasets with bias and toxic data, provide a method to detoxify the dataset;

[0097] For data sets whose perplexity exceeds a preset threshold, a method to reduce the perplexity is provided.

[0098] In the above implementation, preferably, the indicators of the evaluation indicator system include a statistical data set indicator threshold, a data set bias and compliance assessment threshold, an illegal toxicity assessment threshold and a text perplexity mean indicator threshold.

[0099] In the above implementation, preferably, the preset data indicators of the data set include word repetition rate, special character ratio and text similarity.

[0100] According to the quality assessment system of the AI ​​large model text training data set disclosed in the above-mentioned implementation mode, the functions to be implemented by each module thereof correspond to the respective steps of the quality assessment method of the AI ​​large model text training data set disclosed in the above-mentioned implementation mode. During the implementation process, the operations are performed with reference to the above-mentioned implementation mode, which will not be repeated here.

[0101] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A quality assessment method for a large AI model text training data set, characterized in that: include: Preprocess the large model text training dataset to be evaluated; Set up an evaluation indicator system and define the threshold and weight of each indicator to form evaluation standards; Setting a data set sampling rule, and sampling and extracting a data set according to the sampling rule; Counting the preset data indicators of the data set and recording the indicator statistical results; Using the pre-written prompt words and the evaluation specifications, the existing large model is used to evaluate the bias and compliance of the data set, and the bias and compliance evaluation results are recorded; Using text mining technology in collaboration with the big model, identifying illegal toxic content in the data set according to the evaluation specification, and recording toxicity evaluation results; The text perplexity of the data set is calculated based on the BERT model, and the perplexity evaluation result is calculated according to the evaluation specification.

2. The quality assessment method of the AI ​​large model text training data set according to claim 1 is characterized in that: Also includes: Determine whether the quality assessment system is a comprehensive assessment or a focused assessment based on the type of data set; A comprehensive evaluation is conducted on Internet datasets, document datasets, and book datasets, and data indicator statistics and text perplexity calculations are performed on internal business system datasets.

3. The quality assessment method of the AI ​​large model text training data set according to claim 1 or 2, characterized in that: Also includes: Providing corresponding quality improvement suggestions for each of the data sets based on the indicator statistics, bias and compliance assessment results, illegal toxicity assessment results, and perplexity assessment results of the data sets; For data sets whose word repetition rate, special character ratio, and text similarity exceed the preset threshold, direct feedback on improvement points is provided; For datasets with bias and toxic data, provide a method to detoxify the dataset; For data sets whose perplexity exceeds a preset threshold, a method to reduce the perplexity is provided.

4. The quality assessment method of the AI ​​large model text training data set according to claim 3 is characterized in that: The indicators of the evaluation indicator system include statistical data set indicator threshold, data set bias and compliance assessment threshold, illegal toxicity assessment threshold and text perplexity mean indicator threshold.

5. The quality assessment method of the AI ​​large model text training data set according to claim 4 is characterized in that: The preset data indicators of the data set include word repetition rate, special character ratio and text similarity.

6. A quality assessment system for AI large model text training data set, characterized by: The method for evaluating the quality of an AI large model text training data set according to any one of claims 1 to 5 comprises: The dataset preprocessing module is used to preprocess the large model text training dataset to be evaluated; The evaluation specification definition module is used to set up the evaluation indicator system and define the threshold and weight of each indicator to form the evaluation specification; A data sampling and extraction module, used to set a data set sampling rule and sample and extract a data set according to the sampling rule; A data indicator statistics module is used to count the preset data indicators of the data set and record the indicator statistics results; A data bias assessment module, for performing bias and compliance assessment on the data set using an existing large model using pre-written prompt words and the assessment specification, and recording bias and compliance assessment results; A data toxicity assessment module, used to utilize text mining technology in collaboration with a large model to identify illegal toxic content in the data set according to the assessment specification, and record toxicity assessment results; A text perplexity evaluation module is used to calculate the text perplexity of the data set based on the BERT model, and calculate the perplexity evaluation result according to the evaluation specification.

7. The quality assessment system for AI large model text training data set according to claim 6 is characterized in that: It also includes an assessment system determination module, which is specifically used to: Determine whether the quality assessment system is a comprehensive assessment or a focused assessment based on the type of data set; A comprehensive evaluation is conducted on Internet datasets, document datasets, and book datasets, and data indicator statistics and text perplexity calculations are performed on internal business system datasets.

8. The quality assessment system for AI large model text training data set according to claim 6 or 7, characterized in that: It also includes a data improvement suggestion module, which is specifically used to: Providing corresponding quality improvement suggestions for each of the data sets based on the indicator statistics, bias and compliance assessment results, illegal toxicity assessment results, and perplexity assessment results of the data sets; For data sets whose word repetition rate, special character ratio, and text similarity exceed the preset threshold, direct feedback on improvement points is provided; For datasets with bias and toxic data, provide a method to detoxify the dataset; For data sets whose perplexity exceeds a preset threshold, a method to reduce the perplexity is provided.

9. The quality assessment system for AI large model text training data set according to claim 8, characterized in that: The indicators of the evaluation indicator system include statistical data set indicator threshold, data set bias and compliance assessment threshold, illegal toxicity assessment threshold and text perplexity mean indicator threshold.

10. The quality assessment system for AI large model text training data set according to claim 9, characterized in that: The preset data indicators of the data set include word repetition rate, special character ratio and text similarity.

Images