Large language model pre-training low-quality data evaluation method

Abstract

The invention relates to a large language model pre-training low-quality data evaluation method, which comprises the following steps of: obtaining to-be-evaluated data to obtain a sampling sample; the method comprises the following steps of: scoring a sampling sample by adopting a large language model, sampling at a certain proportion, and manually marking sampled data to obtain a low-quality data index; firstly, to-be-evaluated data is evaluated according to a heuristic index, a corresponding heuristic rule is adopted to obtain a heuristic rule score, the score is divided into preliminary low-quality data and preliminary high-quality data, and a preliminary low-quality data proportion is obtained; secondly, evaluating the preliminary high-quality data according to a non-heuristic index, dividing the preliminary high-quality data into final low-quality data and final high-quality data based on dimensions defined for the large language model, and calculating the total proportion of the low-quality data in the to-be-evaluated data; compared with the prior art, the method has the advantages of efficiently and accurately analyzing low-quality data in the large language model data and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of large language model data, and in particular to a method for evaluating low-quality data for pre-training of a large language model. Background Art

[0002] A large proportion of the pre-training data for large language models comes from Internet crawling, such as Common Crawl. However, there is a certain proportion of low-quality (noise) data on the Internet, such as toxic content, advertising content, garbled content, etc. If such low-quality data is directly used to train large prediction models, the model performance will be greatly affected. At present, the difficulty of evaluating the data quality of large language models is relatively large, which is mainly reflected in the following aspects: large data volume, many types of low-quality questions, vague definitions, many subjective components, scattered features, etc.

[0003] Many existing technologies evaluate the statistical information and label information of the data set, and do not involve the definition and evaluation of low-quality indicators in the data. It is still difficult to obtain low-quality data in large language model data efficiently and accurately. Summary of the invention

[0004] The purpose of the present invention is to provide a large language model pre-training low-quality data evaluation method in order to efficiently and accurately analyze low-quality data in large language model data.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for evaluating low-quality data for pre-training of a large language model, the method comprising the following steps:

[0007] Acquire the data to be evaluated, and sample the data to be evaluated to obtain a sample;

[0008] The sampled samples are scored using a large language model, and the scored data are divided into high-quality data and low-quality data. The high-quality data and the low-quality data are sampled according to a certain ratio, and the sampled data are manually labeled to obtain low-quality data indicators, which include heuristic indicators and non-heuristic indicators.

[0009] First, the data to be evaluated is evaluated according to the heuristic indicators, and the corresponding heuristic rules are used to obtain the heuristic rule scores. Based on the heuristic rule scores, the data to be evaluated is divided into preliminary low-quality data and preliminary high-quality data, and the proportion of preliminary low-quality data is obtained;

[0010] Then, the preliminary high-quality data is evaluated according to non-heuristic indicators, and the preliminary high-quality data is divided into final low-quality data and final high-quality data based on the dimensions defined for the large language model, and the total proportion of low-quality data in the data to be evaluated is calculated.

[0011] Furthermore, the use of a large language model to score the sampled samples is specifically as follows:

[0012] The sample was scored on eight dimensions: invalidity, incompleteness, incomprehensibility, dissimilarity, unfluency, irrelevance, insecurity, and others.

[0013] Furthermore, the number of the sampling samples is:

[0014]

[0015] Where n is the number of samples taken, Z is the Z score which is the standard deviation of the normal distribution, p is the probability of success in the population, and E is the margin of error.

[0016] Furthermore, the low-quality data indicators include validity, relevance, completeness, comprehensibility, similarity, fluency and security.

[0017] Furthermore, the heuristic indicator includes effectiveness.

[0018] Furthermore, when the heuristic indicator is effectiveness, the corresponding heuristic rule is used to obtain the heuristic rule score as follows:

[0019] Remove the leading space symbol from the data to be evaluated, and then calculate the string length of the data to be evaluated. If the string length is 0, give the score corresponding to the low-quality data.

[0020] Furthermore, based on the dimensions defined for the large language model, the sampled preliminary high-quality data is divided into final low-quality data and final high-quality data as follows:

[0021] The preliminary high-quality data is sampled and divided into final low-quality data and final high-quality data based on the seven dimensions defined for the large language model. The seven dimensions and the corresponding scoring results are:

[0022] Validity: Check whether the data is valid. If the content is all carriage returns or spaces or contains garbled Chinese and English characters exceeding the character count threshold or meaningless content, a low quality score will be given. Otherwise, a high quality score will be given.

[0023] Relevance: Checks whether the data contains irrelevant information, and gives a final low quality score if it does not. The irrelevant information includes references to authors, journalists, sources, publishers, imports, copyrights, notices, references, dates, and calls to action at the beginning or end. Otherwise, a high quality score is given;

[0024] Completeness: Check whether the data is complete. If not, a low quality score is given. Otherwise, a high quality score is given.

[0025] Understandability: Check whether the data is understandable. If the formulas, tables, and titles in Markdown format are correctly converted to Markdown text and there are no incomprehensible special characters, then the data is understandable and a high quality score is given. Otherwise, a low quality score is given.

[0026] Similarity: Check whether the data contains similar content that reaches the maximum ratio threshold. If so, a low quality score is given, otherwise a high quality score is given;

[0027] Fluency: Check whether the data is fluent in terms of logic and format. If the sentence includes unreasonable punctuation, a low quality score will be given, otherwise a high quality score will be given;

[0028] Security: Check whether the data contains any toxic, harmful or illegal information. If so, a low quality score is given; otherwise, a high quality score is given;

[0029] For a piece of data in the sampled preliminary high-quality data, if the scoring results of the seven dimensions are all final high-quality scores, then the data is intermediate high-quality data, otherwise it is intermediate low-quality data. Based on the proportion of intermediate low-quality data in the sampled preliminary high-quality data, all preliminary high-quality data are divided into final low-quality data and final high-quality data.

[0030] Furthermore, the total proportion of low-quality data in the data to be evaluated is calculated as follows:

[0031] Assuming that the total amount of data to be evaluated is X, after the heuristic indicator evaluation, we get Y1 pieces of preliminary low-quality data and X-Y1 pieces of preliminary high-quality data, and get the proportion of preliminary low-quality data m ii %;

[0032] Assume that the number of intermediate low-quality data is Y2, the number of intermediate high-quality data is S-Y2, and the proportion of intermediate low-quality data in the sampled preliminary high-quality data is m ij %;

[0033] Based on the proportion of preliminary low-quality data m ii % and m ij % Calculate the total proportion of low-quality data in the data to be evaluated.

[0034] Furthermore, the total proportion of the low-quality data in the data to be evaluated is:

[0035]

[0036] Furthermore, the data to be evaluated is sampled to obtain the sampled samples:

[0037] The data to be evaluated is stratified according to its data distribution to obtain sampling samples.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention mines out low-quality indicators of the model to be evaluated through a large language model, so that the subsequent use of heuristic rules and large language models for low-quality data identification has stronger directionality, reducing the calculation time while efficiently and accurately obtaining accurate low-quality data in the large language model data. In addition, by defining 7 discriminant dimensions of low-quality data, the coverage of judging low-quality data is effectively improved, and low-quality data in the large language model data can be accurately analyzed. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0042] The present invention discloses a method for evaluating low-quality data for pre-training of a large language model. The flowchart of the method is as follows: Figure 1 As shown. This method is mainly used to evaluate the proportion of low-quality data in the training data of the large language model. This method includes two main parts: low-quality data indicator mining and low-quality indicator evaluation method. The low-quality data indicator mining part first samples the data according to the data distribution, and then uses the large language model to score and aggregate the low-quality data for manual annotation, and mines out the low-quality data indicators in the pre-training corpus, such as safety (toxicity), relevance (advertising), effectiveness (garbled, empty content), etc. The low-quality indicator evaluation method part is mainly based on the mined low-quality data indicators, and designs corresponding evaluation methods, such as heuristic rules, large language model scoring, etc., so as to evaluate the full amount of pre-training data or sampled data, and finally obtain the actual proportion of pre-training data in each low-quality data indicator.

[0043] The specific steps of the present invention include:

[0044] Acquire the data to be evaluated, and sample the data to be evaluated to obtain a sample;

[0045] The sampled samples are scored using a large language model, and the scored data are divided into high-quality data and low-quality data. The high-quality data and the low-quality data are sampled according to a certain ratio, and the sampled data are manually labeled to obtain low-quality data indicators, which include heuristic indicators and non-heuristic indicators.

[0046] First, the data to be evaluated is evaluated according to the heuristic indicators, and the corresponding heuristic rules are used to obtain the heuristic rule scores. Based on the heuristic rule scores, the data to be evaluated is divided into preliminary low-quality data and preliminary high-quality data, and the proportion of preliminary low-quality data is obtained;

[0047] Then, the preliminary high-quality data is evaluated according to non-heuristic indicators, and the preliminary high-quality data is divided into final low-quality data and final high-quality data based on the dimensions defined for the large language model, and the total proportion of low-quality data in the data to be evaluated is calculated.

[0048] Specifically, the first part: low-quality data indicator mining part first samples data according to data distribution, and then uses a large language model to score and aggregate low-quality data for manual annotation, mining low-quality data indicators in pre-training corpus, such as safety (toxicity), relevance (advertising), effectiveness (garbled code, empty content), etc. The specific details are as follows:

[0049] 1. Sampling data:

[0050] a. Stratified sampling according to the distribution of pre-training data (such as subject, data source, etc.), the sampling quantity calculation formula is:

[0051]

[0052] in:

[0053] n is the number of samples taken.

[0054] Z is the Z score (i.e., the standard deviation of the normal distribution), and this value depends on the confidence level you want. For example, if you want a 95% confidence level, then the value of Z should be 1.96.

[0055] p is the probability of success in the population, if this is not known you can use 0.5 as this will provide the most conservative (largest) sample size.

[0056] E is the margin of error you are willing to accept, also called the margin of error or confidence interval (expressed as a percentage), such as 0.01.

[0057] 2. Large language model scoring distinguishes high-quality and low-quality data:

[0058] a. Score the sampled data in eight dimensions: fluency, comprehensibility, completeness, validity, similarity, relevance, security, and other low quality. The specific prompts are as follows:

[0059] Prompt

[0060] ###Role

[0061] You are an AI evaluator tasked with evaluating the suitability oftext data for a pre-trained language model.

[0062] ###Background

[0063] The text data has been collected from diverse sources includingsocial media,news articles,academic papers,and forums.

[0064] ###Goals

[0065] The goal is to evaluate the quality of the text and determine itssuitability for training a robust language model.

[0066] ###Text Issues Classification

[0067] 1.ineffectiveness A text is considered invalid if it is empty,consists exclusively of a URL,contains only line breaks,or lacks thenecessary length to convey substantiveinformation.

[0068] 2.incompleteness A complete text thoroughly addresses its subjectwithoutomitting essential details.Incomplete texts often end abruptly withpunctuation markslike colons or ellipses,indicating unfinished thoughts.Theycan also lack adequateexplanations,content,or necessary formulas pertinent tothe discussion.

[0069] 3.disunderstandability Evaluate whether the text format is correct,includingpunctuation and layout.

[0070] 4.dissimilarity Evaluate whether there is unnecessary repetition inthe text,such asrepeated symbols or characters,and identify any redundantcontent,including headers,footers,page numbers,and source information.

[0071] 5.disfluency A text is considered unfluent if it has disordered wordsequences,word clumping,punctuation errors,and inconsistencies with naturalreading order.

[0072] 6.irrelevance Assess whether the text includes irrelevant orredundant content,such as repeated symbols or characters,headers,footers,pagenumbers,and sourceinformation.

[0073] 7.insecurity Evaluate whether the text contains any inappropriate orharmfulcontent,such as pornography,violence,politically sensitive material,orpersonalidentifiable information(PII).

[0074] 8.Others This category covers any issues not explicitly mentioned inthe abovecategories but which indicate that the text has problems or otherinconsistencies thataffect the overall quality of the text.

[0075] 9.Good A text is considered good if it meets all the quality criteriaand does notfall into any of the low-quality categories.It is accurate,meaningful,complete,properlyformatted,non-redundant,fluent,relevant,and safe.

[0076] ###Quality label

[0077] 1. High-quality: Assign this label if the text meets all the quality criteria and does not fall into any of the low-quality categories.

[0078] 2. Low-quality: Assign this label if the text falls into any of the low-quality classification.

[0079] Workflow

[0080] 1. Carefully read and understand the text by user

[0081] 2. Assign a quality label to the text.

[0082] 3. Assign a text quality classification for why you label it as low-quality. There could be multiple low-quality classifications for one record of data. Assign classification as good when you label it as high-quality.

[0083] 4. State the reason for your evaluation.

[0084] 5. Return the results in JSON format: "{\"Quality\": Quality label, \"Quality_Classification\": \"Text Issues Classification\", \"reason\": \"evaluation reason\"}"

[0085] The specific eight dimensions for the above scoring are as follows:

[0086] 1. Invalidity: If the text is empty, consists only of URLs, contains only line breaks, or lacks the length required to convey substantive information, it is considered invalid.

[0087] 2. Incompleteness: A complete text addresses its subject thoroughly, without leaving out essential details. Incomplete texts often end abruptly with punctuation marks such as colons or ellipses, indicating unfinished ideas. They may also lack adequate explanations, content, or necessary formulas relevant to the discussion.

[0088] 3. Incomprehensibility: Evaluate whether the text is formatted correctly, including punctuation and layout.

[0089] 4. Dissimilarity: Assess the text for unnecessary repetition, such as repeated symbols or characters, and identify any redundant content, including headers, footers, page numbers, and source information.

[0090] 5. Disfluency: A text is considered disfluent if it has disordered word sequences, word clustering, punctuation errors, and inconsistency with the natural reading order.

[0091] 6. Irrelevant: Evaluate whether the text contains irrelevant or redundant content, such as repeated symbols or characters, headers, footers, page numbers, and source information.

[0092] 7. Unsafe: Evaluates whether the text contains any inappropriate or harmful content, such as pornography, violence, politically sensitive material, or personally identifiable information (PII).

[0093] 8. Other: This category covers any issues not explicitly mentioned in the above categories but which indicate problems or other inconsistencies in the text that affect the overall quality of the text.

[0094] 3. Manual labeling: The scoring results are manually labeled according to high-quality and low-quality N:M sampling, and low-quality data indicators are summarized, such as safety (toxicity), relevance (advertising), effectiveness (garbled text, empty content), etc.

[0095] Part 2: The evaluation method for low-quality indicators is mainly based on the mined low-quality data indicators to design corresponding evaluation methods. Heuristic rules are given priority for indicators that can be evaluated by heuristic rules, such as the effectiveness of low-quality data indicators (the content is empty). Large language models are used to score and evaluate indicators that cannot be achieved by heuristic rules. Considering the efficiency and cost of heuristic rules and large language model scoring methods, the heuristic rule method evaluates the full amount of pre-trained data, and the large language model scoring evaluates the sampled data. Finally, the actual proportion of each low-quality data indicator in the pre-training data is obtained by comprehensive calculation. The specific details are as follows:

[0096] 1. Heuristic rule evaluation method:

[0097] a. Heuristic rules are preferred for indicators that can be evaluated by heuristic rules, such as the validity of low-quality data indicators (content is empty):

[0098]

[0099] 2. Large model scoring and evaluation method:

[0100] a. For indicators that cannot be achieved by heuristic rules, a large language model is used for scoring and evaluation. The scoring prompt is scored from seven dimensions: fluency, comprehensibility, completeness, effectiveness, similarity, relevance, and security. The definition of high-quality data is that all seven dimensions must be met. If any of the dimensions is not met, it is considered low-quality data. The specific prompt is:

[0101] Assume you are a pre-trainer of an artificial intelligence language model.You are now examining a piece of text data scraped from theinternet.Your task is to decide whether this data is suitable for training alarge language model.

[0102] There are many criteria that you need to consider. They are listed below in order of importance, along with examples of how to apply them:

[0103] effectiveness:check whether the data is effective.Ineffective datarefers to content is all carriage returns or spaces.If the data contains alot of Chinese or English garbledcharacters or meaningless content,the data is also ineffective.

[0104] relevance:check whether the data contains irrelevant information.Irrelevantinformation refers to author,reporter,source,publisher,lead-incontent,copyright,statement,references,date,a call to action at the beginningor end.

[0105] completeness:check whether the data is complete.A complete textleaves nosignificant questions unanswered and does not lack necessaryinformation.

[0106] understandability:check whether the data is understandable.Markdownformattedformulas,tables,and headings can be correctly converted intoMarkdown text.There areno special characters that cannot be understood.

[0107] similarity:check whether the data contains similar content.A healthydata shouldnot contain too much repetitive content.

[0108] fluency:check whether the data is fluent in terms of logic andformat.The contentshould not be a pile of illogical characters.There shouldbe no sentences or words areunreasonably truncated.The sentence shouldinclude reasonable punctuation marks.

[0109] security:check whether the data contains any toxic,harmful,or illegalinformation.Safe text should not contain sensitive personal information,gambling,pornography,political or prohibited information.

[0110] After assessing the data based on these criteria,give a score thatreflects thecumulative suitability of the data for training.

[0111] Score has only two value:0 and 1.A score of 0 indicates the data hitsat least oneof the criteria mentioned above.A score of 1 indicates the datadoes not hit any of thecriteria mentioned above.

[0112] Type is one of the list:["ineffectiveness","incompleteness","disunderstandability","dissimilarity","disfluency","irrelevance","insecurity"].

[0113] If there are multiple errors in the text,select the most severe errortype.

[0114] If score is 1, type is "high-quality".

[0115] Reason is explanation for the scoring.

[0116] Return answer in JSON format:{"score":0,"type":"xxx","reason":"xxx"}.

[0117] Please remember to output only the JSON FORMAT,without any additionalcontent.

[0118] Here is the data you need to evaluate.

[0119] The corresponding dimensions are:

[0120] Validity: Check whether the data is valid. Invalid data means that the content is all carriage returns or spaces. If the data contains a lot of garbled Chinese and English characters or meaningless content, the data is also invalid.

[0121] Relevance: Checks whether the data contains irrelevant information. Irrelevant information refers to authors, journalists, sources, publishers, lead-in content, copyrights, notices, references, dates, opening or closing calls to action.

[0122] Completeness: Checks that the data is complete. A complete text does not leave any major questions unanswered and does not lack necessary information.

[0123] Understandability: Check whether the data is understandable. Markdown formulas, tables, and titles can be correctly converted to Markdown text. There are no special characters that cannot be understood.

[0124] Similarity: Check if the data contains similar content. Healthy data should not contain too much duplicate content.

[0125] Fluency: Check if the data flows well in terms of logic and format. The content should not be a jumble of illogical characters. No sentences or words should be truncated unreasonably. Sentences should include reasonable punctuation.

[0126] Safety: Checks whether the data contains any toxic, harmful or illegal information. Safety text should not contain sensitive personal information, gambling, pornography, politics or prohibited information.

[0127] After the data are evaluated against these criteria, a score is given that reflects the data's cumulative suitability for training.

[0128] The score has only two values: 0 and 1. A score of 0 means that the data meets at least one of the above criteria. A score of 1 means that the data does not meet any of the above criteria.

[0129] 3. Overall indicator evaluation results:

[0130] a. Considering the efficiency and cost of heuristic rules and large language model scoring methods, the heuristic rule method evaluates the full amount of pre-training data, and the large language model scoring evaluates the sampled data. Finally, the actual proportion of each low-quality data indicator in the pre-training data is obtained through comprehensive calculation. The calculation process is as follows:

[0131] i. First, assume that the total amount of data to be evaluated is X;

[0132] ii. After the first round of heuristic rule evaluation, we get Y1 low-quality data, X-Y1 high-quality data, and the proportion of low-quality data indicators in the first round: indicator Mi proportion: m ii %;

[0133] iii. In the second round, S data are sampled from the remaining high-quality (X-Y1) data for large-scale model scoring and evaluation, and Y2 low-quality data and S-Y2 high-quality data are obtained, as well as the proportion of low-quality data indicators in the second round: indicator Mi proportion: m ij %;

[0134] iv. Combine the first and second rounds to calculate the actual proportion of low-quality data indicators in the pre-training quantity, that is, 1. The final indicator Mi_i proportion is:

[0135]

[0136] Compared with the prior art, the present invention has the following advantages: 1. More efficient: using a large language model to assist manual mining and evaluation of low-quality data can effectively improve the coverage of mining problems. For example, the manual efficiency is about 200 data / person-day, and the large language model efficiency is about 5000 data / GPUh;

[0137] 2. More accurate: Since the large language model greatly improves the efficiency of manual mining and evaluation of low-quality data, the amount of sampled data can be expanded, thereby obtaining more accurate evaluation results for the full amount of data.

[0138] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for evaluating low-quality data for pre-training of a large language model, characterized in that: The method comprises the following steps: Acquire the data to be evaluated, and sample the data to be evaluated to obtain a sample; The sampled samples are scored using a large language model, and the scored data are divided into high-quality data and low-quality data. The high-quality data and the low-quality data are sampled according to a certain ratio, and the sampled data are manually labeled to obtain low-quality data indicators, which include heuristic indicators and non-heuristic indicators. First, the data to be evaluated is evaluated according to the heuristic indicators, and the corresponding heuristic rules are used to obtain the heuristic rule scores. Based on the heuristic rule scores, the data to be evaluated is divided into preliminary low-quality data and preliminary high-quality data, and the proportion of preliminary low-quality data is obtained; Then, the preliminary high-quality data is evaluated according to non-heuristic indicators, and the preliminary high-quality data is divided into final low-quality data and final high-quality data based on the dimensions defined for the large language model, and the total proportion of low-quality data in the data to be evaluated is calculated.

2. The method for evaluating low-quality data for pre-training of a large language model according to claim 1 is characterized in that: The use of a large language model to score the sampled samples is specifically as follows: The sample was scored on eight dimensions: invalidity, incompleteness, incomprehensibility, dissimilarity, unfluency, irrelevance, insecurity, and others.

3. The method for evaluating low-quality data for pre-training of a large language model according to claim 2 is characterized in that: The number of samples is: Where n is the number of samples taken, Z is the Z score which is the standard deviation of the normal distribution, p is the probability of success in the population, and E is the margin of error.

4. The method for evaluating low-quality data for pre-training of a large language model according to claim 1 is characterized in that: The low-quality data indicators include validity, relevance, completeness, comprehensibility, similarity, fluency and security.

5. The method for evaluating low-quality data for pre-training of a large language model according to claim 4 is characterized in that: The heuristic metrics include effectiveness.

6. The method for evaluating low-quality data for pre-training of a large language model according to claim 5 is characterized in that: When the heuristic indicator is effectiveness, the corresponding heuristic rule is used to obtain the heuristic rule score as follows: Remove the leading space symbol from the data to be evaluated, and then calculate the string length of the data to be evaluated. If the string length is 0, give the score corresponding to the low-quality data.

7. The method for evaluating low-quality data for pre-training of a large language model according to claim 6 is characterized in that: Based on the dimensions defined for the large language model, the sampled preliminary high-quality data is divided into final low-quality data and final high-quality data as follows: The preliminary high-quality data is sampled and divided into final low-quality data and final high-quality data based on the seven dimensions defined for the large language model. The seven dimensions and the corresponding scoring results are: Validity: Check whether the data is valid. If the content is all carriage returns or spaces or contains garbled Chinese and English characters exceeding the character count threshold or meaningless content, a low quality score will be given. Otherwise, a high quality score will be given. Relevance: Checks whether the data contains irrelevant information, and gives a final low quality score if it does not. The irrelevant information includes references to authors, journalists, sources, publishers, imports, copyrights, notices, references, dates, and calls to action at the beginning or end. Otherwise, a high quality score is given; Completeness: Check whether the data is complete. If not, a low quality score is given. Otherwise, a high quality score is given. Understandability: Check whether the data is understandable. If the formulas, tables, and titles in Markdown format are correctly converted to Markdown text and there are no incomprehensible special characters, then the data is understandable and a high quality score is given. Otherwise, a low quality score is given. Similarity: Check whether the data contains similar content that reaches the maximum ratio threshold. If so, a low quality score is given, otherwise a high quality score is given; Fluency: Check whether the data is fluent in terms of logic and format. If the sentence includes unreasonable punctuation, a low quality score will be given, otherwise a high quality score will be given; Security: Check whether the data contains any toxic, harmful or illegal information. If so, a low quality score is given; otherwise, a high quality score is given; For a piece of data in the sampled preliminary high-quality data, if the scoring results of the seven dimensions are all final high-quality scores, then the data is intermediate high-quality data, otherwise it is intermediate low-quality data. Based on the proportion of intermediate low-quality data in the sampled preliminary high-quality data, all preliminary high-quality data are divided into final low-quality data and final high-quality data.

8. The method for evaluating low-quality data for pre-training of a large language model according to claim 7 is characterized in that: The total proportion of the low-quality data in the data to be evaluated is specifically: Assuming that the total amount of data to be evaluated is X, after the heuristic indicator evaluation, we get Y1 pieces of preliminary low-quality data and X-Y1 pieces of preliminary high-quality data, and get the proportion of preliminary low-quality data m ii %; Assume that the number of intermediate low-quality data is Y2, the number of intermediate high-quality data is S-Y2, and the proportion of intermediate low-quality data in the sampled preliminary high-quality data is m ij %; Based on the proportion of preliminary low-quality data m ii % and m ij % Calculate the total proportion of low-quality data in the data to be evaluated.

9. The method for evaluating low-quality data for pre-training of a large language model according to claim 8 is characterized in that: The total proportion of the low-quality data in the data to be evaluated is:

10. The method for evaluating low-quality data for pre-training of a large language model according to claim 1 is characterized in that: The data to be evaluated is sampled to obtain the sampled samples as follows: The data to be evaluated is stratified according to its data distribution to obtain sampling samples.

Images