Data cleaning method and device for large model training

Abstract

The invention discloses a data cleaning method for large model training. The method comprises the following steps of: converting original data in different formats into data in a uniform plain text format; filtering rules of three levels of sentence filtering, paragraph filtering and article filtering are adopted to perform rule filtering on the data in the plain text format. And generating a cleaning rule for the filtered data of each rule by adopting a trained large model, and cleaning according to the cleaning rule to obtain the filtered data of the model. And de-weighting the data filtered by the model by adopting a de-weighting mode of three levels of sentences, paragraphs and articles. And a plurality of mass filtering models are adopted to carry out mass filtering on the data after deduplication, so that the data after mass filtering can be obtained. And finally, carrying out education quality scoring on the data after quality filtering, and only reserving text data of which the education quality score is greater than a preset threshold value. According to the invention, noise data can be removed more thoroughly; repeated data can be removed more thoroughly; and low-quality data can be effectively filtered out.

Description

Technical Field

[0001] The present application relates to a data cleaning method. Background Art

[0002] Data cleaning is the process of detecting and correcting (or removing) corrupted or inaccurate data, which involves identifying incomplete, incorrect, inaccurate, or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or rough data.

[0003] Big models refer to machine learning models with large-scale parameters and complex computational structures; these models are usually built with deep neural networks (DNNs) and have billions or even hundreds of billions of parameters. Big models learn complex patterns and features by training massive amounts of data, have stronger generalization capabilities, and can make accurate predictions on unseen data. Big models are widely used in natural language processing, computer vision, speech recognition, and recommendation systems.

[0004] Data cleaning for large models refers to checking and correcting the training data when preparing it. Since large models require a large amount of training data, which comes from various sources and may contain errors, inconsistencies, missing values ​​or other problems, data cleaning becomes particularly important. The purpose of data cleaning is to ensure that the training data provided to the large model is accurate, consistent and complete, which can improve the training efficiency and final performance of the large model. Summary of the invention

[0005] The technical problem to be solved by this application is: how to make the data cleaning method applicable to the training data of large models.

[0006] To solve the above technical problems, the present application proposes a data cleaning method for large model training, comprising the following steps. Step S1: Convert raw data of different formats into data in a unified plain text format; a piece of data in plain text format is an article, each article includes one or more paragraphs, and each paragraph includes one or more sentences. Step S2: Use filtering rules at three levels of sentence filtering, paragraph filtering and article filtering to filter the data in plain text format according to rules, and obtain data filtered by rules. Step S3: Use a trained large model to generate cleaning rules for each rule-filtered data, and clean each rule-filtered data according to the cleaning rules generated by the large model to obtain model-filtered data. Step S4: Use sentence-level deduplication, paragraph-level deduplication and article-level deduplication to dedupe the data filtered by the model, and obtain deduplicated data. Step S5: Use multiple quality filtering models to perform quality filtering on the deduplicated data, and obtain quality-filtered data; the quality filtering model includes any of the following: language filtering model, advertising detection model, N-gram fluency filtering model, vocabulary filtering model, context consistency detection model. Step S6: Perform an educational quality score on the quality filtered data. If the educational quality score of a certain text data is greater than the ninth threshold, the text data is retained; otherwise, the text data is deleted.

[0007] Preferably, in step S1, the data in the unified plain text format is in JSONL format.

[0008] Further, in step S2, the sentence filtering includes any one or more of the following rules: retaining sentences in md format; filtering sentences containing web page navigation bars; filtering sentences containing personal privacy information; filtering sentences containing web page links; when the proportion of filtered sentences to the total number of sentences in the article exceeds a first threshold, deleting the entire article. The paragraph filtering includes any one or more of the following rules: filtering paragraphs containing special characters; filtering paragraphs containing more than 5 uncommon characters; when the proportion of filtered paragraphs to the total number of paragraphs in the article exceeds a second threshold, deleting the entire article. The article filtering includes any one or more of the following rules: filtering short articles with less than 50 words; filtering articles with more than 80% uppercase letters; filtering articles with more than 80% numbers; filtering articles with more than 40% punctuation marks; filtering articles with less than 90% common characters; filtering articles containing sensitive words.

[0009] Furthermore, in step S3, the input of the large model is text, and the output is the cleaning rules corresponding to the text; there are four cleaning rules: no need to clean the text; delete all text; delete lines * to * in the text; replace a word in the text with another word.

[0010] Preferably, in step S3, the large model is implemented using any one of GPT, Qwen, and Llama, and a supervised fine-tuning training method is adopted.

[0011] Further, in step S4, the sentence-level deduplication refers to: filtering out sentences whose occurrence times exceed the third threshold. The paragraph-level deduplication includes any one or more of the following: filtering out paragraphs whose occurrence times exceed the fourth threshold; filtering out duplicate paragraphs in the text, and retaining only one; filtering out one of the two paragraphs in the text whose edit distance exceeds the fifth threshold; filtering out one of the two paragraphs in the text whose word overlap exceeds the sixth threshold. The article-level deduplication includes any one or more of the following: using the local sensitive hashing algorithm to calculate the hash value of the text, filtering out one of the two texts with the same hash value; using the sentence embedding model to calculate the embedded representation of the text, and then calculating the cosine similarity of the embedded representations of the two texts, filtering out one of the two texts whose cosine similarity exceeds the seventh threshold.

[0012] Further, in step S5, the language filtering model tags the text with a language tag, marks the language to which it belongs, and then filters out text that is not in the target language based on the language tag. The advertising detection model tags the text with a tag to determine whether it belongs to an advertisement, and then filters out text that belongs to an advertisement based on the tag. The N-gram fluency filtering model calculates the perplexity for the text, and then filters out text whose perplexity exceeds an eighth threshold. The vocabulary filtering model tags the text with a tag to determine whether it contains junk words and sensitive words, and then filters out text containing junk words and sensitive words based on the tag. The context consistency detection model tags the text with a tag to determine whether its context is consistent, and then filters out text with inconsistent context based on the tag.

[0013] Preferably, in step S6, the LLAMA3.1-70B model trained by the prompt learning mode is used to perform educational quality scoring on the text data.

[0014] The present application also proposes a data cleaning device for large model training, including a format conversion unit, a rule filtering unit, a model filtering unit, a deduplication unit, a quality filtering unit, and a quality scoring unit. The format conversion unit is used to convert raw data of different formats into data in a unified plain text format; a piece of plain text format data is an article, each article includes one or more paragraphs, and each paragraph includes one or more sentences. The rule filtering unit is used to use filtering rules at three levels: sentence filtering, paragraph filtering, and article filtering, to perform rule filtering on the data in plain text format to obtain rule-filtered data. The model filtering unit is used to use a trained large model to generate cleaning rules for each rule-filtered data, and to clean each rule-filtered data according to the cleaning rules generated by the large model to obtain model-filtered data. The deduplication unit is used to use sentence-level deduplication, paragraph-level deduplication, and article-level deduplication to deduplication the data filtered by the model to obtain deduplication data. The quality filtering unit is used to perform quality filtering on the deduplicated data using multiple quality filtering models to obtain quality filtered data; the quality filtering models include any of the following: language filtering model, advertisement detection model, N-gram fluency filtering model, vocabulary filtering model, context consistency detection model. The quality scoring unit is used to perform educational quality scoring on the quality filtered data. If the educational quality score of a certain text data is greater than the ninth threshold, the text is retained, otherwise the text is removed.

[0015] The technical effects achieved by this application are: it can cover raw data from multiple sources and in multiple formats, providing comprehensive corpus support for the training of large models; it can more thoroughly remove noise data; it can more thoroughly remove duplicate data; it can effectively filter out low-quality data; and the processing process standards are unified. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of the data cleaning method for large model training proposed in this application.

[0017] Figure 2 is a schematic diagram of an example of input text of the large model in step S3.

[0018] Figure 3 It is a structural schematic diagram of the data cleaning device for large model training proposed in this application.

[0019] Explanation of the reference numerals in the figure: format conversion unit 1, rule filtering unit 2, model filtering unit 3, deduplication unit 4, quality filtering unit 5, quality scoring unit 6. DETAILED DESCRIPTION

[0020] See also Figure 1,The data cleaning method for large model training proposed in this application includes the following steps.

[0021] Step S1: Convert raw data of different types (formats) into a unified format to obtain data in plain text format. A piece of data in plain text format is an article, each article includes one or more paragraphs, and each paragraph includes one or more sentences.

[0022] The types of raw data include, for example, PDF, CSV, MOBI, AZW, warc, html and other formats. The unified format is a plain text format, preferably JSON Lines (JSONL). JSONL is a text file format that contains an independent JSON (JavaScript Object Notation) object per line, and is often used to process large-scale data sets because it allows line-by-line reading and writing, reducing memory burden.

[0023] This step of format conversion can be achieved using different tools. For example, the toolkit pdfplumber is used to convert PDF data into plain text format, the toolkit pandas is used to convert CSV data into plain text format, the toolkit calibre is used to convert MOBI data and AZW data into plain text format, the toolkit warcio is used to convert warc data into plain text format, and the toolkit trafilatura is used to convert html data into plain text format.

[0024] Step S2: Using filtering rules at three different levels, sentence filtering, paragraph filtering and article filtering, the data in plain text format is filtered according to the rules, and the data that does not meet the rules is removed to obtain the data after the rules are filtered.

[0025] As an example, the sentence filtering includes any one or more of the following rules: retaining sentences in md (Markdown) format; filtering (i.e., deleting) sentences containing web page navigation bars (e.g., item|item|item or item1>item2>item3); filtering sentences containing personal privacy information (e.g., phone number, mobile phone number, zip code, ID number, email address, etc.); filtering sentences containing web page links. When the proportion of the filtered sentences to the total number of sentences in the article exceeds a first threshold, the entire article is deleted.

[0026] As an example, the paragraph filtering includes any one or more of the following rules: filtering out paragraphs containing special characters, where special characters refer to special symbols, such as ▆▂▆▅◣▄▅▃▂▃ ▃▂▃▅▄◢◥◣◥◣▆▄▂▅▄▅▂▅▂▅▂▅▄▅▂▄▅▆, etc.; filtering out paragraphs containing more than 5 uncommon characters, where uncommon characters refer to characters that are not Chinese, not English, and are not very commonly used symbols. When the number of paragraphs filtered out exceeds the second threshold in the total number of paragraphs in the article, the entire article is deleted.

[0027] As an example, the article filtering includes any one or more of the following rules: filtering short articles with less than 50 words; filtering articles with more than 80% uppercase letters; filtering articles with more than 80% numbers; filtering articles with more than 40% punctuation marks; filtering articles with less than 90% common characters, common characters include: 3,500 common Chinese characters, uppercase and lowercase English letters, common symbols; filtering articles containing sensitive words (harmful content).

[0028] Step S3: Use a trained big model to generate cleaning rules for the data filtered by each rule, and clean the data filtered by each rule according to the cleaning rules generated by the big model to obtain the data filtered by the model.

[0029] The input of the large model is text, and the output is the cleaning rules corresponding to the text. There are four types of cleaning rules: no need to clean the text; delete all text; delete lines * to * in the text; replace a word in the text with another word. For example, the text data input to the large model is Figure 2 As shown, there are 10 rows in total. The cleaning rule for the large model output is: delete rows 1 to 5.

[0030] The large model is implemented, for example, using GPT, Qwen, and Llama, and adopts a supervised fine-tuning (SFT) training method, preferably a full parameter fine-tuning training method. Fine-tuning refers to the implementation of an additional training process using a targeted training data set on a pre-trained large language model that already has a broad knowledge base, aiming to make the large model more accurately fit the specific task requirements or go deeper into a certain professional field.

[0031] The rule filtering in step S2 is to filter the text using the pre-defined rules manually, and the model filtering in step S3 is to clean the text using the cleaning rules generated by the trained large model for each text. Because the manually defined rules cannot completely remove useless sentences, useless paragraphs, and useless articles, it is necessary to use the cleaning rules generated by the trained large model to supplement the rules and further delete useless sentences, useless paragraphs, and useless articles. Using a large model can avoid the shortcomings of the lack of flexibility and low coverage of manual rules.

[0032] Step S4: Deduplication is performed on the data filtered by the model using three different deduplication methods: sentence-level deduplication, paragraph-level deduplication, and article-level deduplication, to obtain deduplicated data. Deduplication can reduce storage space and improve information density.

[0033] As an example, the sentence level deduplication refers to filtering out sentences that appear too many times, for example, establishing a global sentence frequency table, and removing sentences that appear more than a third threshold.

[0034] As an example, the paragraph-level deduplication includes any one or more of the following: filtering out paragraphs that appear too many times, for example, establishing a global paragraph frequency table, and removing paragraphs whose appearance times exceed a fourth threshold; filtering out duplicate paragraphs in the text, retaining only one; filtering out one of two paragraphs whose edit distance in the text exceeds a fifth threshold; filtering out one of two paragraphs whose word overlap in the text exceeds a sixth threshold.

[0035] As an example, the article-level deduplication includes any one or more of the following: using a locality-sensitive hashing (LSH) algorithm to calculate the hash value of the text (i.e., the article), and filtering out one of the two texts with the same hash value; using a sentence embedding model to calculate the embedded representation of the text, and then calculating the cosine similarity (Cosine Similarity) of the embedded representations of the two texts, and filtering out one of the two texts whose cosine similarity exceeds the seventh threshold.

[0036] Step S5: Use multiple quality filtering models to perform quality filtering on the deduplicated data to obtain quality filtered data. The quality filtering models include any one or more of the following: language filtering model, advertisement detection model, N-gram fluency filtering model, vocabulary filtering model, context consistency detection model.

[0037] The language filtering model tags the text with a language tag to mark the language to which it belongs, and then filters out text that is not in the target language based on the language tag. The target language is, for example, Chinese and English.

[0038] The advertisement detection model labels text as to whether it is an advertisement, and then filters out the text that is an advertisement based on the label.

[0039] The N-gram fluency filtering model calculates the perplexity of the text. The higher the perplexity, the less common the text is and the lower the fluency is. The lower the perplexity, the more common the text is and the higher the fluency is. Then, the text exceeding the eighth threshold is filtered out according to the perplexity. For example, "This is really nice weather, but I don't know that", the text in double quotes is obviously not fluent and can be filtered out by the N-gram fluency filtering model.

[0040] The vocabulary filtering model labels the text as to whether it contains junk words and sensitive words, and then filters out the text containing junk words and sensitive words according to the labels.

[0041] The context consistency detection model labels the text as to whether its context is consistent, and then filters out the text with inconsistent context according to the label. For example, "A Canadian Air Force CF-18 fighter crashed near Lenghu City. Jebit, King of the Kings, and Son Goku, Piccolo, Vegeta and others also arrived at this time." The text in double quotes is obviously unrelated to the previous and the next, and can be filtered out by the context consistency detection model.

[0042] Step S6: Perform an educational quality score on the quality-filtered data. If the educational quality score of a certain text data is greater than the ninth threshold, the text is retained, otherwise the text is removed. Preferably, this step uses the LLAMA3.1-70B model trained in prompt learning mode to perform educational quality scoring on the text data. Llama 3 is a language model that supports multiple languages, code writing, reasoning, and tool use. Prompt learning is a pre-training model paradigm that introduces templates during the pre-training process to transform the original input into a format similar to cloze, allowing the language model to answer, helping the language model to better utilize the contextual information of the task, thereby improving the performance of the language model.

[0043] See also Figure 3 The data cleaning device for large model training proposed in this application includes a format conversion unit 1, a rule filtering unit 2, a model filtering unit 3, a deduplication unit 4, a quality filtering unit 5, and a quality scoring unit 6. Figure 3 The device shown corresponds to Figure 1 The method shown.

[0044] The format conversion unit 1 is used to convert original data in different formats into data in a unified plain text format.

[0045] The rule filtering unit 2 is used to perform rule filtering on data in a plain text format by using filtering rules at three levels: sentence filtering, paragraph filtering and article filtering, to obtain rule-filtered data.

[0046] The model filtering unit 3 is used to generate cleaning rules for the data after each rule filtering using the trained large model, and clean the data after each rule filtering according to the cleaning rules generated by the large model to obtain the data after the model filtering.

[0047] The deduplication unit 4 is used to deduplicate the data after model filtering by adopting three levels of deduplication methods: sentence level deduplication, paragraph level deduplication and article level deduplication, to obtain deduplicated data.

[0048] The quality filtering unit 5 is used to perform quality filtering on the deduplicated data using multiple quality filtering models to obtain quality filtered data.

[0049] The quality scoring unit 6 is used to perform educational quality scoring on the data after quality filtering. If the educational quality score of a certain text data is greater than a ninth threshold, the text is retained, otherwise the text is removed.

[0050] Compared with the prior art, the present application has the following beneficial effects.

[0051] First, the training of large models usually requires a large-scale training data set, which needs to come from a variety of different sources and have a variety of different formats to ensure the diversity and comprehensiveness of the training data. Step S1 of this application fully processes and integrates data from different channels and formats, and converts data in different formats from all sources into a plain text format, which is conducive to subsequent data processing and model training.

[0052] Second, the performance of the large model depends largely on the quality of the training data. Therefore, data cleaning must remove noise data and ensure that the quality of the retained data is high enough to improve the training efficiency and final performance of the large model. General data cleaning mainly relies on manually preset rules. This application combines the rule filtering of step S2 and the model filtering of step S3, and uses the large model to generate specific cleaning rules for each text to supplement the deficiencies of manual rules, and more thoroughly removes noise data, such as garbled characters, uncommon characters, abnormal punctuation, etc., to ensure the readability of the retained data; it also filters out privacy content to reduce the risk of data leakage and improve data security; it also removes harmful content, improves the social responsibility awareness of the large model, and improves the flexibility and coverage of cleaning.

[0053] Third, data cleaning in large model training pays more attention to deduplication operations to reduce data redundancy and improve information density. This application can more thoroughly remove duplicate data through the three-level deduplication operations in step S4, thereby making more efficient use of data.

[0054] Fourth, the data cleaning of large model training not only focuses on the accuracy and consistency of training data, but also on the high quality of training data. The quality assessment is more comprehensive. Step S5 of this application introduces multiple text quality assessment indicators - such as language filtering, advertising detection, N-gram fluency filtering, vocabulary filtering, and context consistency detection, etc. - to perform quality filtering on training data to comprehensively improve data quality. Step S6 of this application will also score the data for educational quality to ensure that the retained text data has high educational value and information content, providing high-quality training data sets for large models.

[0055] Fifth, the processing flow of this application is standardized. The entire data cleaning process is standardized, ensuring the consistency of processing and facilitating the subsequent work.

[0056] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A data cleaning method for large model training, characterized in that: The method comprises the following steps: Step S1: converting raw data in different formats into data in a unified plain text format; a piece of data in a plain text format is an article, each article includes one or more paragraphs, and each paragraph includes one or more sentences; Step S2: using filtering rules at three levels, sentence filtering, paragraph filtering and article filtering, to filter the data in plain text format to obtain data after rule filtering; Step S3: using a trained big model to generate cleaning rules for the data filtered by each rule, cleaning the data filtered by each rule according to the cleaning rules generated by the big model to obtain the data filtered by the model; Step S4: Deduplication is performed on the data filtered by the model using sentence-level deduplication, paragraph-level deduplication and article-level deduplication methods to obtain deduplicated data; Step S5: using multiple quality filtering models to perform quality filtering on the deduplicated data to obtain quality filtered data; the quality filtering models include any of the following: language filtering model, advertisement detection model, N-gram fluency filtering model, vocabulary filtering model, context consistency detection model; Step S6: Perform an educational quality score on the quality filtered data. If the educational quality score of a certain text data is greater than the ninth threshold, the text data is retained; otherwise, the text data is deleted.

2. The data cleaning method for large model training according to claim 1 is characterized in that: In step S1, the data in the unified plain text format is in JSONL format.

3. The data cleaning method for large model training according to claim 1 is characterized in that: In step S2, the sentence filtering includes any one or more of the following rules: retaining sentences in md format; filtering sentences containing web page navigation bars; filtering sentences containing personal privacy information; filtering sentences containing web page links; when the proportion of filtered sentences to the total number of sentences in the article exceeds a first threshold, deleting the entire article; The paragraph filtering includes any one or more of the following rules: filtering out paragraphs containing special characters; filtering out paragraphs containing more than 5 uncommon characters; when the number of filtered paragraphs exceeds a second threshold in the total number of paragraphs in the article, deleting the entire article; The article filtering includes any one or more of the following rules: filtering short articles with less than 50 words; filtering articles with more than 80% capital letters; filtering articles with more than 80% numbers; filtering articles with more than 40% punctuation marks; Filter articles with less than 90% common words; filter articles containing sensitive words.

4. The data cleaning method for large model training according to claim 1 is characterized in that: In step S3, the input of the large model is text, and the output is the cleaning rules corresponding to the text; there are four types of cleaning rules: no need to clean the text; delete all text; delete lines * to * in the text; replace a word in the text with another word.

5. The data cleaning method for large model training according to claim 1 is characterized in that: In step S3, the large model is implemented using any one of GPT, Qwen, and Llama, and a supervised fine-tuning training method is adopted.

6. The data cleaning method for large model training according to claim 1 is characterized in that: In step S4, the sentence level deduplication refers to: filtering out sentences whose occurrence times exceed a third threshold; The paragraph-level deduplication includes any one or more of the following: filtering out paragraphs whose occurrence times exceed a fourth threshold; filtering out duplicate paragraphs in the text and retaining only one; filtering out one of two paragraphs whose edit distance in the text exceeds a fifth threshold; filtering out one of two paragraphs whose word overlap degree in the text exceeds a sixth threshold; The article-level deduplication includes any one or more of the following: using a local sensitive hashing algorithm to calculate the hash value of the text, and filtering out one of the two texts with the same hash value; using a sentence embedding model to calculate the embedded representation of the text, and then calculating the cosine similarity of the embedded representations of the two texts, and filtering out one of the two texts whose cosine similarity exceeds a seventh threshold.

7. The data cleaning method for large model training according to claim 1 is characterized in that: In step S5, the language filtering model tags the text with a language tag, marks the language to which it belongs, and then filters out text that is not in the target language based on the language tag; The advertisement detection model labels the text as being an advertisement, and then filters out the text that is an advertisement based on the label; The N-gram fluency filtering model calculates perplexity for the text, and then filters out the text whose perplexity exceeds an eighth threshold; The vocabulary filtering model labels the text as to whether it contains junk words and sensitive words, and then filters out the text containing junk words and sensitive words according to the labels; The context consistency detection model labels texts to determine whether their contexts are consistent, and then filters out texts with inconsistent contexts based on the labels.

8. The data cleaning method for large model training according to claim 1 is characterized in that: In step S6, the LLAMA3.1-70B model trained by the prompt learning mode is used to score the educational quality of the text data.

9. A data cleaning device for large model training, characterized in that: It includes format conversion unit, rule filtering unit, model filtering unit, deduplication unit, quality filtering unit and quality scoring unit; The format conversion unit is used to convert raw data in different formats into data in a unified plain text format; a piece of data in a plain text format is an article, each article includes one or more paragraphs, and each paragraph includes one or more sentences; The rule filtering unit is used to filter the data in plain text format by using filtering rules at three levels: sentence filtering, paragraph filtering and article filtering, to obtain rule-filtered data; The model filtering unit is used to generate cleaning rules for the data filtered by each rule using the trained big model, and clean the data filtered by each rule according to the cleaning rules generated by the big model to obtain the data filtered by the model; The deduplication unit is used to remove duplicates from the data filtered by the model by adopting three levels of deduplication methods: sentence level deduplication, paragraph level deduplication and article level deduplication, to obtain deduplicated data; The quality filtering unit is used to perform quality filtering on the deduplicated data using multiple quality filtering models to obtain quality filtered data; the quality filtering models include any of the following: language filtering model, advertisement detection model, N-gram fluency filtering model, vocabulary filtering model, context consistency detection model; The quality scoring unit is used to perform educational quality scoring on the data after quality filtering. If the educational quality score of a certain text data is greater than a ninth threshold, the text is retained, otherwise the text is removed.

Images