Pre-training data cleaning method and device, electronic equipment and storage medium

Abstract

The invention provides a pre-training data cleaning method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a target process in an idle state in multiple processes; pre-training data of the single-batch processing number is allocated to the target process, and the single-batch processing number is determined according to the parallel process number and the total number of the pre-training data contained in the pre-training data set; performing data cleaning processing on the distributed pre-training data through the target process to obtain target data; wherein the data cleaning processing comprises condition filtering processing and duplicate removal processing; and when the cleaning of the pre-training data set is completed, obtaining a cleaned data set corresponding to the pre-training data set according to the corresponding target data in each data batch. According to the method, the data quality can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for cleaning pre-training data. Background Art

[0002] Pre-training data refers to data that has been collected before training the model and is used for model training. The quality of pre-training data directly affects the performance and generation capabilities of the model. In practical applications, pre-training data may come from different data sources and may contain noise, missing values, outliers, and other problems. Therefore, in order to ensure that the model can learn useful information from the data, the pre-training data needs to be cleaned.

[0003] However, many steps in current cleaning methods are imperfect, resulting in poor data cleaning results.

[0004] Therefore, there is an urgent need to provide a more efficient and reliable pre-training data cleaning solution to improve data quality. Summary of the invention

[0005] The present application provides a method, device, electronic device and storage medium for cleaning pre-training data, so as to solve the technical problem of poor data cleaning effect.

[0006] In a first aspect, the present application provides a method for cleaning pre-training data, comprising:

[0007] Determine the target process in idle state among multiple processes;

[0008] Allocate a single batch of pre-training data for the target process, where the single batch is determined based on the number of parallel processes and the total number of pre-training data included in the pre-training data set.

[0009] Performing data cleaning processing on the assigned pre-training data through the target process to obtain target data; wherein the data cleaning processing includes conditional filtering processing and deduplication processing;

[0010] When the pre-training data set is cleaned, the cleaned data set corresponding to the pre-training data set is obtained according to the corresponding target data in each data batch.

[0011] Optionally, in the above method, the allocated pre-training data is cleaned by the target process, including:

[0012] In the order of the amount of computation from small to large, the target process performs multiple conditional filtering processes on the allocated pre-training data to obtain target filtered data, wherein different conditional filtering processes correspond to different amounts of computation;

[0013] According to the deduplication rates from large to small, multiple deduplication processes are performed on the target filtered data to obtain the target data, wherein different deduplication processes correspond to different deduplication rates.

[0014] Optionally, in the above method, when the operation amount includes the first operation amount, the second operation amount, and the third operation amount, the conditional filtering process includes the first conditional filtering process, the second conditional filtering process, and the third conditional filtering process, in the order of the operation amount from small to large, the target process performs multiple conditional filtering processes on the allocated pre-training data to obtain the target filtered data, including:

[0015] According to the first operation amount, the target process performs a first condition filtering process on the allocated pre-training data to obtain first filtered data; wherein the first operation amount is less than the second operation amount, and the second operation amount is less than the third operation amount; the first operation amount corresponds to the first condition filtering process;

[0016] Performing a second condition filtering process on the first filtering data according to the second operation amount to obtain second filtering data; wherein the second operation amount corresponds to the second condition filtering process;

[0017] A third condition filtering process is performed on the second filtering data according to the third operation amount to obtain target filtering data; wherein the third operation amount corresponds to the third condition filtering process.

[0018] Optionally, in the above method, when the deduplication rate includes a first deduplication rate and a second deduplication rate, and the deduplication processing includes a first deduplication processing and a second deduplication processing, deduplication processing is performed on the target filtered data according to the deduplication rates from large to small to obtain a target data set, including:

[0019] Performing a first deduplication process on the target filtered data according to a first deduplication rate to obtain first deduplication data; wherein the first deduplication process corresponds to the first deduplication rate; and the first deduplication process is to perform deduplication process using Python program code;

[0020] The first deduplication data is subjected to a second deduplication processing according to a second deduplication rate to obtain target data; the second deduplication processing corresponds to the second deduplication rate; the second deduplication processing is performed using C++ program code.

[0021] Optionally, in the above method, when the first condition filtering process includes privacy information coding, special content removal, text standardization, and copyright information removal, performing the first condition filtering process on the allocated pre-training data according to the first operation amount to obtain the first filtered data includes:

[0022] Replacing the private data in the allocated pre-trained data with the preset coded data to obtain first data;

[0023] Removing special content from the first data to obtain second data; wherein the special content represents data containing one of a URL link or an HTML tag;

[0024] Performing text standardization processing on the second data to obtain third data; wherein the text standardization processing includes one of Unicode standardization processing and converting traditional Chinese characters into simplified Chinese characters;

[0025] The copyright information in the third data is removed to obtain the first filtered data.

[0026] Optionally, in the above method, when the second condition filtering includes count filtering and length filtering, performing the second condition filtering process on the first filtered data according to the second operation amount to obtain the second filtered data includes:

[0027] Obtaining a ratio of the number of letters in the first filtered data to the number of characters in the first filtered data, which is the letter ratio, and obtaining a ratio of the number of numbers in the first filtered data to the number of characters in the first filtered data, which is the number ratio;

[0028] If the letter ratio is within the preset letter ratio interval, and the number ratio is within the preset number ratio interval, then confirm that the mark field of the first filtering data is the first mark value, and determine the mark field of other first filtering data as the second mark value; wherein the other first filtering data is the first filtering data whose number ratio is not within the letter ratio interval, or whose number ratio is not within the number ratio interval;

[0029] Remove the data whose mark field in the first filtered data is the second mark value, and aggregate the first filtered data whose mark field in the first filtered data is the first mark value to obtain fourth data;

[0030] If the length of the fourth data is within the preset length interval, confirming that the mark field of the fourth data is the first mark value, and determining the mark field of other fourth data as the second mark value, wherein the other fourth data is the fourth data whose length is not within the preset length interval;

[0031] The data whose mark field in the fourth data is the second mark value is removed, and the data whose mark field in the fourth data is the first mark value is aggregated to obtain the second filtered data.

[0032] Optionally, in the above method, when the third condition filtering includes sensitive word filtering, language identification and filtering, perplexity filtering, and special character ratio filtering, the third condition filtering processing is performed on the second filtering data according to the third operation amount to obtain the target filtering data, including:

[0033] If the second filtering data includes a sensitive word in the preset sensitive word list, confirming that the tag field of the second filtering data is the second tag value, and determining the tag field of other second filtering data as the first tag value; wherein the other second filtering data is the second filtering data that does not include the sensitive word in the preset sensitive word list;

[0034] Remove the data whose mark field is the second mark value in the second filtered data, and aggregate the data whose mark field is the first mark value in the second filtered data to obtain fifth data;

[0035] Determine the language category of the fifth data and calculate the language confidence of the fifth data according to a preset language discrimination model;

[0036] If the language category of the fifth data is a preset language category and the language confidence of the fifth data is within the language confidence interval, then confirming that the tag field of the fifth data is the first tag value;

[0037] Determine the tag field of other fifth data as the second tag value, wherein the other fifth data is fifth data whose language category is not a preset language category or whose language confidence is not within the language confidence interval;

[0038] Remove the data whose mark field in the fifth data is the second mark value, and aggregate the data whose mark field in the fifth data is the first mark value to obtain the sixth data;

[0039] Calculate the confusion of the sixth data;

[0040] If the perplexity is within the perplexity interval, confirming that the tag field of the sixth data is the first tag value;

[0041] Determine the tag field of other sixth data as the second tag value, wherein the other sixth data is the sixth data whose perplexity is not within the perplexity interval;

[0042] Remove the data whose mark field in the sixth data is the second mark value, and aggregate the data whose mark field in the sixth data is the first mark value, to obtain the seventh data;

[0043] According to the preset special character table, determining the ratio of the special characters in the seventh data to the characters of the seventh data is the special character ratio;

[0044] If the special character ratio is within the special character ratio interval, confirming that the mark field of the seventh data is the first mark value;

[0045] Determine the tag field of the other seventh data as the second tag value, wherein the other seventh data is the seventh data whose special character ratio is not within the special character ratio interval;

[0046] The data whose mark field in the seventh data is the second mark value is removed, and the data whose mark field in the seventh data is the first mark value is aggregated to obtain the target filtered data.

[0047] In a second aspect, the present application provides a device for cleaning pre-training data, comprising:

[0048] A determination module, used for determining a target process in an idle state among multiple processes;

[0049] An allocation module is used to allocate a single batch of pre-training data to the target process, wherein the single batch of pre-training data is determined according to the number of parallel processes and the total number of pre-training data included in the pre-training data set;

[0050] The first obtaining module is used to perform data cleaning processing on the allocated pre-training data through the target process to obtain target data; wherein the data cleaning processing includes conditional filtering processing and deduplication processing;

[0051] The second obtaining module is used to obtain a cleaned data set corresponding to the pre-training data set according to the corresponding target data in each data batch after the pre-training data set is cleaned.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0053] Memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0057] The present application provides a method, device, electronic device and storage medium for cleaning pre-training data, which determines a target process in an idle state among multiple processes; allocates a single batch of pre-training data to the target process, wherein the single batch of pre-training data is determined according to the number of parallel processes and the total number of pre-training data contained in the pre-training data set; by setting the target process, the data cleaning speed can be increased, and data cleaning can be made more efficient and fast. The target process performs data cleaning processing on the allocated pre-training data to obtain the target data; wherein the data cleaning processing includes conditional filtering processing and deduplication processing; when the pre-training data set is cleaned, the cleaned data set corresponding to the pre-training data set is obtained according to the corresponding target data in each data batch. The target process performs data cleaning processing including conditional filtering processing and deduplication processing on the allocated pre-training data to obtain the target data, and the quality of the data can be improved by combining conditional filtering processing and deduplication processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 A flowchart of a method for cleaning pre-training data provided in an embodiment of the present application;

[0060] Figure 2 A flowchart of another method for cleaning pre-training data provided in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of pre-training data provided in an embodiment of the present application;

[0062] Figure 4 A first data schematic diagram provided for an embodiment of the present application;

[0063] Figure 5 A second data schematic diagram provided for an embodiment of the present application;

[0064] Figure 6 A third data schematic diagram provided for an embodiment of the present application;

[0065] Figure 7 A schematic diagram of the first filtering data provided in an embodiment of the present application;

[0066] Figure 8 A fourth data schematic diagram provided for an embodiment of the present application;

[0067] Fig. 9 A schematic diagram of a second filtering data provided in an embodiment of the present application;

[0068] Fig.10 A fifth data schematic diagram provided for an embodiment of the present application;

[0069] Fig.11 A sixth data schematic diagram provided for an embodiment of the present application;

[0070] Fig.12 A seventh data schematic diagram provided for an embodiment of the present application;

[0071] Fig.13 A seventh data schematic diagram provided for an embodiment of the present application;

[0072] Fig.14 A data schematic diagram of a first deduplication process provided in an embodiment of the present application;

[0073] Fig.15 A schematic diagram of the second deduplication process data provided in an embodiment of the present application;

[0074] Fig.16 A schematic diagram of data segmentation for the second deduplication process provided in an embodiment of the present application;

[0075] Fig.17 A first deduplication data schematic diagram provided in an embodiment of the present application;

[0076] Fig.18 A second deduplication data schematic diagram provided in an embodiment of the present application;

[0077] Fig.19 An example diagram of the structure of a device for cleaning pre-training data provided in an embodiment of the present application;

[0078] Fig. 20 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0079] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0080] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0081] At present, data cleaning of pre-training data leads to poor data cleaning and inability to accurately remove duplicates, which affects the quality of pre-training. Since the amount of pre-training data for large models is very large, it often takes a long time to clean it, resulting in long processing time and low processing efficiency, which affects the efficiency of pre-training.

[0082] The method, device, electronic device and storage medium for cleaning pre-trained data provided in the present application can improve the quality of the cleaned data by performing conditional filtering and deduplication processing on the pre-trained data, and at the same time, by setting up multiple processes for data cleaning processing, it is possible to improve the data cleaning quality while increasing the data cleaning rate.

[0083] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0084] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0085] Figure 1 A flowchart of a method for cleaning pre-training data provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0086] S101, determining a target process in an idle state among multiple processes.

[0087] The target process in the idle state refers to the process that has not processed the pre-training data.

[0088] S102: Allocate a single batch of processing number of pre-training data to the target process, wherein the single batch processing number is determined according to the number of parallel processes and the total number of pre-training data included in the pre-training data set.

[0089] First, the number of processes is set to n. For n processes, the processing program will divide the pre-training data to be processed into 4*n batches. Each time a process is idle, a batch of data will be sent to the process. Specifically, for example, if a pre-training data contains 200 data items, and the number of program processes is set to 5, the program will divide the pre-training data set into 4*5=20 batches, with 10 data items in each batch. Then, when the target process is idle, 10 data items will be allocated to the target process.

[0090] S103, performing data cleaning processing on the allocated pre-training data through the target process to obtain target data; wherein the data cleaning processing includes conditional filtering processing and deduplication processing.

[0091] Among them, the valid field text is used to save the pre-training data.

[0092] The target data is the pre-training data after data cleaning. The conditional filtering processing is all processed by Python program code, which includes privacy information coding, special content removal, text standardization, copyright information removal, count filtering, length filtering, sensitive word filtering, language recognition and filtering, perplexity filtering, and special character ratio filtering. Deduplication processing includes MD5 deduplication, character-level N-Gram deduplication, data segmentation, and word-level N-Gram deduplication based on data segmentation, processed by Python program code, and minimum hash local sensitive hash deduplication, processed by C++ program code.

[0093] The assigned pre-training data is conditionally filtered and deduplicated through the target process.

[0094] In the embodiment of the present application, data cleaning processing is performed on the allocated pre-training data through the target process, including:

[0095] In the order of the amount of computation from small to large, the target process performs multiple conditional filtering processes on the allocated pre-training data to obtain target filtered data, wherein different conditional filtering processes correspond to different amounts of computation;

[0096] According to the deduplication rates from large to small, multiple deduplication processes are performed on the target filtered data to obtain the target data, wherein different deduplication processes correspond to different deduplication rates.

[0097] Among them, the conditional filtering process with a smaller amount of calculation is performed first to reduce the amount of data, and then the conditional filtering process with a larger amount of calculation is performed. At this time, the amount of data is reduced, thereby improving the overall filtering efficiency of the conditional filtering.

[0098] Similarly, deduplication processing with a fast deduplication rate is performed first to reduce the amount of data, and then deduplication processing with a slow deduplication rate is performed. At this time, the amount of data is reduced, thereby improving the overall deduplication efficiency of the deduplication processing.

[0099] Specifically, for conditional filtering, first filter out information that is easy to process (i.e., information with small computational complexity), such as privacy information, special content, and Copyright information (copyright information), which are easy to judge and filter. Then filter the information that needs to be counted, such as count and length. Finally, filter sensitive words, languages, special characters, and other information that needs to be compared with a preset list or involves complex calculations, thereby improving the overall cleaning efficiency.

[0100] The advantage of this setting is that this method can improve processing speed and efficiency and save computing resources while ensuring data quality by optimizing the data processing process.

[0101] In the embodiment of the present application, when the operation amount includes the first operation amount, the second operation amount, and the third operation amount, the conditional filtering process includes the first conditional filtering process, the second conditional filtering process, and the third conditional filtering process, in the order of the operation amount from small to large, the target process performs multiple conditional filtering processes on the allocated pre-training data to obtain the target filtered data, including:

[0102] According to the first operation amount, the target process performs a first condition filtering process on the allocated pre-training data to obtain first filtered data; wherein the first operation amount is less than the second operation amount, and the second operation amount is less than the third operation amount; the first operation amount corresponds to the first condition filtering process;

[0103] Performing a second condition filtering process on the first filtering data according to the second operation amount to obtain second filtering data; wherein the second operation amount corresponds to the second condition filtering process;

[0104] A third condition filtering process is performed on the second filtering data according to the third operation amount to obtain target filtering data; wherein the third operation amount corresponds to the third condition filtering process.

[0105] The first conditional filtering process includes privacy information coding, special content removal, text standardization, and copyright information removal. The idle process performs privacy information coding, special content removal, text standardization, and copyright information removal on the allocated pre-trained data to obtain the first filtered data. The order of executing the privacy information coding, special content removal, text standardization, and copyright information steps can be exchanged, but the steps in the first conditional filtering process, the second conditional filtering process, and the third conditional filtering process cannot be exchanged.

[0106] The second condition filtering process includes count filtering and length filtering. Count filtering and length filtering are performed on the first filtering data to obtain the second filtering data. There is no requirement for the order of performing the count filtering and length filtering steps.

[0107] The third condition filtering process includes sensitive word filtering, language recognition and filtering, perplexity filtering, and special character proportion filtering. The second filtering data is subjected to sensitive word filtering, language recognition and filtering, perplexity filtering, and special character proportion filtering to obtain the third data. There is no requirement for the order of performing sensitive word filtering, language recognition and filtering, perplexity filtering, and special character proportion filtering.

[0108] The advantage of such a setting is that this method can significantly improve processing efficiency and resource utilization while ensuring data processing quality by reasonably arranging the order of conditional filtering processing.

[0109] In the embodiment of the present application, when the deduplication rate includes a first deduplication rate and a second deduplication rate, and the deduplication processing includes a first deduplication processing and a second deduplication processing, deduplication processing is performed on the target filtered data according to the deduplication rates from large to small to obtain a target data set, including:

[0110] Performing a first deduplication process on the target filtered data according to a first deduplication rate to obtain first deduplication data; wherein the first deduplication process corresponds to the first deduplication rate; and the first deduplication process is to perform deduplication process using Python program code;

[0111] The first deduplication data is subjected to a second deduplication processing according to a second deduplication rate to obtain target data; the second deduplication processing corresponds to the second deduplication rate; the second deduplication processing is performed using C++ program code.

[0112] Among them, C++ is a compiled language. Python is a compiled language. The running speed of C++ program code is much faster than that of Python program code. Considering that the minimum hash local sensitive hash deduplication is a comparison between documents, the amount of calculation is large, and the Python program code is slow, it is placed at the end of the deduplication process for processing. When the previous conditional filtering and the previous deduplication steps are completed by the Python program code, the C++ program code is used to process the minimum hash local sensitive hash deduplication, which can improve the deduplication speed and efficiency.

[0113] The first deduplication process includes MD5 deduplication, character-level N-Gram deduplication, data segmentation, and word-level N-Gram deduplication based on data segmentation.

[0114] The second deduplication process includes minimum hash locality sensitive hashing deduplication.

[0115] Specifically, after obtaining the target filtering data, MD5 deduplication is first performed to calculate the text hash values ​​(MD5 values) of multiple target filtering data. For texts with the same MD5 value, they are determined to be duplicate data. Only one duplicate data is retained and the rest are removed, that is, the keep field is set to False. Through MD5 deduplication, the completely identical data in the target filtering data can be removed to obtain the first deduplication process data.

[0116] Then, the data of the first deduplication process is processed by character-level N-Gram to obtain the data of the second deduplication process. N-Gram is a text feature representation method used to capture local context information in text in order to compare the similarity between texts. N-Gram deduplication is to filter samples according to the N-Gram repetition ratio of the sample. Character-level N-Gram decomposes the sixth data into N continuous sequences, that is, the content in the field is subjected to a sliding window operation of size N according to the characters to form a fragment sequence of length N. Each fragment is called a Gram. The number of occurrences of all Grams is counted, and finally the ratio of the sum of the frequencies of Grams with a frequency greater than 1 / the sum of the frequencies of all Grams is counted as the repetition ratio. If the repetition ratio is between the minimum and maximum values ​​of the set threshold, it is determined to be duplicate data, the data is deleted, and the keep field is set to False.

[0117] In order to further analyze and process the pre-training data, the second deduplication process data is segmented, that is, the text data is split into individual words, and the results are saved in a new field column text_tokenizer to obtain the second deduplication process data segmentation. Specifically, the jieba segmentation tool can be used to segment the second deduplication process data.

[0118] The word-level N-Gram first converts the content of the text_tokenizer field of the second deduplication process data segmentation into lowercase format, and then performs a sliding window operation of size N according to the word to form a fragment sequence of length N. Each fragment is called a Gram. The number of occurrences of all Grams is counted, and finally the ratio of the sum of the frequencies of Grams with a frequency greater than 1 / the sum of the frequencies of all Grams is counted as the repetition ratio for sample filtering. If the repetition ratio is between the set minimum threshold and the maximum threshold, it is determined to be duplicate data, and the data is deleted, and the keep field is set to False to obtain the first deduplication data.

[0119] Then, the first deduplicated data is deduplicated by using the minimum hash local sensitive hashing deduplication to obtain the second deduplicated data. MinHash is a technique for approximate set similarity calculation and is widely used for fast similarity estimation in large-scale data sets. The main steps of the MinHash algorithm are as follows: 1) Set conversion into signature: For an element in a set, it is mapped to a fixed-length hash value through a hash function. Multiple hash functions are usually used to generate multiple hash values, so that a signature is obtained; 2) Select the minimum value: Select the smallest one from the generated hash values ​​as the MinHash value of the set; 3) For each set, repeat the above two steps to obtain the MinHash value of all sets. MinHash is usually used in conjunction with other techniques, such as Locality Sensitive Hashing (LSH), to form MinHash LSH (Minwise Hashing Locality-Sensitive Hashing), which clusters similar sets together, reduces the search range, and only compares sets that may have high similarity, avoiding comparing dissimilar sets, so as to quickly find similar sets in large data sets.

[0120] The advantage of such a setting is that the embodiment of the present application provides an efficient, flexible and accurate data deduplication solution by combining the advantages of different deduplication rates and programming languages.

[0121] In the embodiment of the present application, when the first condition filtering process includes privacy information coding, special content removal, text standardization, and copyright information removal, the first condition filtering process is performed on the allocated pre-training data according to the first operation amount to obtain the first filtered data, including:

[0122] Replacing the private data in the allocated pre-trained data with the preset coded data to obtain first data;

[0123] Removing special content from the first data to obtain second data; wherein the special content represents data containing one of a URL link or an HTML tag;

[0124] Performing text standardization processing on the second data to obtain third data; wherein the text standardization processing includes one of Unicode standardization processing and converting traditional Chinese characters into simplified Chinese characters;

[0125] The copyright information in the third data is removed to obtain the first filtered data.

[0126] Among them, the preset coded data is preset by the user and can be: [EMAIL], [TELEPHONE] or [MOBILEPHONE], IDNUM. Specifically, the privacy data in the allocated pre-trained data is first coded, that is: the privacy data is replaced with the preset coded data, for example, the email address characters are replaced with [EMAIL], the mobile phone number is replaced with [TELEPHONE] or [MOBILEPHONE], and the ID number is replaced with IDNUM to obtain the first data.

[0127] The special content includes URL links and HTML tags. The URL links and tag data in the first data are removed to obtain the second data.

[0128] The second data is subjected to text standardization processing, that is, the text of the second data is subjected to Unicode standardization processing and the traditional Chinese characters are converted to simplified Chinese characters, so as to obtain the third data.

[0129] The advantage of such a setting is that the embodiment of the present application significantly improves the quality and security of pre-training data through a series of data processing steps, and provides a reliable data foundation for subsequent machine learning model training.

[0130] S104: When the pre-training data set is cleaned, a cleaned data set corresponding to the pre-training data set is obtained according to the corresponding target data in each data batch.

[0131] The target data refers to the data that has been cleaned in each data batch.

[0132] After conditional filtering and deduplication processing are performed on all pre-training data, the cleaned data corresponding to each data batch is obtained to obtain the cleaned data set corresponding to the pre-training data set.

[0133] A method for cleaning pre-training data provided in an embodiment of the present application can improve the quality of the cleaned data by performing conditional filtering and deduplication processing on the pre-training data, and at the same time, combined with setting up multiple processes for data cleaning processing, it can improve the data cleaning quality while improving the data cleaning rate.

[0134] In the above embodiment of the present application, when the second condition filtering includes count filtering and length filtering, the second condition filtering process is performed on the first filtered data according to the second operation amount to obtain the second filtered data, including:

[0135] Obtaining a ratio of the number of letters in the first filtered data to the number of characters in the first filtered data, which is the letter ratio, and obtaining a ratio of the number of numbers in the first filtered data to the number of characters in the first filtered data, which is the number ratio;

[0136] If the letter ratio is within the preset letter ratio interval, and the number ratio is within the preset number ratio interval, then confirm that the mark field of the first filtering data is the first mark value, and determine the mark field of other first filtering data as the second mark value; wherein the other first filtering data is the first filtering data whose number ratio is not within the letter ratio interval, or whose number ratio is not within the number ratio interval;

[0137] Remove the data whose mark field in the first filtered data is the second mark value, and aggregate the first filtered data whose mark field in the first filtered data is the first mark value to obtain fourth data;

[0138] If the length of the fourth data is within the preset length interval, confirming that the mark field of the fourth data is the first mark value, and determining the mark field of other fourth data as the second mark value, wherein the other fourth data is the fourth data whose length is not within the preset length interval;

[0139] The data whose mark field in the fourth data is the second mark value is removed, and the data whose mark field in the fourth data is the first mark value is aggregated to obtain the second filtered data.

[0140] Among them, the first filtered data is counted and filtered, that is, the samples in the text field that do not meet the ratio of numbers and letters are removed. Because most of the characters in the data set are composed of letters and numbers, some dirty data can be removed through this step. The number of numbers and letters in the text field are counted separately, and the ratio of them to the length of the text field is used as the number ratio and the letter ratio respectively. If the ratio is between the set minimum threshold and the maximum threshold, the data meets the requirements and the keep field is set to True; otherwise, the data is deleted and the keep field is set to False to obtain the fourth data. Specifically, the dirty data is garbled code composed of numbers and letters.

[0141] The fourth data is filtered by length, and samples are filtered according to the average length of the text field. The average length is calculated based on the line break character \n segmentation sample. If the average length is between the set minimum threshold and maximum threshold, the data meets the requirements, and the keep field is set to True; otherwise, the data is deleted, and the keep field is set to False to obtain the second filtered data.

[0142] The advantage of this setting is that through a series of filtering and aggregation steps, the final data is ensured to be of high quality, high relevance and high efficiency.

[0143] In the above embodiment of the present application, when the third condition filtering includes sensitive word filtering, language identification and filtering, perplexity filtering, and special character ratio filtering, the third condition filtering processing is performed on the second filtering data according to the third operation amount to obtain the target filtering data, including:

[0144] If the second filtering data includes a sensitive word in the preset sensitive word list, confirming that the tag field of the second filtering data is the second tag value, and determining the tag field of other second filtering data as the first tag value; wherein the other second filtering data is the second filtering data that does not include the sensitive word in the preset sensitive word list;

[0145] Remove the data whose mark field is the second mark value in the second filtered data, and aggregate the data whose mark field is the first mark value in the second filtered data to obtain fifth data;

[0146] Determine the language category of the fifth data and calculate the language confidence of the fifth data according to a preset language discrimination model;

[0147] If the language category of the fifth data is a preset language category and the language confidence of the fifth data is within the language confidence interval, then confirming that the tag field of the fifth data is the first tag value;

[0148] Determine the tag field of other fifth data as the second tag value, wherein the other fifth data is fifth data whose language category is not a preset language category or whose language confidence is not within the language confidence interval;

[0149] Remove the data whose mark field in the fifth data is the second mark value, and aggregate the data whose mark field in the fifth data is the first mark value to obtain the sixth data;

[0150] Calculate the confusion of the sixth data;

[0151] If the perplexity is within the perplexity interval, confirming that the tag field of the sixth data is the first tag value;

[0152] Determine the tag field of other sixth data as the second tag value, wherein the other sixth data is the sixth data whose perplexity is not within the perplexity interval;

[0153] Remove the data whose mark field in the sixth data is the second mark value, and aggregate the data whose mark field in the sixth data is the first mark value, to obtain the seventh data;

[0154] According to the preset special character table, determining the ratio of the special characters in the seventh data to the characters of the seventh data is the special character ratio;

[0155] If the special character ratio is within the special character ratio interval, confirming that the mark field of the seventh data is the first mark value;

[0156] Determine the tag field of the other seventh data as the second tag value, wherein the other seventh data is the seventh data whose special character ratio is not within the special character ratio interval;

[0157] The data whose mark field in the seventh data is the second mark value is removed, and the data whose mark field in the seventh data is the first mark value is aggregated to obtain the target filtered data.

[0158] Among them, the second filtered data is filtered for sensitive words: first, sensitive words are preset, and the sensitive word list is stored in the form of a dictionary tree, and the type of sensitive words is stored at the same time. Then, the preset sensitive word file is used to filter the text field. If it contains sensitive words, the data is deleted, and the keep field is set to False, thereby obtaining the fifth data.

[0159] The fifth data is subjected to language identification and filtering to obtain the sixth data: a preset language category is set, and the language type of the fifth data is determined by using the language judgment model of fasttext, and the confidence of each language in the fifth data is calculated. If the fifth data is of the preset language category and the confidence is greater than or equal to the set threshold, the data meets the requirements and the keep field is set to True; otherwise, the data is deleted and the keep field is set to False, thereby obtaining the sixth data.

[0160] The sixth data is filtered by perplexity to obtain the seventh data: the perplexity of the sixth data is calculated by the python toolkit. If the perplexity is between the set threshold minimum and maximum values, the data meets the requirements and the keep field is set to True; otherwise, the data is deleted and the keep field is set to False, thereby obtaining the seventh data.

[0161] The seventh data is filtered by special characters to obtain the target filtered data: first, a special character list is preset in the system, and then the preset special character list is used to filter the text field, and samples in the text field that do not meet the special character ratio are removed. If the special character ratio is between the set threshold minimum and maximum values, the data meets the requirements, and the keep field is set to True; otherwise, the data is deleted, and the keep field is set to False, thereby obtaining the target filtered data.

[0162] Figure 2 A flowchart of another method for cleaning pre-training data provided in an embodiment of the present application. Figure 2 As shown, the method includes:

[0163] Input the pre-trained data to be cleaned, and then perform conditional filtering and deduplication processing on the pre-trained data to be cleaned to obtain the cleaned pre-trained data. The conditional filtering includes: privacy information coding, special content removal, text standardization, and copyright information removal, count filtering, length filtering, sensitive word filtering, language recognition and filtering, confusion filtering, and special character ratio filtering. The deduplication processing includes MD5 deduplication, character-level N-Gram deduplication, data segmentation, word-level N-Gram deduplication, and MinHash LSH deduplication. At the same time, multiple processes are set to perform conditional filtering and deduplication processing in parallel.

[0164] Set the minimum min_ratio of the repetition ratio of the count filter to 0.3 and the maximum max_ratio to 0.5, set the retained languages ​​to Chinese, English, Japanese and Korean, set the minimum threshold of the perplexity to 0 and the maximum threshold to 10000, set the minimum threshold of the special character ratio to 0 and the maximum threshold to 0.1, set the length of N-Gram ngram_length to 2, the minimum min_ratio of the repetition ratio to 0.0 and the maximum max_ratio to 0.3, the number of random permutation functions for MinHash LSH deduplication num_perm to 250, and the threshold to 0.87.

[0165] The sample data set (i.e. the pre-trained data to be cleaned) is test.jsonl. Figure 3 A schematic diagram of pre-training data provided in an embodiment of the present application, such as Figure 3 As shown, the text character represents the pre-training data. Specifically, in the figure, {"text": the email address is abcd1234@567.com}, then the email address abcd1234@567.com is the pre-training data, and the rest are similar.

[0166] After the privacy data of the pre-training data is coded, the first data is obtained. Figure 4 A first data schematic diagram provided for an embodiment of the present application, such as Figure 4 As shown, the first data replaces the email information in the pre-training data with [EMAIL].

[0167] Remove the special content from the first data to obtain the second data. Figure 5 A second data schematic diagram provided for an embodiment of the present application, such as Figure 5 As shown, the URL link in the first data is deleted to obtain the second data.

[0168] Perform text normalization on the second data to obtain the third data, Figure 6 A third data schematic diagram provided for an embodiment of the present application, such as Figure 6As shown, the second data is converted from traditional Chinese to simplified Chinese to obtain the third data.

[0169] After the Copyright (i.e., copyright information) information in the third data is removed, the first filtered data is obtained. Figure 7 A first filtering data schematic diagram provided in an embodiment of the present application, such as Figure 7 As shown, the Copyright information in the third data is removed, and the keep (ie, mark field) data is set to False to obtain the first filtered data.

[0170] After performing count filtering on the first filtered data, the fourth data is obtained. The count filtering includes letter ratio filtering and number ratio filtering. Figure 8 A fourth data schematic diagram provided for an embodiment of the present application, such as Figure 8 As shown, the keep (ie, the mark field) of the first filtered data that does not meet the letter proportion interval or the number proportion interval is set to False, and is removed to obtain the fourth data.

[0171] After length filtering the fourth data, the second filtered data is obtained. Fig. 9 A second filtering data schematic diagram provided in an embodiment of the present application, such as Fig. 9 As shown, the fourth data that does not meet the length ratio interval is removed, keep (ie, the mark field) is set to False, and the second filtered data is obtained.

[0172] The second filtered data is filtered for sensitive words to obtain fifth data. Fig.10 A fifth data schematic diagram provided in an embodiment of the present application, such as Fig.10 As shown, the keep (ie, the mark field) of the second filtering data containing the sensitive word is set to False, and it is filtered out to obtain the fifth data.

[0173] After language recognition and filtering of the fifth data, the sixth data is obtained. Fig.11 A sixth data schematic diagram provided for an embodiment of the present application, such as Fig.11 As shown, the language category of the fifth data is identified, the keep (i.e., mark field) of the fifth data whose language category does not belong to Chinese, English, Japanese, and Korean, or whose confidence is not within the confidence interval is set to False, and it is filtered out to obtain the sixth data.

[0174] After filtering the sixth data by perplexity, the seventh data is obtained. Fig.12 The seventh data schematic diagram provided in the embodiment of the present application is as follows: Fig.12 As shown, the perplexity of the sixth data is obtained, the keep of the sixth data that does not meet the perplexity interval is set to false, and it is removed to obtain the seventh data.

[0175] After filtering the seventh data by the proportion of special characters, the target filtered data is obtained. Fig.13 The seventh data schematic diagram provided in the embodiment of the present application is as follows: Fig.13 As shown, the special character ratio of the seventh data is obtained, the data that does not meet the special character interval is set to False, and it is removed to obtain the target filtered data.

[0176] After performing MD5 deduplication on the target filtered data, the first deduplication process data is obtained. Fig.14 A schematic diagram of the first deduplication process data provided in the embodiment of the present application is shown in FIG. Fig.14 As shown, Figure 3 The keep of the 15th data which is exactly the same as the 11th data is set to False, and the 15th data is removed to obtain the first deduplication process data.

[0177] After performing character-level N-Gram deduplication on the first deduplication process data, the second deduplication process data is obtained. Fig.15 A schematic diagram of the second deduplication process data provided in the embodiment of the present application, such as Fig.15 As shown, the keep of the first deduplication process data whose repetition ratio is between the set minimum and maximum threshold values ​​is set to False, and then the data is filtered to obtain the second deduplication process data.

[0178] The second process data is subjected to word segmentation and word N-Gram filtering in turn to obtain the first deduplicated data. Fig.16 A schematic diagram of the second process data segmentation provided in an embodiment of the present application, such as Fig.16 As shown, the results of the tokenization of all data text fields are saved in the new field text_tokenizer to obtain the second process data tokenization. Fig.17 A first deduplication data schematic diagram provided in an embodiment of the present application is as follows: Fig.17 As shown, after deduplication through word-level N-Gram, the data with a repetition ratio of the text_tokenizer field within the threshold in the second process data segmentation is set to False, and is removed to obtain the first deduplication data.

[0179] After performing MinHash LSH deduplication on the first deduplication data, second deduplication data is obtained. Fig.18 A second schematic diagram of deduplication data provided in an embodiment of the present application.

[0180] Another method for cleaning pre-training data provided in an embodiment of the present application can improve the accuracy and efficiency of cleaning large model pre-training data, making the entire process more efficient, flexible and reliable, thereby improving the quality and efficiency of subsequent large model pre-training.

[0181] Fig.19 This is a diagram showing an example of the structure of a device for cleaning pre-training data provided in an embodiment of the present application. Fig.19 As shown, the pre-training data cleaning device 190 includes: a determination module 1901, an allocation module 1902, a first obtaining module 1903, and a second obtaining module 1904. Among them:

[0182] A determination module 1901 is used to determine a target process in an idle state among multiple processes;

[0183] An allocation module 1902 is used to allocate a single batch of pre-training data to the target process, wherein the single batch of pre-training data is determined according to the number of parallel processes and the total number of pre-training data included in the pre-training data set;

[0184] The first obtaining module 1903 is used to perform data cleaning processing on the allocated pre-training data through the target process to obtain target data; wherein the data cleaning processing includes conditional filtering processing and deduplication processing;

[0185] The second obtaining module 1904 is used to obtain a cleaned data set corresponding to the pre-training data set according to the corresponding target data in each data batch after the pre-training data set is cleaned.

[0186] In one embodiment, the first obtaining module 1903 is used to:

[0187] In the order of the amount of computation from small to large, the target process performs multiple conditional filtering processes on the allocated pre-training data to obtain target filtered data, wherein different conditional filtering processes correspond to different amounts of computation;

[0188] According to the deduplication rates from large to small, multiple deduplication processes are performed on the target filtered data to obtain the target data, wherein different deduplication processes correspond to different deduplication rates.

[0189] In one embodiment, the first obtaining module 1903 is further used for:

[0190] According to the first operation amount, the target process performs a first condition filtering process on the allocated pre-training data to obtain first filtered data; wherein the first operation amount is less than the second operation amount, and the second operation amount is less than the third operation amount; the first operation amount corresponds to the first condition filtering process;

[0191] Performing a second condition filtering process on the first filtering data according to the second operation amount to obtain second filtering data; wherein the second operation amount corresponds to the second condition filtering process;

[0192] A third condition filtering process is performed on the second filtering data according to the third operation amount to obtain target filtering data; wherein the third operation amount corresponds to the third condition filtering process.

[0193] In one embodiment, the first obtaining module 1903 is further used for:

[0194] Replacing the private data in the allocated pre-trained data with the preset coded data to obtain first data;

[0195] Removing special content from the first data to obtain second data; wherein the special content represents data containing one of a URL link or an HTML tag;

[0196] Performing text standardization processing on the second data to obtain third data; wherein the text standardization processing includes one of Unicode standardization processing and converting traditional Chinese characters into simplified Chinese characters;

[0197] The copyright information in the third data is removed to obtain the first filtered data.

[0198] In one embodiment, the first obtaining module 1903 is further used for:

[0199] Obtaining a ratio of the number of letters in the first filtered data to the number of characters in the first filtered data, which is the letter ratio, and obtaining a ratio of the number of numbers in the first filtered data to the number of characters in the first filtered data, which is the number ratio;

[0200] If the letter ratio is within the preset letter ratio interval, and the number ratio is within the preset number ratio interval, then confirm that the mark field of the first filtering data is the first mark value, and determine the mark field of other first filtering data as the second mark value; wherein the other first filtering data is the first filtering data whose number ratio is not within the letter ratio interval, or whose number ratio is not within the number ratio interval;

[0201] Remove the data whose mark field in the first filtered data is the second mark value, and aggregate the first filtered data whose mark field in the first filtered data is the first mark value to obtain fourth data;

[0202] If the length of the fourth data is within the preset length interval, confirming that the mark field of the fourth data is the first mark value, and determining the mark field of other fourth data as the second mark value, wherein the other fourth data is the fourth data whose length is not within the preset length interval;

[0203] The data whose mark field in the fourth data is the second mark value is removed, and the data whose mark field in the fourth data is the first mark value is aggregated to obtain the second filtered data.

[0204] In one embodiment, the first obtaining module 1603 is further used for:

[0205] If the second filtering data includes a sensitive word in the preset sensitive word list, confirming that the tag field of the second filtering data is the second tag value, and determining the tag field of other second filtering data as the first tag value; wherein the other second filtering data is the second filtering data that does not include the sensitive word in the preset sensitive word list;

[0206] Remove the data whose mark field is the second mark value in the second filtered data, and aggregate the data whose mark field is the first mark value in the second filtered data to obtain fifth data;

[0207] Determine the language category of the fifth data and calculate the language confidence of the fifth data according to a preset language discrimination model;

[0208] If the language category of the fifth data is a preset language category and the language confidence of the fifth data is within the language confidence interval, then confirming that the tag field of the fifth data is the first tag value;

[0209] Determine the tag field of other fifth data as the second tag value, wherein the other fifth data is fifth data whose language category is not a preset language category or whose language confidence is not within the language confidence interval;

[0210] Remove the data whose mark field in the fifth data is the second mark value, and aggregate the data whose mark field in the fifth data is the first mark value to obtain the sixth data;

[0211] Calculate the confusion of the sixth data;

[0212] If the perplexity is within the perplexity interval, confirming that the tag field of the sixth data is the first tag value;

[0213] Determine the tag field of other sixth data as the second tag value, wherein the other sixth data is the sixth data whose perplexity is not within the perplexity interval;

[0214] Remove the data whose mark field in the sixth data is the second mark value, and aggregate the data whose mark field in the sixth data is the first mark value, to obtain the seventh data;

[0215] According to the preset special character table, determining the ratio of the special characters in the seventh data to the characters of the seventh data is the special character ratio;

[0216] If the special character ratio is within the special character ratio interval, confirming that the mark field of the seventh data is the first mark value;

[0217] Determine the tag field of the other seventh data as the second tag value, wherein the other seventh data is the seventh data whose special character ratio is not within the special character ratio interval;

[0218] The data whose mark field in the seventh data is the second mark value is removed, and the data whose mark field in the seventh data is the first mark value is aggregated to obtain the target filtered data.

[0219] The device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0220] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0221] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0222] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrations. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, Digital Video Discs (DVD)), or semiconductor media (eg, solid state disks (SSD)).

[0223] Fig. 20 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 20 As shown, the electronic device 200 includes:

[0224] The electronic device 200 may include a processor 2001 with one or more processing cores, a memory 2002 with one or more computer-readable storage media, a communication component 2003 and other components. The processor 2001, the memory 2002 and the communication component 2003 are connected via a bus 2004.

[0225] In the specific implementation process, at least one processor 2001 executes the computer execution instructions stored in the memory 2002, so that at least one processor 2001 executes the above method for cleaning pre-training data.

[0226] The specific implementation process of the processor 2001 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0227] In the above Fig. 20In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0228] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0229] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0230] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which, when executed by a processor, implements the steps in any of the above-mentioned methods for cleaning pre-training data.

[0231] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0232] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0233] To this end, an embodiment of the present application provides a computer-readable storage medium, which stores multiple instructions, and the instructions can be loaded by a processor to execute the steps in any method of cleaning pre-training data provided in the embodiment of the present application.

[0234] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0235] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program comprises computer instructions stored in a computer-readable storage medium.

[0236] Since the instructions stored in the storage medium can execute the steps in any method for cleaning pre-trained data provided in the embodiments of the present application, the beneficial effects that can be achieved by any method for cleaning pre-trained data provided in the embodiments of the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0237] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0238] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for cleaning pre-training data, characterized in that: Using multiple processes to perform data cleaning in parallel, the method includes: Determine a target process in an idle state among the multiple processes; Allocating a single batch processing number of pre-training data to the target process, wherein the single batch processing number is determined according to the number of parallel processes and the total number of pre-training data included in the pre-training data set; Performing data cleaning processing on the allocated pre-training data through the target process to obtain target data; wherein the data cleaning processing includes conditional filtering processing and deduplication processing; When the pre-training data set is cleaned, a cleaned data set corresponding to the pre-training data set is obtained according to the corresponding target data in each data batch.

2. The method according to claim 1, characterized in that The data cleaning process of the allocated pre-training data by the target process includes: In the order of the amount of computation from small to large, the target process performs a plurality of conditional filtering processes on the allocated pre-training data to obtain target filtered data, wherein different conditional filtering processes correspond to different amounts of computation; According to the deduplication rates from large to small, a plurality of deduplication processes are performed on the target filtered data to obtain the target data, wherein different deduplication processes correspond to different deduplication rates.

3. The method according to claim 2, characterized in that When the operation amount includes a first operation amount, a second operation amount, and a third operation amount, and the conditional filtering process includes a first conditional filtering process, a second conditional filtering process, and a third conditional filtering process, the target process performs a plurality of conditional filtering processes on the allocated pre-training data in the order of the operation amount from small to large, to obtain target filtered data, including: According to the first operation amount, the target process performs the first condition filtering process on the allocated pre-training data to obtain first filtered data; wherein the first operation amount is smaller than the second operation amount, and the second operation amount is smaller than the third operation amount; the first operation amount corresponds to the first condition filtering process; Perform a second conditional filtering process on the first filtered data according to the second operation amount to obtain second filtered data; wherein the second operation amount corresponds to the second conditional filtering process; A third condition filtering process is performed on the second filtering data according to the third operation amount to obtain target filtering data; wherein the third operation amount corresponds to the third condition filtering process.

4. The method according to claim 2, characterized in that: When the deduplication rate includes a first deduplication rate and a second deduplication rate, and the deduplication process includes a first deduplication process and a second deduplication process, performing deduplication process on the target filtered data according to the deduplication rates from large to small to obtain the target data set includes: Performing a first deduplication process on the target filtered data according to the first deduplication rate to obtain first deduplication data; wherein the first deduplication process corresponds to the first deduplication rate; and the first deduplication process is to perform deduplication process using Python program code; The first deduplication data is subjected to a second deduplication processing according to the second deduplication rate to obtain the target data; the second deduplication processing corresponds to the second deduplication rate; the second deduplication processing is performed using C++ program code.

5. The method according to claim 3, characterized in that: When the first conditional filtering process includes privacy information coding, special content removal, text standardization, and copyright information removal, performing the first conditional filtering process on the allocated pre-training data according to the first operation amount to obtain first filtered data includes: Replacing the privacy data in the allocated pre-trained data with the preset coded data to obtain first data; Removing special content from the first data to obtain second data; wherein the special content represents data containing one of a URL link or an HTML tag; Performing text standardization processing on the second data to obtain third data; wherein the text standardization processing includes one of Unicode standardization processing and converting traditional Chinese characters to simplified Chinese characters; The copyright information in the third data is removed to obtain first filtered data.

6. The method according to claim 3, characterized in that When the second condition filtering includes count filtering and length filtering, performing the second condition filtering process on the first filtered data according to the second operation amount to obtain the second filtered data includes: Obtaining a ratio of the number of letters in the first filtering data to the number of characters in the first filtering data, which is the letter ratio, and obtaining a ratio of the number of numbers in the first filtering data to the number of characters in the first filtering data, which is the number ratio; If the letter ratio is within the preset letter ratio interval, and the number ratio is within the preset number ratio interval, confirm that the mark field of the first filtering data is the first mark value, and determine the mark field of other first filtering data as the second mark value; wherein the other first filtering data is the first filtering data whose number ratio is not within the letter ratio interval, or whose number ratio is not within the number ratio interval; Removing data whose mark field in the first filtered data has a second mark value, and aggregating the first filtered data whose mark field in the first filtered data has a first mark value, to obtain fourth data; If the length of the fourth data is within the preset length interval, confirming that the tag field of the fourth data is the first tag value, and determining the tag field of other fourth data as the second tag value, wherein the other fourth data is the fourth data whose length is not within the preset length interval; The data whose mark field in the fourth data is the second mark value is removed, and the data whose mark field in the fourth data is the first mark value is aggregated to obtain the second filtered data.

7. The method according to claim 3, characterized in that When the third condition filtering includes sensitive word filtering, language identification and filtering, perplexity filtering, and special character ratio filtering, the third condition filtering processing is performed on the second filtered data according to the third operation amount to obtain target filtered data, including: If the second filtering data includes a sensitive word in the preset sensitive word list, confirming that the tag field of the second filtering data is the second tag value, and determining the tag field of other second filtering data as the first tag value; wherein the other second filtering data is the second filtering data that does not include the sensitive word in the preset sensitive word list; Removing data whose mark field in the second filtered data has a second mark value, and aggregating data whose mark field in the second filtered data has a first mark value, to obtain fifth data; Determine the language category of the fifth data and calculate the language confidence of the fifth data according to a preset language discrimination model; If the language category of the fifth data is a preset language category and the language confidence of the fifth data is within the language confidence interval, then confirming that the tag field of the fifth data is the first tag value; Determine the tag field of other fifth data as the second tag value, wherein the other fifth data is fifth data whose language category is not a preset language category or whose language confidence is not within the language confidence interval; Remove the data whose mark field in the fifth data is the second mark value, and aggregate the data whose mark field in the fifth data is the first mark value to obtain sixth data; Calculate the confusion of the sixth data; If the perplexity is within the perplexity interval, confirming that the tag field of the sixth data is the first tag value; Determine the tag field of other sixth data as the second tag value, wherein the other sixth data is the sixth data whose perplexity is not within the perplexity interval; Remove the data whose mark field in the sixth data is the second mark value, and aggregate the data whose mark field in the sixth data is the first mark value to obtain the seventh data; According to a preset special character table, determining the ratio of the special characters in the seventh data to the characters in the seventh data as the special character ratio; If the special character ratio is within the special character ratio interval, confirming that the mark field of the seventh data is the first mark value; Determine the tag field of other seventh data as the second tag value, wherein the other seventh data is the seventh data whose special character ratio is not within the special character ratio interval; The data whose mark field in the seventh data is the second mark value is removed, and the data whose mark field in the seventh data is the first mark value is aggregated to obtain the target filtered data.

8. A device for cleaning pre-training data, characterized in that: include: A determination module, used to determine a target process in an idle state among the multiple processes; An allocation module, used to allocate a single batch processing number of pre-training data to the target process, wherein the single batch processing number is determined according to the number of parallel processes and the total number of pre-training data included in the pre-training data set; A first obtaining module is used to perform data cleaning processing on the allocated pre-training data through the target process to obtain target data; wherein the data cleaning processing includes conditional filtering processing and deduplication processing; The second obtaining module is used to obtain a cleaned data set corresponding to the pre-training data set according to the corresponding target data in each data batch when the pre-training data set is cleaned.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images