Duplicate removal method for pre-training data, electronic equipment, storage medium and program product
Through parallel processing and double deduplication methods, the problem of low deduplication efficiency of pre-trained data is solved, efficient and accurate deduplication processing is achieved, and the efficiency and quality of data deduplication are improved.
Patent Information
- Application Number
- CN202411391126.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-03
AI Technical Summary
The deduplication efficiency of pre-training data in the prior art is low, and large-scale data cannot be effectively processed, resulting in wasted time and resources, and the deduplication is not thorough, so accurate deduplication cannot be achieved.
Using parallel processing strategy, the pre-trained data is divided into multiple data batches, and the corresponding data batch is allocated to each processing process, and the double deduplication process is performed, including deduplication between documents and within documents, and the global data hashing is used. , Minimum hash locally sensitive hash, character-level and word-level text deduplication to improve the accuracy and efficiency of deduplication.
Through parallel processing and dual deduplication methods, the efficiency and accuracy of pre-trained data deduplication are significantly improved, time and resource waste are reduced, and the quality and reliability of the target data set after deduplication are ensured.
Smart Images
Figure CN119250054A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method for deduplicating pre-training data, an electronic device, a storage medium, and a program product. Background Art
[0002] Pre-training data refers to data that has been collected before training the model. These data may come from different data sources and may contain noise, missing values, outliers and other issues. Large-scale high-quality corpora are the key "nutrients" for training large language models. These corpora provide a global knowledge system that can improve the understanding and generation quality of language models, while also supporting a variety of application scenarios. High-quality text has a very important impact on the training and performance of large language models. In order to ensure the performance and stability of the model, the pre-training data needs to be cleaned. Therefore, the deduplication method of pre-training data has become a promising direction.
[0003] In the prior art, the method for deduplication of pre-training data mainly uses a single inter-document deduplication method or an intra-document deduplication method in a single process mode to achieve deduplication of the pre-training data.
[0004] In the prior art, pre-training data is processed through a single process and a single deduplication method. When faced with a huge amount of pre-training data, it takes too much time and resources to perform deduplication processing, and the data deduplication is not thorough and accurate deduplication cannot be achieved. There is a technical problem of low efficiency of pre-training data deduplication. Summary of the invention
[0005] The embodiments of the present application provide a method for deduplicating pre-training data, an electronic device, a storage medium, and a program product to achieve the technical effect of improving the efficiency of deduplicating pre-training data.
[0006] In a first aspect, the present application provides a method for deduplication of pre-training data, comprising:
[0007] According to the pre-training data set, a plurality of pre-training data to be processed are obtained;
[0008] According to the total amount of pre-training data, determine the corresponding number of parallel processing processes, and divide the pre-training data into multiple data batches according to the number of parallel processing processes and the total amount of pre-training data, wherein the number of parallel processing processes is used to indicate the amount of data that can be processed in parallel by the system at the same time, and each data batch contains at least one pre-training data;
[0009] According to the parallel processing strategy, a corresponding data batch is allocated to each processing process, and each processing process is controlled to perform double deduplication processing on the pre-training data of each data batch to obtain the processed target data, wherein the double deduplication processing includes deduplication between documents and deduplication within documents;
[0010] After ensuring that each processing process has completed double deduplication processing, the corresponding target data in each data batch is aggregated to obtain the deduplicated target data set.
[0011] Optionally, deduplication between documents includes global data hash deduplication and minimum hash local sensitive hash deduplication, and deduplication within documents includes character-level text deduplication and word-level text deduplication. For each processing process, double deduplication processing is performed on each batch of pre-training data to obtain processed target data, including:
[0012] For each processing process, a hash value corresponding to each pre-training data is obtained, and according to the hash value, global data hash deduplication is performed on the multiple pre-training data to remove duplicate pre-training data, thereby obtaining multiple first data after deduplication processing, wherein each pre-training data corresponds to a hash value;
[0013] Decomposing a text field corresponding to each first data into a plurality of first segment sequences, and performing character-level text deduplication on the first data according to the first segment sequences, to obtain a plurality of second data after the character-level text deduplication processing;
[0014] Perform data segmentation on each second data to obtain multiple independent words, and save each independent word in a text segmentation column;
[0015] According to the text segmentation column, perform minimum hash local sensitive hashing deduplication on multiple independent words to obtain multiple third data after the minimum hash local sensitive hashing deduplication processing;
[0016] The text field corresponding to each third data is decomposed into a plurality of second segment sequences, and word-level text deduplication is performed on the third data according to the second segment sequences to obtain target data after word-level text deduplication processing.
[0017] Optionally, global data hash deduplication is performed on the plurality of pre-training data according to the hash value to remove duplicate pre-training data, and a plurality of first data after deduplication processing is obtained, including:
[0018] For each pre-training data, obtain the corresponding index number;
[0019] The pre-trained data with the same hash value are regarded as duplicate data, and the first marking character corresponding to the pre-trained data with the smallest index number value in the duplicate data is kept unchanged, and the first marking character corresponding to the remaining duplicate data is changed to the second marking character to remove the remaining duplicate data, wherein the second marking character is used to indicate the removal of duplicate data;
[0020] The pre-training data with the first labeled character are aggregated and used as the first data.
[0021] Optionally, according to the first segment sequence, character-level text deduplication is performed on the first data to obtain a plurality of second data after character-level text deduplication processing, including:
[0022] For each first data, determining the occurrence frequency of each first segment sequence;
[0023] Summarizing the number of first target segment sequences whose occurrence frequency exceeds the first threshold, and determining a first repetition ratio of each first data according to a ratio of the number of the first target segment sequences to the first segment sequences;
[0024] comparing the first repetition ratio with a second threshold, wherein the second threshold includes a first maximum threshold and a first minimum threshold;
[0025] If the first repetition ratio is between the first maximum threshold and the first minimum threshold, determining that the first target segment sequence is a first repetitive sequence, and changing the first marking character of the first data corresponding to the first repetitive sequence to the second marking character to remove the first data corresponding to the first repetitive sequence;
[0026] The first data with the first marking character are aggregated and used as the second data.
[0027] Optionally, according to the text segmentation column, a minimum hash local sensitive hashing method is performed on multiple independent words to obtain multiple third data after the minimum hash local sensitive hashing method is performed, including:
[0028] According to the minimum hash local sensitive hash algorithm, confirm the text similarity corresponding to the text word segmentation column between each second data and each remaining second data;
[0029] comparing the text similarity with a third threshold;
[0030] If the text similarity exceeds a third threshold, determining that the current second data and the remaining second data are similar data;
[0031] Obtaining index numbers corresponding to the current second data and the remaining second data, keeping the first marking character corresponding to the second data with the smallest index number value among the similar data unchanged, and changing the first marking character of the remaining similar data to the second marking character, so as to remove the remaining similar data;
[0032] The second data with the first marking character are aggregated and used as the third data.
[0033] Optionally, performing word-level text deduplication on the third data according to the second segment sequence to obtain target data after word-level text deduplication processing includes:
[0034] For each text word segmentation column corresponding to each third data, determine the occurrence frequency of each second segment sequence;
[0035] Summarizing the number of second target segment sequences whose occurrence frequency exceeds a fourth threshold, and determining a second repetition ratio of each third data according to a ratio of the number of the second target segment sequences to the number of the second segment sequences;
[0036] comparing the second repetition ratio with a fifth threshold, wherein the fifth threshold includes a second maximum threshold and a second minimum threshold;
[0037] If the second repetition ratio is between the second maximum threshold and the second minimum threshold, determining that the second target segment sequence is a second repetitive sequence, and changing the first identification character of the third data corresponding to the second repetitive sequence to the second identification character to remove the third data corresponding to the second repetitive sequence;
[0038] The third data with the first marking character are summarized and used as the target data.
[0039] Optionally, for each processing process, double deduplication processing is performed on each batch of pre-training data, and before obtaining the processed target data, the following steps are also included:
[0040] According to each processing process, a corresponding first marking character is added to the pre-training data in each data batch, wherein the first marking character is used to indicate that the pre-training data is retained.
[0041] In a second aspect, the present application provides a device for removing duplicate data from pre-training data, comprising:
[0042] An acquisition module, used for acquiring a plurality of pre-training data to be processed according to the pre-training data set;
[0043] A first processing module, used to determine the corresponding number of parallel processing processes according to the total amount of pre-training data, and divide the pre-training data into multiple data batches according to the number of parallel processing processes and the total amount of pre-training data, wherein the number of parallel processing processes is used to indicate the amount of data that can be processed in parallel by the system at the same time, and each data batch contains at least one pre-training data;
[0044] A control module is used to allocate a corresponding data batch to each processing process according to a parallel processing strategy, and control each processing process to perform double deduplication processing on the pre-training data of each data batch to obtain processed target data, wherein the double deduplication processing includes deduplication between documents and deduplication within documents;
[0045] The second processing module is used to aggregate the corresponding target data in each data batch after determining that each processing process has completed the double deduplication processing, and obtain the deduplicated target data set.
[0046] Optionally, the control module is further configured to:
[0047] For each processing process, a hash value corresponding to each pre-training data is obtained, and according to the hash value, global data hash deduplication is performed on the multiple pre-training data to remove duplicate pre-training data, thereby obtaining multiple first data after deduplication processing, wherein each pre-training data corresponds to a hash value;
[0048] Decomposing a text field corresponding to each first data into a plurality of first segment sequences, and performing character-level text deduplication on the first data according to the first segment sequences, to obtain a plurality of second data after the character-level text deduplication processing;
[0049] Perform data segmentation on each second data to obtain multiple independent words, and save each independent word in a text segmentation column;
[0050] According to the text segmentation column, perform minimum hash local sensitive hashing deduplication on multiple independent words to obtain multiple third data after the minimum hash local sensitive hashing deduplication processing;
[0051] The text field corresponding to each third data is decomposed into a plurality of second segment sequences, and word-level text deduplication is performed on the third data according to the second segment sequences to obtain target data after word-level text deduplication processing.
[0052] Optionally, the control module is further configured to:
[0053] For each pre-training data, obtain the corresponding index number;
[0054] The pre-trained data with the same hash value are regarded as duplicate data, and the first marking character corresponding to the pre-trained data with the smallest index number value in the duplicate data is kept unchanged, and the first marking character corresponding to the remaining duplicate data is changed to the second marking character to remove the remaining duplicate data, wherein the second marking character is used to indicate the removal of duplicate data;
[0055] The pre-training data with the first labeled character are aggregated and used as the first data.
[0056] Optionally, the control module is further configured to:
[0057] For each first data, determining the occurrence frequency of each first segment sequence;
[0058] Summarizing the number of first target segment sequences whose occurrence frequency exceeds the first threshold, and determining a first repetition ratio of each first data according to a ratio of the number of the first target segment sequences to the first segment sequences;
[0059] comparing the first repetition ratio with a second threshold, wherein the second threshold includes a first maximum threshold and a first minimum threshold;
[0060] If the first repetition ratio is between the first maximum threshold and the first minimum threshold, determining that the first target segment sequence is a first repetitive sequence, and changing the first marking character of the first data corresponding to the first repetitive sequence to the second marking character to remove the first data corresponding to the first repetitive sequence;
[0061] The first data with the first marking character are aggregated and used as the second data.
[0062] Optionally, the control module is further configured to:
[0063] According to the minimum hash local sensitive hash algorithm, confirm the text similarity corresponding to the text word segmentation column between each second data and each remaining second data;
[0064] comparing the text similarity with a third threshold;
[0065] If the text similarity exceeds a third threshold, determining that the current second data and the remaining second data are similar data;
[0066] Obtaining index numbers corresponding to the current second data and the remaining second data, keeping the first marking character corresponding to the second data with the smallest index number value among the similar data unchanged, and changing the first marking character of the remaining similar data to the second marking character, so as to remove the remaining similar data;
[0067] The second data with the first marking character are aggregated and used as the third data.
[0068] Optionally, the control module is further configured to:
[0069] For each text word segmentation column corresponding to each third data, determine the occurrence frequency of each second segment sequence;
[0070] Summarizing the number of second target segment sequences whose occurrence frequency exceeds a fourth threshold, and determining a second repetition ratio of each third data according to a ratio of the number of the second target segment sequences to the number of the second segment sequences;
[0071] comparing the second repetition ratio with a fifth threshold, wherein the fifth threshold includes a second maximum threshold and a second minimum threshold;
[0072] If the second repetition ratio is between the second maximum threshold and the second minimum threshold, determining that the second target segment sequence is a second repetitive sequence, and changing the first identification character of the third data corresponding to the second repetitive sequence to the second identification character to remove the third data corresponding to the second repetitive sequence;
[0073] The third data with the first marking character are summarized and used as the target data.
[0074] Optionally, the control module is further configured to:
[0075] According to each processing process, a corresponding first marking character is added to the pre-training data in each data batch, wherein the first marking character is used to indicate that the pre-training data is retained.
[0076] In a third aspect, the present application provides a device for deduplicating pre-training data, comprising:
[0077] Processor and memory;
[0078] Memory stores computer-executable instructions;
[0079] The processor executes the computer-executable instructions stored in the memory, so that the processor executes various possible implementations of the first aspect as described above.
[0080] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement various possible implementation methods as in the first aspect.
[0081] In a fifth aspect, the present application provides a computer program product, which, when executed by a processor, is used to implement various possible implementations as in the first aspect.
[0082] The present application provides a method for deduplicating pre-training data, an electronic device, a storage medium and a program product, which obtain a plurality of pre-training data to be processed according to a pre-training data set; determine the corresponding number of parallel processing processes according to the total amount of pre-training data, and divide the pre-training data into a plurality of data batches according to the number of parallel processing processes and the total amount of pre-training data; allocate a corresponding data batch to each processing process according to a parallel processing strategy, and control each processing process to perform double deduplication processing on the pre-training data of each data batch to obtain processed target data; after determining that each processing process has completed the double deduplication processing, summarize the corresponding target data in each data batch to obtain a deduplicated target data set, so that by setting a parallel processing strategy, when facing a data set with a large amount of data, the corresponding number of processes can be determined according to the amount of data to complete the deduplication operation in parallel, thereby improving the flexibility, convenience and speed of the data deduplication process, reducing the waste of time and resources during the deduplication processing, and at the same time, by setting a double deduplication method to perform deduplication processing on the pre-training data, the accuracy and reliability of the deduplication target data set are improved, the technical problem of low deduplication efficiency of pre-training data is solved, and the technical effect of improving the deduplication efficiency of pre-training data is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0083] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0084] Figure 1 The process of the method for deduplication of pre-trained data provided in the embodiment of the present application Figure 1 ;
[0085] Figure 2 A schematic diagram of a pre-training data set provided in an embodiment of the present application;
[0086] Figure 3 The process of the method for deduplication of pre-trained data provided in the embodiment of the present application Figure 2 ;
[0087] Figure 4 A schematic diagram of data after word segmentation provided in an embodiment of the present application;
[0088] Figure 5 The process of the method for deduplication of pre-trained data provided in the embodiment of the present application Figure 3 ;
[0089] Figure 6 A schematic diagram of first data provided in an embodiment of the present application;
[0090] Figure 7 A schematic diagram of second data provided in an embodiment of the present application;
[0091] Figure 8 The process of the method for deduplication of pre-trained data provided in the embodiment of the present application Figure 4 ;
[0092] Fig. 9 A schematic diagram of third data provided in an embodiment of the present application;
[0093] Fig.10 A schematic diagram of target data provided by an embodiment of the present application;
[0094] Fig.11 A schematic diagram of the structure of a device for deduplicating pre-trained data provided in an embodiment of the present application;
[0095] Fig.12 A hardware structure diagram of a device for deduplicating pre-trained data provided in an embodiment of the present application.
[0096] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0097] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0098] In the prior art, pre-training data is processed through a single process and a single deduplication method. When faced with a huge amount of pre-training data, it takes too much time and resources to perform deduplication processing, and the data deduplication is not thorough and accurate deduplication cannot be achieved. There is a technical problem of low efficiency of pre-training data deduplication.
[0099] The present application provides a method for deduplication of pre-trained data, an electronic device, a storage medium and a program product. By setting a parallel processing strategy, when faced with a large data set, the corresponding number of processes can be determined according to the data volume to complete the deduplication operation in parallel, thereby improving the flexibility, convenience and speed of the data deduplication process, and reducing the waste of time and resources during deduplication processing. At the same time, by setting a double deduplication method to deduplicate the pre-training data, the accuracy and reliability of the target data set after deduplication are improved, the technical problem of low deduplication efficiency of pre-training data is solved, and the technical effect of improving the deduplication efficiency of pre-training data is achieved.
[0100] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0101] Figure 1 The process of the method for deduplication of pre-trained data provided in the embodiment of the present application Figure 1 .like Figure 1 As shown, the method for deduplication of pre-training data provided in the embodiment of the present application includes:
[0102] S101, obtaining a plurality of pre-training data to be processed according to a pre-training data set;
[0103] Figure 2 A schematic diagram of a pre-training data set provided in an embodiment of the present application, such as Figure 2 As shown, based on the pre-training data set, multiple pre-training data that need to be deduplicated are obtained.
[0104] S102, determining the corresponding number of parallel processing processes according to the total amount of pre-training data, and dividing the pre-training data into multiple data batches according to the number of parallel processing processes and the total amount of pre-training data;
[0105] In this embodiment, the number of parallel processing processes is used to indicate the amount of data that can be processed in parallel by the system at the same time, and each data batch contains at least one pre-training data.
[0106] When faced with a pre-training data set with a huge amount of data, the number of parallel processing processes corresponding to the processor is set to n according to the total number of pre-training data, and all pre-training data to be processed in the pre-training set are divided into 4*n data batches according to the set number of parallel processing processes and the total number of pre-training data.
[0107] Specifically, taking a pre-training data set containing 200 pre-training data as an example, if the number of parallel processing processes corresponding to the processor is set to 5, the processor will divide the pre-training data set into 4*5=20 data batches, each data batch contains 10 pre-training data.
[0108] S103, according to the parallel processing strategy, assigning a corresponding data batch to each processing process, and controlling each processing process, performing double deduplication processing on the pre-training data of each data batch, and obtaining processed target data;
[0109] In this embodiment, the double deduplication processing includes inter-document deduplication and intra-document deduplication. Inter-document deduplication includes global data hash deduplication and minimum hash local sensitive hash deduplication. In-document deduplication includes character-level text deduplication and word-level text deduplication.
[0110] According to the parallel processing strategy, a corresponding data batch is assigned to each processing process, and each processing process is controlled to perform inter-document and intra-document deduplication on the pre-training data of each data batch. After all deduplication operations are completed, the target data corresponding to each processing process is obtained.
[0111] S104: After determining that each processing process has completed the double deduplication processing, the corresponding target data in each data batch are aggregated to obtain a deduplicated target data set.
[0112] After ensuring that all parallel processing processes have completed the deduplication operation, the target data corresponding to each data batch in each parallel processing process is aggregated to obtain a complete target data set after deduplication.
[0113] The present application provides a method for deduplicating pre-training data, which obtains a plurality of pre-training data to be processed according to a pre-training data set; determines the corresponding number of parallel processing processes according to the total amount of pre-training data, and divides the pre-training data into a plurality of data batches according to the number of parallel processing processes and the total amount of pre-training data; allocates a corresponding data batch to each processing process according to a parallel processing strategy, and controls each processing process to perform double deduplication processing on the pre-training data of each data batch to obtain processed target data; after determining that each processing process has completed the double deduplication processing, the corresponding target data in each data batch is summarized to obtain a deduplicated target data set, so that by setting a parallel processing strategy, when facing a data set with a large amount of data, the corresponding number of processes can be determined according to the amount of data to complete the deduplication operation in parallel, thereby improving the flexibility, convenience and speed of the data deduplication process, reducing the waste of time and resources during the deduplication processing, and at the same time, by setting a double deduplication method to perform deduplication processing on the pre-training data, the accuracy and reliability of the deduplication target data set are improved, the technical problem of low deduplication efficiency of pre-training data is solved, and the technical effect of improving the deduplication efficiency of pre-training data is achieved.
[0114] Figure 3 The deduplication method process of the pre-training data provided in the embodiment of the present application Figure 2 .like Figure 3 As shown, this embodiment provides a supplementary description of the target data acquisition process based on the above embodiment, including:
[0115] S301, for each processing process, obtaining a hash value corresponding to each pre-training data, and performing global data hash deduplication on multiple pre-training data according to the hash value to remove duplicate pre-training data, thereby obtaining multiple first data after deduplication processing;
[0116] In this embodiment, each pre-training data corresponds to a hash value, and global data hash deduplication is used to perform deduplication processing on multiple pre-training data in the pre-training data set.
[0117] For each parallel processing process, first obtain the text hash value (MD5 value) of the text field of each pre-trained data, and perform global data hash deduplication on all pre-trained data in the current data batch based on the MD5 value of each pre-trained data, so as to deduplicate the duplicate pre-trained data in the current data batch and obtain multiple first data after deduplication processing.
[0118] S302, decomposing a text field corresponding to each first data into a plurality of first segment sequences, and performing character-level text deduplication on the first data according to the first segment sequences, to obtain a plurality of second data after the character-level text deduplication processing;
[0119] In this embodiment, character-level text deduplication refers to character-level N-Gram deduplication.
[0120] The text field corresponding to each first data is decomposed into N continuous first fragment sequences, that is, a sliding window operation of size N is performed on the content in the field according to the characters to form a first fragment sequence of length N, and character-level text deduplication is performed on the first data according to the first fragment sequence to obtain multiple second data after deduplication processing.
[0121] Specifically, if the first data is "This is a record that passed the test", when N of the character-level N-Gram is 2, the data is operated according to a sliding window with a character value of 2, and the content of the first fragment sequence obtained includes: "This is", "Is one", "One", "Pass", "Pass", "Passed", "Test", "Tried", "Record", and "Record". According to the first fragment sequence, character-level text deduplication is performed on the first data to obtain multiple second data after deduplication processing.
[0122] S303, performing data segmentation on each second data, obtaining multiple independent words, and saving each independent word in a text segmentation column;
[0123] In this embodiment, the text tokenizer column refers to text_tokenizer.
[0124] Figure 4 The data diagram after word segmentation provided in the embodiment of the present application is as follows: Figure 4As shown, in order to further analyze and process the second data, the text field of the second data is segmented, that is, the text data is split into individual words, and the result is saved in a new column text_tokenizer.
[0125] S304, performing minimum hash local sensitive hashing deduplication on multiple independent words according to the text segmentation column, and obtaining multiple third data after the minimum hash local sensitive hashing deduplication processing;
[0126] In this embodiment, the tag information includes title tag information, table tag information and link tag information. The minimum hash locality sensitive hashing deduplication is used to deduplicate the text segmentation columns of multiple second data after segmentation. The minimum hash locality sensitive hashing refers to MinHash LSH (MinHash, Locality Sensitive Hashing).
[0127] According to the text word segmentation column, minimum hash local sensitive hashing is performed on the corresponding multiple independent words in the multiple second data to obtain multiple third data after the minimum hash local sensitive hashing deduplication processing.
[0128] S305: Decompose the text field corresponding to each third data into multiple second segment sequences, and perform word-level text deduplication on the third data according to the second segment sequences to obtain target data after the word-level text deduplication processing.
[0129] In this embodiment, word-level text deduplication refers to sub-estimated N-Gram deduplication.
[0130] Convert the content of the text_tokenizer field in the third data into lowercase format, perform a sliding window operation of size N according to the words to form a second fragment sequence of length N, and perform word-level text deduplication on the third data based on the second fragment sequence to obtain the target data after deduplication processing.
[0131] Specifically, if a third data after word segmentation is "This is a record that passed the test", when N of the word-level N-Gram is 2, the data is operated according to a sliding window with a word process size of 2, and the second fragment sequence obtained includes: "This is a", "A passed", "Passed the test", "Tested record", and based on the second fragment sequence, the third data is deduplicated at the word level to obtain the target data after deduplication processing.
[0132] The present application provides a method for deduplicating pre-training data, which deduplicates the pre-training data by simultaneously applying inter-document deduplication and intra-document deduplication methods, performs comprehensive and detailed deduplication processing on the pre-training data set, and obtains deduplicated target data, thereby realizing fuzzy deduplication and precise deduplication of the pre-training data, improving the quality of the target data after deduplication, ensuring the accuracy and reliability of the target data, solving the technical problem of low deduplication efficiency of the pre-training data, and achieving the technical effect of improving the deduplication efficiency of the pre-training data.
[0133] Figure 5 The deduplication method process of the pre-training data provided in the embodiment of the present application Figure 3 .like Figure 5 As shown, based on the above embodiment, this embodiment provides a supplementary description of the process of obtaining the first data and the second data, including:
[0134] S501, according to each processing process, adding a corresponding first marking character to the pre-training data in each data batch;
[0135] In this embodiment, the first marking character is used to indicate that the pre-training data is retained.
[0136] Before performing a deduplication operation on each processing process, a first marker character for retaining the data is added to the keep field of the pre-trained data in each data batch.
[0137] S502: For each pre-training data, obtain a corresponding index number;
[0138] For the multiple pre-training data corresponding to each data batch, obtain the index number of each pre-training data in the data batch.
[0139] S503, treating the pre-trained data with the same hash value as duplicate data, keeping the first marking character corresponding to the pre-trained data with the smallest index number value in the duplicate data unchanged, changing the first marking character corresponding to the remaining duplicate data to the second marking character to remove the remaining duplicate data, and summarizing the pre-trained data with the first marking character as the first data;
[0140] In this embodiment, the second marking character is used to indicate removal of duplicate data.
[0141] Figure 6 A schematic diagram of the first data provided in the embodiment of the present application, such as Figure 6As shown, the first marker character is set to "true" and the second marker character is set to "false". For pre-trained data with the same hash value, they are determined to be duplicate data. For duplicate data, only the first marker character (true) corresponding to the pre-trained data with the smallest index number value in the data is retained, and the first marker character of the remaining duplicate data is changed to the second marker character (false) used to remove the data. After all changes are completed, the pre-trained data with the first marker character in the current data batch are summarized and determined as the first data.
[0142] S504: for each first data, determine the occurrence frequency of each first segment sequence;
[0143] For each first data, each first segment sequence is called a Gram, and the number of occurrences of all Grams is counted.
[0144] S505, summing up the number of first target segment sequences whose occurrence frequency exceeds the first threshold, and determining a first repetition ratio of each first data according to the ratio of the number of the first target segment sequences to the first segment sequences;
[0145] Grams with a number of occurrences greater than a first threshold are taken as first target segment sequences, the corresponding sum of numbers is counted, and the ratio of the sum of the number of the first target segment sequences to the number of all Grams is taken as a first repetition ratio.
[0146] S506, comparing the first repetition ratio with the second threshold value, and if the first repetition ratio is between the first maximum threshold value and the first minimum threshold value, determining that the first target segment sequence is a first repetitive sequence, and changing the first marking character of the first data corresponding to the first repetitive sequence to the second marking character, so as to remove the first data corresponding to the first repetitive sequence;
[0147] In this embodiment, the second threshold includes a first maximum threshold and a first minimum threshold.
[0148] Figure 7 A schematic diagram of the second data provided in the embodiment of the present application, such as Figure 7 As shown, the first mark character is set to "true", the second mark character is set to "false", the first repetition ratio is compared with the second threshold, if the first repetition ratio is within the interval between the set first maximum threshold and the first minimum threshold, the first target fragment sequence is determined to be the first repetitive sequence, and the first mark character (true) of the first data corresponding to the first repetitive sequence is changed to the second mark character (false) to remove the first data corresponding to the first repetitive sequence.
[0149] S507: Summarize the first data with the first marking character and use it as the second data.
[0150] After the mark character of the first repeating sequence is changed, the first data with the first mark character in the current data batch are aggregated and determined as the second data.
[0151] The present application provides a method for deduplicating pre-trained data. By applying global data hash deduplication, duplicate data with the same hash value in the pre-trained data are deduplicated to obtain corresponding first data, and character-level text deduplication is applied to the first data to determine the first repetition ratio of each first data, and according to the relationship between the first repetition ratio and the second threshold, the first data corresponding to the first repetition sequence whose first repetition ratio is between a first maximum threshold and a first minimum threshold are removed, and the remaining first data are determined as the second data after character-level text deduplication. This not only realizes fuzzy deduplication of data and effectively eliminates documents with similar or highly similar semantics, but also removes duplicate fragments or redundant information within the document through precise deduplication means, thereby improving the quality of the target data after deduplication and ensuring the accuracy and reliability of the target data, solving the technical problem of low deduplication efficiency of pre-training data and achieving the technical effect of improving the deduplication efficiency of pre-training data.
[0152] Figure 8 The deduplication method process of the pre-training data provided in the embodiment of the present application Figure 4 .like Figure 8 As shown, based on the above embodiment, after each second data is segmented, the present embodiment provides a supplementary description of the process of obtaining the third data and the target data, including:
[0153] S801, confirming the text similarity corresponding to the text word segmentation column between each second data and each remaining second data according to the minimum hash local sensitive hash algorithm;
[0154] In this embodiment, the text similarity is used to indicate the Jaccard similarity between texts in the text_tokenizer field.
[0155] For the second data in the current data batch, the minimum hash local sensitive hashing algorithm is used to calculate the text similarity corresponding to the text segmentation column between each second data after segmentation and each remaining second data.
[0156] S802, comparing the text similarity with a third threshold, and if the text similarity exceeds the third threshold, determining that the current second data and the remaining second data are similar data;
[0157] The text similarity is compared with a third threshold value. If the text similarity exceeds the third threshold value, it is determined that the current second data and the remaining second data in the current data batch are similar data to each other.
[0158] S803, obtaining index numbers corresponding to the current second data and the remaining second data, and keeping the first marking character corresponding to the second data with the smallest index number value in the similar data unchanged, changing the first marking character of the remaining similar data to the second marking character to remove the remaining similar data, and summarizing the second data with the first marking character as the third data;
[0159] Fig. 9 A schematic diagram of the third data provided in the embodiment of the present application, such as Fig. 9 As shown, the first mark character is set to "true" and the second mark character is set to "false". After determining that the current second data and the remaining second data in the current data batch are similar to each other, the first mark character (true) corresponding to the second data with the smallest index number value in the similar data is kept unchanged, the first mark character of the remaining similar data is changed to the second mark character (false), and the second data with the first mark character are summarized and used as the third data.
[0160] S804: for each text word segmentation column corresponding to the third data, determine the occurrence frequency of each second segment sequence;
[0161] For each text segmentation column corresponding to the third data, the content of each text segmentation column is converted into lowercase format, and a sliding window operation of size N is performed according to the words to form a second segment sequence of length N. Each second segment sequence is called a Gram, and the occurrence frequency of all Grams is counted.
[0162] S805, summing up the number of second target segment sequences whose occurrence frequency exceeds the fourth threshold, and determining a second repetition ratio of each third data according to the ratio of the number of the second target segment sequences to the number of the second segment sequences;
[0163] Grams with an occurrence frequency greater than a fourth threshold are used as second target segment sequences, the total number of second target segment sequences is determined, and the ratio of the second target segment sequences to the total number of second segment sequences is used as the second repetition ratio of each third data.
[0164] S806, comparing the second repetition ratio with the fifth threshold, and if the second repetition ratio is between the second maximum threshold and the second minimum threshold, determining that the second target fragment sequence is a second repetitive sequence;
[0165] In this embodiment, the fifth threshold includes a second maximum threshold and a second minimum threshold.
[0166] The second repetition ratio is compared with the fifth threshold, and if the second repetition ratio is between the set second minimum threshold and the second maximum threshold, it is determined to be a second repetition sequence.
[0167] S807: Change the first identification character of the third data corresponding to the second repeating sequence to the second identification character to remove the third data corresponding to the second repeating sequence, and aggregate the third data with the first identification character as target data.
[0168] Fig.10 A schematic diagram of target data provided by an embodiment of the present application, such as Fig.10 As shown, the first marking character is set to "true", the second marking character is set to "false", the first identification character (true) of the third data corresponding to the second repeating sequence is changed to the second identification character (false) used to remove the current third data, and after all the third data corresponding to the second repeating sequences are changed, the third data with the first marking character are summarized and used as the target data.
[0169] The present application provides a method for deduplicating pre-trained data. By applying a minimum hash local sensitive hashing algorithm, similar data in second data whose text similarity exceeds a third threshold is deduplicated to obtain deduplicated third data. For the third data, word-level text deduplication is applied to determine the second repetition ratio of the text word sequence corresponding to each third data. According to the relationship between the second repetition ratio and the fifth threshold, the third data corresponding to the second repetition sequence whose second repetition ratio is between the second maximum threshold and the second minimum threshold is removed. The remaining third data is determined as the target data after word-level text deduplication. This not only realizes fuzzy deduplication of data and effectively eliminates documents with similar or highly similar semantics, but also removes repeated fragments or redundant information within the document through precise deduplication means. While improving the quality of the target data after deduplication, the accuracy and reliability of the target data are ensured, the technical problem of low deduplication efficiency of pre-training data is solved, and the technical effect of improving the deduplication efficiency of pre-training data is achieved.
[0170] Fig.11 Schematic diagram of the structure of the device for deduplication of pre-trained data provided in the embodiment of the present application. The device of this embodiment can be in the form of software and / or hardware. Fig.11 As shown, the pre-training data deduplication device 1100 provided in the embodiment of the present application includes: an acquisition module 1101, a first processing module 1102, a control module 1103 and a second processing module 1104:
[0171] An acquisition module 1101 is used to acquire a plurality of pre-training data to be processed according to a pre-training data set;
[0172] The first processing module 1102 is used to determine the corresponding number of parallel processing processes according to the total amount of pre-training data, and divide the pre-training data into multiple data batches according to the number of parallel processing processes and the total amount of pre-training data, wherein the number of parallel processing processes is used to indicate the amount of data that can be processed in parallel by the system at the same time, and each data batch contains at least one pre-training data;
[0173] The control module 1103 is used to allocate a corresponding data batch to each processing process according to the parallel processing strategy, and control each processing process to perform double deduplication processing on the pre-training data of each data batch to obtain the processed target data, wherein the double deduplication processing includes deduplication between documents and deduplication within documents;
[0174] The second processing module 1104 is used to aggregate the corresponding target data in each data batch after determining that each processing process has completed the double deduplication processing, and obtain a deduplicated target data set.
[0175] In a possible implementation, the control module 1103 is further configured to:
[0176] For each processing process, a hash value corresponding to each pre-training data is obtained, and according to the hash value, global data hash deduplication is performed on the multiple pre-training data to remove duplicate pre-training data, thereby obtaining multiple first data after deduplication processing, wherein each pre-training data corresponds to a hash value;
[0177] Decomposing a text field corresponding to each first data into a plurality of first segment sequences, and performing character-level text deduplication on the first data according to the first segment sequences, to obtain a plurality of second data after the character-level text deduplication processing;
[0178] Perform data segmentation on each second data to obtain multiple independent words, and save each independent word in a text segmentation column;
[0179] According to the text segmentation column, perform minimum hash local sensitive hashing deduplication on multiple independent words to obtain multiple third data after the minimum hash local sensitive hashing deduplication processing;
[0180] The text field corresponding to each third data is decomposed into a plurality of second segment sequences, and word-level text deduplication is performed on the third data according to the second segment sequences to obtain target data after word-level text deduplication processing.
[0181] In a possible implementation, the control module 1103 is further configured to:
[0182] For each pre-training data, obtain the corresponding index number;
[0183] The pre-trained data with the same hash value are regarded as duplicate data, and the first marking character corresponding to the pre-trained data with the smallest index number value in the duplicate data is kept unchanged, and the first marking character corresponding to the remaining duplicate data is changed to the second marking character to remove the remaining duplicate data, wherein the second marking character is used to indicate the removal of duplicate data;
[0184] The pre-training data with the first labeled character are aggregated and used as the first data.
[0185] In a possible implementation, the control module 1103 is further configured to:
[0186] For each first data, determining the occurrence frequency of each first segment sequence;
[0187] Summarizing the number of first target segment sequences whose occurrence frequency exceeds the first threshold, and determining a first repetition ratio of each first data according to a ratio of the number of the first target segment sequences to the first segment sequences;
[0188] comparing the first repetition ratio with a second threshold, wherein the second threshold includes a first maximum threshold and a first minimum threshold;
[0189] If the first repetition ratio is between the first maximum threshold and the first minimum threshold, determining that the first target segment sequence is a first repetitive sequence, and changing the first marking character of the first data corresponding to the first repetitive sequence to the second marking character to remove the first data corresponding to the first repetitive sequence;
[0190] The first data with the first marking character are aggregated and used as the second data.
[0191] In a possible implementation, the control module 1103 is further configured to:
[0192] According to the minimum hash local sensitive hash algorithm, confirm the text similarity corresponding to the text word segmentation column between each second data and each remaining second data;
[0193] comparing the text similarity with a third threshold;
[0194] If the text similarity exceeds a third threshold, determining that the current second data and the remaining second data are similar data;
[0195] Obtaining index numbers corresponding to the current second data and the remaining second data, keeping the first marking character corresponding to the second data with the smallest index number value among the similar data unchanged, and changing the first marking character of the remaining similar data to the second marking character, so as to remove the remaining similar data;
[0196] The second data with the first marking character are aggregated and used as the third data.
[0197] In a possible implementation, the control module 1103 is further configured to:
[0198] For each text word segmentation column corresponding to each third data, determine the occurrence frequency of each second segment sequence;
[0199] Summarizing the number of second target segment sequences whose occurrence frequency exceeds a fourth threshold, and determining a second repetition ratio of each third data according to a ratio of the number of the second target segment sequences to the number of the second segment sequences;
[0200] comparing the second repetition ratio with a fifth threshold, wherein the fifth threshold includes a second maximum threshold and a second minimum threshold;
[0201] If the second repetition ratio is between the second maximum threshold and the second minimum threshold, determining that the second target segment sequence is a second repetitive sequence, and changing the first identification character of the third data corresponding to the second repetitive sequence to the second identification character to remove the third data corresponding to the second repetitive sequence;
[0202] The third data with the first marking character are summarized and used as the target data.
[0203] In a possible implementation, the control module 1103 is further configured to:
[0204] According to each processing process, a corresponding first marking character is added to the pre-training data in each data batch, wherein the first marking character is used to indicate that the pre-training data is retained.
[0205] The present application provides a device for deduplicating pre-trained data, which can implement the above method embodiment. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.
[0206] Fig.12 The hardware structure diagram of the deduplication device for pre-trained data provided in the embodiment of the present application. Fig.12 As shown, the pre-training data deduplication device 1200 includes:
[0207] Processor 1201 and memory 1202;
[0208] Memory stores computer-executable instructions;
[0209] The processor executes the computer-executable instructions stored in the memory 1202, so that the pre-training data deduplication device performs the pre-training data deduplication method as described above.
[0210] It should be understood that the processor 1201 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASIC). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.
[0211] The memory 1202 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.
[0212] An embodiment of the present application correspondingly provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the deduplication method for pre-trained data as described above.
[0213] The embodiment of the present application also provides a computer program product accordingly. When the computer program is executed by a processor, it is used to implement the deduplication method of pre-training data as described above.
[0214] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0215] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0216] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0217] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.
[0218] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.
[0219] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0220] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0221] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0222] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for removing duplicates from pre-training data, characterized in that: include: According to the pre-training data set, a plurality of pre-training data to be processed are obtained; Determine the corresponding number of parallel processing processes according to the total amount of the pre-training data, and divide the pre-training data into a plurality of data batches according to the number of parallel processing processes and the total amount of the pre-training data, wherein the number of parallel processing processes is used to indicate the amount of data that can be processed in parallel by the system at the same time, and each of the data batches contains at least one pre-training data; According to the parallel processing strategy, the corresponding data batch is allocated to each processing process, and each processing process is controlled to perform double deduplication processing on the pre-training data of each data batch to obtain the processed target data, wherein the double deduplication processing includes deduplication between documents and deduplication within documents; After determining that each processing process has completed the double deduplication processing, the target data corresponding to each data batch is aggregated to obtain a deduplicated target data set.
2. The method according to claim 1, characterized in that The inter-document deduplication includes global data hash deduplication and minimum hash local sensitive hash deduplication, the intra-document deduplication includes character-level text deduplication and word-level text deduplication, and for each of the processing processes, each of the batches of pre-trained data is subjected to double deduplication processing to obtain the processed target data, including: For each of the processing processes, a hash value corresponding to each of the pre-training data is obtained, and according to the hash value, the global data hash deduplication is performed on the plurality of the pre-training data to remove duplicate pre-training data, so as to obtain a plurality of first data after deduplication processing, wherein each of the pre-training data corresponds to one of the hash values; Decomposing each text field corresponding to the first data into a plurality of first segment sequences, and performing character-level text deduplication on the first data according to the first segment sequences, to obtain a plurality of second data after the character-level text deduplication processing; Perform data segmentation on each of the second data to obtain multiple independent words, and save each of the independent words into a text segmentation column; According to the text word segmentation column, perform the minimum hash local sensitive hash deduplication on the multiple independent words to obtain multiple third data after the minimum hash local sensitive hash deduplication processing; The text field corresponding to each of the third data is decomposed into a plurality of second segment sequences, and the third data is subjected to the word-level text deduplication processing according to the second segment sequences to obtain the target data after the word-level text deduplication processing.
3. The method according to claim 2, characterized in that The performing global data hash deduplication on the plurality of pre-training data according to the hash value to remove duplicate pre-training data to obtain a plurality of first data after deduplication processing includes: For each of the pre-training data, obtain a corresponding index number; The pre-trained data with the same hash value are treated as duplicate data, and the first marking character corresponding to the pre-trained data with the smallest index number value in the duplicate data is kept unchanged, and the first marking character corresponding to the remaining duplicate data is changed to the second marking character, so as to remove the remaining duplicate data, wherein the second marking character is used to indicate the removal of the duplicate data; The pre-training data with the first marking character are aggregated and used as the first data.
4. The method according to claim 3, characterized in that: The step of performing character-level text deduplication on the first data according to the first segment sequence to obtain a plurality of second data after the character-level text deduplication processing includes: For each of the first data, determining the occurrence frequency of each of the first fragment sequences; Summarizing the number of the first target segment sequences whose occurrence frequency exceeds the first threshold, and determining a first repetition ratio of each of the first data according to a ratio of the number of the first target segment sequences to the first segment sequences; comparing the first repetition ratio with a second threshold, wherein the second threshold includes a first maximum threshold and a first minimum threshold; If the first repetition ratio is between the first maximum threshold and the first minimum threshold, determining that the first target segment sequence is a first repetitive sequence, and changing the first marking character of the first data corresponding to the first repetitive sequence to the second marking character to remove the first data corresponding to the first repetitive sequence; The first data with the first marking character are aggregated and used as the second data.
5. The method according to claim 4, characterized in that The step of performing the minimum hash local sensitive hash deduplication on the multiple independent words according to the text word segmentation column to obtain multiple third data after the minimum hash local sensitive hash deduplication process includes: According to the minimum hash local sensitive hash algorithm, determining the text similarity corresponding to the text word segmentation column between each of the second data and each of the remaining second data; comparing the text similarity with a third threshold; If the text similarity exceeds the third threshold, determining that the current second data and the remaining second data are similar data; Obtain the index numbers corresponding to the current second data and the remaining second data, keep the first marking character corresponding to the second data with the smallest index number value among the similar data unchanged, and change the first marking character of the remaining similar data to the second marking character, so as to remove the remaining similar data; The second data with the first marking character are aggregated and used as the third data.
6. The method according to claim 5, characterized in that The step of performing the word-level text deduplication on the third data according to the second segment sequence to obtain target data after the word-level text deduplication processing includes: For each of the text word segments corresponding to the third data, determine the occurrence frequency of each of the second segment sequences; Summarizing the number of the second target segment sequences whose occurrence frequency exceeds a fourth threshold, and determining a second repetition ratio of each of the third data according to a ratio of the number of the second target segment sequences to the number of the second segment sequences; comparing the second repetition ratio with a fifth threshold, wherein the fifth threshold includes a second maximum threshold and a second minimum threshold; If the second repetition ratio is between the second maximum threshold and the second minimum threshold, determining that the second target segment sequence is a second repetitive sequence, and changing the first identification character of the third data corresponding to the second repetitive sequence to the second identification character, so as to remove the third data corresponding to the second repetitive sequence; The third data with the first marking character are aggregated and used as the target data.
7. The method according to claim 3, characterized in that For each processing process, performing double deduplication processing on each batch of pre-training data, before obtaining the processed target data, further comprising: According to each of the processing processes, the corresponding first marking character is added to the pre-training data in each of the data batches, wherein the first marking character is used to indicate that the pre-training data is retained.
8. A device for deduplication of pre-training data, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.
10. A computer program product, characterized in that The method comprises a computer program, which is used to implement the method according to any one of claims 1 to 7 when the computer program is executed by a processor.