Data deduplication methods, devices, storage media, and terminals

By performing deduplication at different levels on the training data of large language models, the problem of duplicate corpus data was solved, the data quality was improved, and the flexibility and efficiency of deduplication were enhanced, thus avoiding baseline pollution.

CN117609201BActive Publication Date: 2026-11-17GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311485944.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-08
Publication Date
2026-11-17
Estimated Expiration
2043-11-08

AI Technical Summary

Technical Problem

In model training scenarios, the corpus data from web crawlers has the problem of duplication, which affects data quality and may lead to benchmark contamination and data leakage.

Method used

By determining the target deduplication level for the training data and employing deduplication methods at the substring, document, or global levels, the initial corpus training data is subjected to corresponding levels of deduplication processing, including character matching, hash function bucketing, and data grouping similarity calculation.

Benefits of technology

It improves the quality of training data, increases the flexibility and efficiency of deduplication, and reduces the risk of benchmark contamination.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117609201B_ABST
    Figure CN117609201B_ABST
Patent Text Reader

Abstract

This application discloses a data deduplication method, apparatus, storage medium, and terminal. The method includes: acquiring initial corpus training data in a large language model training scenario; determining a target deduplication level corresponding to the initial corpus training data; and performing data deduplication processing on the initial corpus training data based on the target deduplication level to obtain corpus training data. By employing this method, this application embodiment first determines the deduplication level for the training data and then performs corresponding level deduplication processing on the training data, thereby achieving different levels of deduplication processing on the training data. This not only improves the data quality of the training data but also increases the flexibility of training data deduplication.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data deduplication method, apparatus, storage medium, and terminal. Background Technology

[0002] In related technologies, data deduplication is a crucial operation for ensuring data quality. This is especially true in model training scenarios, where the training corpus data comes from web crawlers, and a significant problem arises from data duplication. Related research shows that deduplication of corpus data can make model training more efficient and also avoid problems such as benchmark contamination and data leakage. Summary of the Invention

[0003] This application provides a data deduplication method, apparatus, computer storage medium, and terminal. By first determining the deduplication level for the training data and then performing corresponding level deduplication processing on the training data, different levels of deduplication processing can be achieved for the training data. This not only improves the data quality of the training data but also increases the flexibility of training data deduplication. The technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a data deduplication method, the method comprising:

[0005] In the context of large language model training, initial corpus training data is obtained;

[0006] Determine the target deduplication level corresponding to the initial corpus training data, and perform data deduplication processing on the initial corpus training data based on the target deduplication level to obtain the corpus training data.

[0007] Secondly, embodiments of this application provide a data deduplication device, the device comprising:

[0008] The data acquisition module is used to acquire initial corpus training data in large language model training scenarios;

[0009] The data processing module is used to determine the target deduplication level corresponding to the initial corpus training data, and to perform data deduplication processing on the initial corpus training data based on the target deduplication level to obtain the corpus training data.

[0010] Thirdly, embodiments of this application provide a computer storage medium having multiple instructions adapted for loading by a processor and executing the above-described method steps.

[0011] Fourthly, embodiments of this application provide a terminal that may include: a memory and a processor; wherein the memory stores a computer program adapted to be loaded by the memory and to execute the above-described method steps.

[0012] The beneficial effects of the technical solutions provided in this application include at least the following:

[0013] In this embodiment of the application, under the large language model training scenario, initial corpus training data is acquired, the target deduplication level corresponding to the initial corpus training data is determined, and then data deduplication processing is performed on the initial corpus training data according to the target deduplication level to obtain the corpus training data. By adopting this method, this embodiment of the application first determines the deduplication level for the training data, and then performs corresponding level deduplication processing on the training data, thereby achieving different levels of deduplication processing on the training data. This not only improves the data quality of the training data but also increases the flexibility of training data deduplication. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a data deduplication method provided in an embodiment of this application;

[0016] Figure 2 This is a flowchart illustrating another data deduplication method provided in an embodiment of this application;

[0017] Figure 3 This is a schematic diagram of the structure of a directed acyclic graph of characters provided in an embodiment of this application.

[0018] Figure 4 This is a flowchart illustrating another data deduplication method provided in the embodiments of this application;

[0019] Figure 5 This is a flowchart illustrating another data deduplication method provided in the embodiments of this application;

[0020] Figure 6 This is a schematic diagram of the structure of a data deduplication device provided in an embodiment of this application;

[0021] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0022] To make the inventive objectives, features, and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0024] In related technologies, a large amount of corpus data for model training comes from web crawlers. In this case, corpus data obtained from different sources often suffers from quality issues, such as content duplication. Therefore, improving the quality of corpus data in model training is a pressing technical problem that needs to be solved.

[0025] To address the aforementioned technical problems, this application will be described in detail below with reference to specific embodiments.

[0026] In the following method embodiments, for ease of explanation, only the execution subject of each step is used as the terminal for description.

[0027] Please see Figure 1 This is a flowchart illustrating a data deduplication method provided in an embodiment of this application. Figure 1 As shown, the method described in this application embodiment may include the following steps:

[0028] S101, in the context of large language model training, obtains initial corpus training data.

[0029] In simple terms, the training scenario for a large language model refers to the scenario in which a large language model is trained. A large language model is an artificial intelligence model designed to understand and generate natural language text. A large language model can refer to a deep learning model trained using large amounts of text data, such as a Large Language Model (LLM). Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue.

[0030] Initial training corpus data refers to large-scale, undeduplicated text resource data for large language models. Initial training corpus data can contain text resource data from at least one source, including Wikipedia, books, journals, open databases crawled by web crawlers, content aggregation communities, etc. The volume of initial training corpus data can reach tens of millions, hundreds of millions, or even larger.

[0031] In some embodiments, the step of obtaining initial corpus training data may involve obtaining first corpus training data from at least one source for a large language model, and integrating the first corpus training data to obtain initial corpus training data.

[0032] Specifically, training data for the large language model can be obtained from at least one source using web crawling, such as Wikipedia, books, journals, open web crawler databases, and content aggregation communities. The functions implemented by the large language model may differ depending on the application scenario; therefore, the initial training data obtained in this embodiment is training data for the functions achievable by the large language model.

[0033] Specifically, the initial corpus training data is obtained by integrating the first corpus training data. This can be done by integrating the first corpus training data from the same source into the same document; or by integrating the first corpus training data with similar content from different sources into the same document and performing content tagging on the document to distinguish between different documents.

[0034] S102, determine the target deduplication level corresponding to the initial corpus training data, and perform data deduplication processing on the initial corpus training data based on the target deduplication level to obtain the corpus training data.

[0035] In simple terms, the target deduplication level can include at least one of the following: substring level, document level, and global level. Substring-level deduplication targets sentences in the training data, and methods at this level remove duplicates from the sentences within the training data. Document-level deduplication targets documents in the training data, and methods at this level remove duplicates from the documents within the training data. Global-level deduplication targets the entire training dataset, and methods at this level remove duplicates from the entire training dataset.

[0036] In some embodiments, the step of determining the target deduplication level corresponding to the initial corpus training data may specifically involve generating a deduplication level query request for the initial corpus training data, sending the deduplication level query request to an electronic device, and receiving the target deduplication level returned by the electronic device in response to the deduplication level query request. In this way, even when the terminal cannot determine the deduplication level for the initial corpus training data, a query request can be generated to accurately determine the deduplication level for the initial corpus training data.

[0037] In some embodiments, the step of determining the target deduplication level corresponding to the initial corpus training data may specifically involve obtaining the model training configuration parameters and training corpus metadata corresponding to the large language model training scenario, and determining the target deduplication level corresponding to the initial corpus training data based on the model training configuration parameters and training corpus metadata corresponding to the large language model. The model training configuration parameters may include, but are not limited to, parameters such as the number of model training epochs, loss function type, and large language model structure. The training corpus metadata may include, but is not limited to, data such as training data size, training data type, and training data source.

[0038] The steps of performing data deduplication on the initial corpus training data based on the target deduplication level to obtain the corpus training data can be as follows: perform data deduplication on the initial corpus training data according to the target deduplication method corresponding to the target deduplication level to obtain the corpus training data.

[0039] When the target deduplication level includes the substring level, the target deduplication method is the substring deduplication method. The implementation process of the substring deduplication method can be as follows: determine at least one target repeated character contained in the initial corpus training data, and remove the repeated characters in the sentence according to the target repeated character to achieve deduplication processing of the initial corpus training data.

[0040] When the target deduplication level includes the document level, the target deduplication method is the document deduplication method. The implementation process of the document deduplication method can be as follows: use a preset hash function to calculate at least one target hash value corresponding to each document in the initial corpus training data; use the target hash value corresponding to each document to allocate the documents into different storage buckets; perform similarity calculation on the documents in each storage bucket to determine the similar documents in each storage bucket; and remove the similar documents in each storage bucket to achieve the deduplication processing of the initial corpus training data.

[0041] When the target deduplication level includes the global level, the target deduplication method is the global deduplication method. The implementation process of the global deduplication method can be as follows: the initial corpus training data is grouped to obtain multiple target data blocks, each containing multiple documents; document similarity is calculated for the current target data block and other target data blocks respectively to remove duplicate documents in the current target data block. The current target data block is any one of the target data blocks, and the other target data blocks are the target data blocks other than the current target data block among the multiple target data blocks. For example, let multiple target data blocks be word1, word2, ..., word(N-1), wordN, where N is an integer greater than 2. When the current target data block is word1, the similarity between documents in word1 and documents in word2 is calculated to remove duplicate documents in word1; the similarity between documents in word1 and documents in word3 is calculated to remove duplicate documents in word1; and so on, the similarity between documents in word1 and documents in word(N-1) is calculated to remove duplicate documents in word1; the similarity between documents in word1 and documents in wordN is calculated to remove duplicate documents in word1. In the current target data block, word2, we can skip the similarity calculation between documents in word2 and word1, as word1 no longer contains duplicate documents. Instead, we can calculate the similarity between documents in word2 and word3 to remove duplicates from word2; similarly, we can calculate the similarity between documents in word2 and word4 to remove duplicates from word2; and so on, calculating the similarity between documents in word2 and wordN to remove duplicates from word2. By performing these operations on word1 and word2, we can remove duplicate documents from all target data blocks, achieving global deduplication of the initial training data.

[0042] In this embodiment of the application, under the large language model training scenario, initial corpus training data is acquired, the target deduplication level corresponding to the initial corpus training data is determined, and then data deduplication processing is performed on the initial corpus training data according to the target deduplication level to obtain the corpus training data. By adopting this method, this embodiment of the application first determines the deduplication level for the training data, and then performs corresponding level deduplication processing on the training data, thereby achieving different levels of deduplication processing on the training data. This not only improves the data quality of the training data but also increases the flexibility of training data deduplication.

[0043] Please see Figure 2 This is a flowchart illustrating a data deduplication method provided in an embodiment of this application. Figure 2 As shown, the method described in this application embodiment may include the following steps:

[0044] S201, in the context of large language model training, obtain initial corpus training data.

[0045] Specifically, see Figure 1 The description of S101 in the illustrated embodiment will not be repeated here.

[0046] S202, determine the target deduplication level corresponding to the initial corpus training data.

[0047] In some embodiments, the step of determining the target deduplication level corresponding to the initial corpus training data may specifically include: obtaining model training configuration parameters corresponding to the large language model training scenario; obtaining training corpus metadata of the initial corpus training data; and determining the target deduplication level corresponding to the initial corpus training data based on the model training configuration parameters and the training corpus metadata.

[0048] The model training configuration parameters may include, but are not limited to, the number of training epochs, the type of loss function, and the structure of the large language model. The training corpus metadata may include, but is not limited to, the size of the training data, the type of training data, and the source of the training data.

[0049] Specifically, obtaining the model training configuration parameters corresponding to the large language model training scenario can be understood as obtaining the model training configuration parameters from the configuration file corresponding to the large language model training scenario. The model training configuration parameters stored in the configuration file can be parameters manually configured based on the large language model training scenario.

[0050] Specifically, obtaining the training corpus metadata from the initial training data can be understood as identifying the training corpus metadata based on the attribute information of the initial training data.

[0051] The target deduplication level corresponding to the initial corpus training data is determined based on the model training configuration parameters and training corpus metadata. Specifically, the model training configuration parameters and training corpus metadata are input into the deduplication level recognition model to obtain the target deduplication level corresponding to the initial corpus training data. The deduplication level recognition model is obtained after being trained with the sample model training configuration parameters and sample training corpus metadata that have been labeled with deduplication level tags.

[0052] The training process of the deduplication level recognition model is explained below:

[0053] Model creation: Create an initial deduplication level recognition model based on a machine learning model for the deduplication level recognition scenario;

[0054] Sample data acquisition: Acquire a large amount of sample data. The sample data is based on the historical corpus training data and historical model configuration parameters of the historical large language model in the training scenario. The sample model training configuration parameters and sample training corpus metadata are processed and extracted to generate sample data containing sample model training configuration parameters and sample training corpus metadata.

[0055] Sample data labeling: Based on the needs of deduplication level recognition scenarios, an expert service is introduced to manually label the sample data with corresponding sample labels, including deduplication level labels for each sample data.

[0056] Model training process: Input sample data into the initial deduplication level recognition model for at least one round of model training to obtain the predicted deduplication level. Based on the predicted deduplication level and sample data labels (deduplication level labels), the model loss function is used to determine the model loss value. Based on the model loss value, the model parameters of the initial deduplication level recognition model are adjusted until the model training termination condition is met to obtain the deduplication level recognition model.

[0057] Optionally, the model's training termination conditions may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific training termination conditions can be determined based on actual circumstances and are not specifically limited here.

[0058] It should be noted that the machine learning models involved in one or more embodiments of this application include, but are not limited to, fitting of one or more of the following machine learning models: Convolutional Neural Network (CNN) model, Deep Neural Network (DNN) model, Recurrent Neural Networks (RNN) model, embedding model, Gradient Boosting Decision Tree (GBDT) model, Logistic Regression (LR) model, etc.

[0059] S203, obtain the target deduplication method corresponding to the target deduplication level, where the target deduplication method is the substring deduplication method.

[0060] In this embodiment, the target deduplication level is at the substring level, and the target deduplication method is a substring deduplication method. The deduplication object at the substring level is the sentences in the initial corpus training data, and the substring deduplication method performs deduplication processing on the sentences.

[0061] S204, determine at least one target repeating character corresponding to the initial corpus training data.

[0062] In simple terms, target repeating characters refer to repeated characters in a sentence. Specifically, target repeating characters can be keywords.

[0063] In some embodiments, the step S204 may specifically be: determining multiple reference repeating characters corresponding to the initial corpus training data based on the frequency of occurrence of each character in the initial corpus training data; performing character filtering processing on the reference repeating characters to obtain at least one target repeating character corresponding to the initial corpus training data.

[0064] Determining multiple reference repeating characters based on the frequency of occurrence of each character in the initial corpus training data can be understood as using characters whose frequency of occurrence in sentences of the initial corpus training data is greater than a preset frequency as reference repeating characters.

[0065] Character filtering of reference duplicate characters to obtain at least one target duplicate character corresponding to the initial corpus training data can be understood as obtaining a preset character filtering template containing multiple filtering characters, and filtering the reference duplicate characters according to the preset character filtering template to obtain at least one target duplicate character. Since the reference duplicate characters are determined based on the frequency of character occurrence, these reference duplicate characters include multiple entity names, such as person names, place names, and organization names. If these duplicate characters are not filtered out, subsequent deduplication based on these duplicate characters will cause false positives, leading to a decrease in the quality of the training data. Therefore, the preset character filtering template can be used to filter the reference duplicate characters to obtain at least one target duplicate character.

[0066] As you can easily understand, the filter characters in the preset character filter template are not fixed. They can be updated according to the actual application scenario to achieve a better deduplication effect.

[0067] S205, based on the repeated characters of each target, the initial corpus training data is deduplicated to obtain the corpus training data.

[0068] In some embodiments, the step S205 may specifically include: generating a directed acyclic graph of characters based on each target repeating character; and performing data deduplication processing on the initial corpus training data based on the directed acyclic graph of characters to obtain corpus training data.

[0069] A character-directed acyclic graph (DAG) is a directed acyclic graph whose nodes are composed of the constituent characters of a target repeating character. A character-directed acyclic graph is a loop-free directed graph; starting from a vertex in a character-directed acyclic graph, it is impossible to return to that vertex after traversing several edges. A node in a path in a character-directed acyclic graph can be a constituent character of a single target repeating character, or a node in a path can be a constituent character of multiple target repeating characters. For example, see [example provided]. Figure 3 The diagram shows a structure of a directed acyclic graph of characters. Figure 3 As shown, the directed acyclic graph of characters includes 6 target repeating characters, namely "apple", "apeal", "apear", "bee", "beef" and "cat". Figure 3 Each gray box represents the ending character of a target repeating character. On the same path, there can be two types of target repeating characters, such as "bee" and "beef" having the same partial path node.

[0070] Specifically, generating a directed acyclic graph (DAG) based on each target repeating character can be understood as selecting multiple first repeating characters with the same characteristics from the target repeating characters, and using the constituent characters of the first repeating characters as nodes to generate a DAG. The first repeating characters can be repeating characters with several identical constituent characters, repeating characters with the same number of characters, or repeating characters with the same character structure.

[0071] Specifically, the initial corpus training data is deduplicated based on a directed acyclic graph (DAG). This can be understood as matching each sentence in the initial corpus training data against multiple target repeating characters contained in the DAG, identifying the target repeating characters in each sentence, and then removing them. This process effectively deduplicates the initial corpus training data. In this way, by simultaneously performing matching on multiple target repeating characters within the same sentence using a DAG, the matching efficiency for repeating characters is improved.

[0072] In this embodiment, when performing substring deduplication on the initial corpus training data, the target repeating characters for the initial corpus training data are first determined, and then the initial corpus training data is deduplicated based on the target repeating characters. This method achieves deduplication by simultaneously performing multiple character matching processes on the same sentence using multiple target repeating characters. This improves both the quality of the training data and the matching efficiency of repeating characters, thereby enhancing the deduplication efficiency of the training data.

[0073] Please see Figure 4 This is a flowchart illustrating a data deduplication method provided in an embodiment of this application. Figure 4 As shown, the method described in this application embodiment may include the following steps:

[0074] S401, in the context of large language model training, acquires initial corpus training data.

[0075] S402, determine the target deduplication level corresponding to the initial corpus training data.

[0076] Specifically, see S401 and S402. Figure 2 The descriptions of S201 and S202 in the illustrated embodiment will not be repeated here.

[0077] S403, retrieves the target deduplication method corresponding to the target deduplication level, where the target deduplication method is the document deduplication method.

[0078] In this embodiment, the target deduplication level is document-level, and the target deduplication method is document-level deduplication. The substring-level deduplication object is the documents in the initial corpus training data, and the substring deduplication method involves deduplicating the documents.

[0079] S404, calculate at least one target hash value corresponding to each training document in the initial corpus training data based on a preset hash function.

[0080] In some embodiments, the execution of step S404 may specifically include: A1, calculating at least one reference hash value corresponding to each training document in the initial corpus training data based on a preset hash function; A2, arranging the at least one reference hash value corresponding to each training document respectively, and determining at least one target hash value corresponding to the training document based on the arrangement order of the reference hash values.

[0081] The training document refers to the multiple documents contained in the initial corpus training data.

[0082] Optionally, the default hash function can be the minimum hash function, MinHash.

[0083] Specifically, step A1 can be understood as follows: for each training document in the initial corpus training data, a preset hash function is used to perform hash calculations on a preset number of characters in the training document to obtain at least one reference hash value corresponding to the training document. Before performing hash calculations on the preset number of characters in the training document, a word segmentation algorithm can be used to segment each training document, and the preset number of characters obtained from the word segmentation can be stored in a character set. In other words, each training document can be converted into a character set through word segmentation. Then, performing hash calculations on the preset number of characters in the training document can be done by using the preset hash function to perform hash calculations on each character in the character set corresponding to the training document.

[0084] Specifically, step A2 can be understood as follows: for each training document, the reference hash values ​​of each character in the character set obtained in step A1 are arranged, and then, according to the arrangement order of each reference hash value, at least one target hash value is selected from the reference hash values. The target hash value selected from the reference hash values ​​can be used as at least one target hash value corresponding to that training document. When arranging the reference hash values, a preset arrangement algorithm can be used. Optionally, the preset arrangement algorithm can be a random arrangement, a selection arrangement, or other similar algorithms.

[0085] S405, allocate training documents to buckets based on at least one target hash value.

[0086] In some embodiments, the similarity between the current training document and other training documents is calculated based on the target hash value, and training documents with a similarity greater than a similarity threshold are allocated to the same storage bucket. Here, the current training document refers to any one of all training documents, and other training documents refer to all training documents excluding the current training document. Each training document corresponds to the same number of target hash values.

[0087] For example, the training documents in the initial corpus training data can include word1, word2, word3, ..., word(N-1), wordN, where N is an integer greater than 3. When the current training document is word1, the similarity between word1 and word2 is calculated based on the first target hash value corresponding to word1 and the second target hash value corresponding to word2. That is, the similarity between the first target hash value and the second target hash value is calculated. If there are multiple target hash values, the similarity between the set containing the first target hash value and the set containing the second target hash value is used as the similarity between word1 and word2. Similarly, the similarity between word1 and word3 is calculated based on the first target hash value corresponding to word1 and the third target hash value corresponding to word3. That is, the similarity between the first target hash value and the third target hash value is calculated. If there are multiple target hash values, the similarity between the set containing the first target hash value and the set containing the third target hash value is used as the similarity between word1 and word3. Similarly, the similarity between word1 and word(N-1) is calculated based on the first target hash value corresponding to word1 and the reference target hash value corresponding to word(N-1). That is, the similarity between the first target hash value and the reference target hash value is calculated. If there are multiple target hash values, the similarity between the set containing the first target hash value and the set containing the reference target hash value is calculated as the similarity between word1 and word(N-1). The similarity between word1 and wordN is calculated based on the first target hash value corresponding to word1 and the specified target hash value corresponding to wordN. That is, the similarity between the first target hash value and the specified target hash value is calculated. If there are multiple target hash values, the similarity between the set containing the first target hash value and the set containing the specified target hash value is calculated as the similarity between word1 and wordN.

[0088] When the current training document is word2, the operation of calculating the similarity between word2 and word1 can be omitted, because it has already been calculated when the current training document is word1, and there is no need to calculate the similarity between word2 and word1 again. Next, the similarity between word2 and word3 is calculated based on the second target hash value corresponding to word2 and the third target hash value corresponding to word3. That is, the similarity between the second target hash value and the third target hash value is calculated. If there are multiple target hash values, the similarity between the set containing the second target hash value and the set containing the third target hash value is used as the similarity between word2 and word3. Similarly, the similarity between word2 and word(N-1) is calculated based on the second target hash value corresponding to word2 and the reference target hash value corresponding to word(N-1). That is, the similarity between the second target hash value and the reference target hash value is calculated. If there are multiple target hash values, the similarity between the set containing the second target hash value and the set containing the reference target hash value is used as the similarity between word2 and word(N-1). Finally, the similarity between word2 and wordN is calculated based on the second target hash value corresponding to word2 and the specified target hash value corresponding to wordN. That is, the similarity between the second target hash value and the specified target hash value is calculated. If there are multiple target hash values, the similarity between the set containing the second target hash value and the set containing the specified target hash value is used as the similarity between word2 and wordN.

[0089] Following the same steps as described above for calculating the similarity between training documents, the similarity between any two training documents can be calculated, and training documents with a similarity greater than the similarity threshold can be assigned to the same storage bucket.

[0090] S406, determine the similarity between target training documents based on the reference hash value corresponding to each target training document in the storage bucket.

[0091] In some embodiments, as shown in S404, each training document corresponds to at least one reference hash value; similarly, each target training document within each bucket also corresponds to at least one reference hash value. Within each bucket, the similarity between the current target training document and other target training documents can be calculated based on the reference hash value. Specifically, the calculation method for the similarity between the current target training document and other target training documents based on the reference hash value can be referred to the calculation method listed in S405, and will not be repeated here.

[0092] S407, Based on similarity, the target training documents in the storage bucket are deduplicated to obtain the corpus training data.

[0093] In some embodiments, for each bucket, two target training documents with a similarity greater than a similarity threshold can be considered as similar documents. Similar documents within the bucket can be removed to obtain buckets without duplicate documents. Further, the remaining target training documents after removing duplicate documents from all buckets constitute the corpus training data.

[0094] In this embodiment, when performing document deduplication on the initial corpus training data, at least one target hash value corresponding to each training document in the initial corpus training data is first calculated according to a preset hash function. Then, the training documents are allocated into storage buckets based on the at least one target hash value. The similarity between each target training document is determined based on the reference hash value corresponding to each target training document in the storage bucket. Finally, document deduplication is performed on the target training documents in the storage buckets based on the similarity to obtain the corpus training data. This method, by allocating documents to storage buckets, initially filters out similar documents. Then, similarity filtering is performed again only on the similar documents within the storage buckets. This not only achieves the effect of removing duplicate documents to improve the quality of training data but also improves the deduplication efficiency of training data through two similarity calculations.

[0095] Please see Figure 5 This is a flowchart illustrating a data deduplication method provided in an embodiment of this application. Figure 5 As shown, the method described in this application embodiment may include the following steps:

[0096] S501, in the context of large language model training, acquires initial corpus training data.

[0097] S502, determine the target deduplication level corresponding to the initial corpus training data.

[0098] Specifically, see S501 and S502. Figure 2 The descriptions of S201 and S202 in the illustrated embodiment will not be repeated here.

[0099] S503, obtain the target deduplication method corresponding to the target deduplication level, the target deduplication method is the global deduplication method.

[0100] In this embodiment, the target deduplication level is global, and the target deduplication method is a global deduplication method. The deduplication object at the substring level is the entire dataset in the initial corpus training data, while the global deduplication method refers to performing deduplication processing on the entire training data dataset.

[0101] S504 performs data grouping processing based on the initial corpus training data to obtain multiple target data blocks.

[0102] In some embodiments, the initial corpus training data can be grouped according to its size to obtain multiple target data blocks of the same size. Alternatively, the initial corpus training data can be grouped according to its source to obtain multiple target data blocks from different sources. Furthermore, the initial corpus training data can be grouped according to its acquisition time to obtain multiple target data blocks acquired during different time periods.

[0103] S505, determine the first target data block and the second target data block in the target data block.

[0104] The second target data block is a data block other than the first target data block among multiple target data blocks.

[0105] In some embodiments, a first target data block can be determined from multiple target data blocks. Specifically, a target data block can be randomly selected as the first target data block, or the first target data block can be selected from multiple target data blocks according to a preset selection rule. Further, all target data blocks other than the first target data block from the multiple target data blocks can be used as second target data blocks.

[0106] S506, calculate the document similarity between the second document in each second target data block and the first document in the first target data block, and perform document deduplication processing on the first target data block based on the document similarity.

[0107] Each second target data block may include multiple second documents, and the first target data block may include multiple first documents.

[0108] In some embodiments, for each second target data block, the document similarity between the second document in the second target data block and the first document in the first target data block is calculated. Specifically, this can be done as follows: B1, calculating at least one target hash value corresponding to each second document based on a preset hash function; B2, calculating at least one target hash value corresponding to each first document based on a preset hash function; B3, allocating the first document and the second document in a storage bucket based on at least one target hash value; B4, determining the similarity between each target document based on the reference hash value corresponding to each target document in the storage bucket.

[0109] For details on performing steps B1 and B2, please refer to [link / reference]. Figure 5 The steps in S504 of the implementation shown will not be repeated here. For details on executing step B3, please refer to [link to relevant documentation]. Figure 5 The steps of S505 in the illustrated embodiment will not be repeated here. For details on executing step B4, please refer to [link to relevant documentation]. Figure 5 The steps of S506 in the illustrated embodiment will not be repeated here.

[0110] By going through steps B1 to B5 above, the similarity between the first document and the second document can be obtained. This similarity is the document similarity. The first document and the second document with a document similarity greater than the similarity threshold are identified as similar documents. Then, the first document identified as a similar document in the first target data block can be removed, thereby realizing the document deduplication process of the first target data block.

[0111] S507, determine whether the number of the second target data blocks is the preset number.

[0112] S508, if the number of the second target data blocks is not the preset number, then update the second target data blocks to target data blocks.

[0113] S509, if the number of the second target data blocks is the preset number, then obtain the corpus training data.

[0114] The following is an explanation of S507 to S509.

[0115] The preset quantity can be set to 1.

[0116] In this embodiment, when the number of second target data blocks is not 1, the second target data blocks can be updated to target data blocks, that is, each second target data block is used as a target data block, and the steps S505 to S507 described above are executed. In this way, by recursively searching for similar documents in different data blocks, deduplication of data blocks is achieved, thereby realizing global deduplication.

[0117] When the number of the second target data block is 1, the above loop execution steps are stopped, and the documents remaining after deduplication in each target data block are used as corpus training data. In this way, the documents obtained after global deduplication are the corpus training data. The data repetition rate in the corpus training data is low, which greatly improves the quality of the corpus training data. Therefore, when using high-quality corpus training data to train or test large language models, the model capabilities of large language models can be evaluated more accurately.

[0118] In the embodiments of this application, when performing document deduplication on the initial corpus training data, the initial corpus training data is first processed by data grouping to obtain multiple target data blocks. The multiple target data blocks obtained by grouping are deduplicated by recursively searching for similar documents. This achieves global deduplication on limited computer resources, that is, deduplication of large-scale training data can be performed in batches, reducing the dependence of global deduplication on huge memory overhead, and also avoiding the problem of benchmark pollution.

[0119] The following will combine Figure 6 This application provides a detailed description of the data deduplication device provided in its embodiments. It should be noted that... Figure 6The data deduplication device shown is used to perform the functions described in this application. Figures 1-5 The methods shown in the embodiments are for illustrative purposes only, illustrating the parts relevant to the embodiments of this application. For specific technical details not disclosed, please refer to this application. Figures 1-5 The example shown.

[0120] Please see Figure 6 This diagram illustrates the structure of a data deduplication device according to an embodiment of this application. The data deduplication device 1 can be implemented as all or part of a device through software, hardware, or a combination of both. According to some embodiments, the data deduplication device 1 includes a data acquisition module 11 and a data processing module 12, specifically used for:

[0121] Data acquisition module 11 is used to acquire initial corpus training data in large language model training scenarios;

[0122] Data processing module 12 is used to determine the target deduplication level corresponding to the initial corpus training data, and perform data deduplication processing on the initial corpus training data based on the target deduplication level to obtain corpus training data.

[0123] Optionally, the data processing module includes:

[0124] The first processing unit is used to obtain the target deduplication method corresponding to the target deduplication level;

[0125] The second processing unit is used to perform data deduplication processing on the initial corpus training data based on the target deduplication method to obtain corpus training data.

[0126] Optionally, the second processing unit includes:

[0127] The first subunit is used to determine at least one target repeating character corresponding to the initial corpus training data;

[0128] The second subunit is used to perform data deduplication processing on the initial corpus training data based on each of the target repeating characters to obtain corpus training data.

[0129] Optionally, the second subunit is specifically used for:

[0130] Generate a directed acyclic graph of characters based on the target repeating characters;

[0131] Based on the character directed acyclic graph, the initial corpus training data is deduplicated to obtain the corpus training data.

[0132] Optionally, the second processing unit includes:

[0133] The first calculation unit is used to calculate at least one target hash value corresponding to each training document in the initial corpus training data based on a preset hash function.

[0134] The second computing unit is used to allocate the training documents to storage buckets based on the at least one target hash value;

[0135] The third calculation unit is used to determine the similarity between each target training document based on the reference hash value corresponding to each target training document in the storage bucket.

[0136] The fourth calculation unit is used to perform document deduplication processing on the target training documents in the storage bucket based on the similarity to obtain corpus training data.

[0137] Optionally, the first computing unit is specifically used for:

[0138] Calculate at least one reference hash value corresponding to each training document in the initial corpus training data based on a preset hash function;

[0139] Each training document is sorted by at least one reference hash value, and at least one target hash value is determined based on the sorting order of the reference hash values.

[0140] Optionally, the second processing unit is specifically used for:

[0141] Based on the initial corpus training data, data grouping processing is performed to obtain multiple target data blocks;

[0142] A first target data block and a second target data block are determined in the target data block, wherein the second target data block is a data block in the target data block other than the first target data block;

[0143] Calculate the document similarity between the second document in each second target data block and the first document in each first target data block, and perform document deduplication processing on the first target data block based on the document similarity.

[0144] Determine whether the number of the second target data blocks is the preset number;

[0145] If not, then update the second target data block to the target data block and execute the steps of determining the first target data block and the second target data block in each of the target data blocks, calculating the document similarity between the second document in each second target data block and the first document in the first target data block, and performing document deduplication processing on the first target data block based on the document similarity, until the number of the second target data blocks is a preset number.

[0146] Optionally, the data processing module includes:

[0147] The first parameter acquisition module is used to acquire the model training configuration parameters corresponding to the large language model training scenario.

[0148] The second parameter acquisition module is used to acquire training corpus metadata of the initial corpus training data;

[0149] The third parameter acquisition module is used to determine the target deduplication level corresponding to the initial corpus training data based on the model training configuration parameters and the training corpus metadata.

[0150] Optionally, the third parameter acquisition module is specifically used for:

[0151] The model training configuration parameters and the training corpus metadata are input into the deduplication level recognition model to obtain the target deduplication level corresponding to the initial corpus training data. The deduplication level recognition model is obtained after training with the sample model training configuration parameters and sample training corpus metadata labeled with deduplication level.

[0152] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Figure 7 As shown, terminal 700 may include: at least one processor 701, at least one network interface 704, user interface 703, memory 705, display assembly 706, and at least one communication bus 702.

[0153] The communication bus 702 is used to enable communication between these components.

[0154] The user interface 703 may include a display screen and a camera. Optionally, the user interface 703 may also include a standard wired interface and a wireless interface.

[0155] The network interface 704 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0156] The processor 701 may include one or more processing cores. The processor 701 connects to various parts within the terminal 700 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 705, and by calling data stored in the memory 705. Optionally, the processor 701 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 701 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 701 and may be implemented as a separate chip.

[0157] The memory 705 may include random access memory (RAM) or read-only memory. Optionally, the memory 705 may include a non-transitory computer-readable storage medium. The memory 705 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 705 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 705 may also be at least one storage device located remotely from the aforementioned processor 701. Figure 7 As shown, the memory 705, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a program for data deduplication.

[0158] In addition, those skilled in the art will understand that the structure of the terminal 700 shown in the above figures does not constitute a limitation on the terminal 700. The user terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the terminal 700 also includes radio frequency circuits, audio circuits, WiFi components, power supplies, Bluetooth components, etc., which will not be described in detail here.

[0159] exist Figure 7 In the terminal shown, the processor 701 can be used to call the program for the data deduplication processing method stored in the memory 705, and specifically perform the following operations:

[0160] In the context of large language model training, initial corpus training data is obtained;

[0161] Determine the target deduplication level corresponding to the initial corpus training data, and perform data deduplication processing on the initial corpus training data based on the target deduplication level to obtain the corpus training data.

[0162] In one embodiment, when the processor 701 performs the step of performing data deduplication processing on the initial corpus training data based on the target deduplication level to obtain corpus training data, it specifically performs the following operations:

[0163] Obtain the target deduplication method corresponding to the target deduplication level;

[0164] The initial corpus training data is obtained by performing data deduplication processing based on the target deduplication method.

[0165] In one embodiment, when the processor 701 executes the step of performing data deduplication on the initial corpus training data based on the target deduplication method (substring deduplication method) to obtain corpus training data, the processor 701 specifically performs the following operations:

[0166] Identify at least one target repeating character corresponding to the initial corpus training data;

[0167] The initial corpus training data is obtained by deduplicating the initial corpus training data based on the target repeated characters.

[0168] In one embodiment, when the processor 701 performs the step of deduplicating the initial corpus training data based on each of the target repeating characters to obtain the corpus training data, it specifically performs the following operations:

[0169] Generate a directed acyclic graph of characters based on the target repeating characters;

[0170] The initial corpus training data is obtained by deduplicating the data based on the character directed acyclic graph.

[0171] In one embodiment, when the processor 701 executes the step of performing data deduplication on the initial corpus training data based on the target deduplication method (document deduplication method) to obtain corpus training data, the processor 701 specifically performs the following operations:

[0172] Calculate at least one target hash value corresponding to each training document in the initial corpus training data based on a preset hash function;

[0173] The training documents are allocated to buckets based on the at least one target hash value;

[0174] The similarity between each target training document is determined based on the reference hash value corresponding to each target training document in the storage bucket.

[0175] Based on the similarity, the target training documents in the storage bucket are deduplicated to obtain the corpus training data.

[0176] In one embodiment, when the processor 701 performs the step of calculating at least one target hash value corresponding to each training document in the initial corpus training data based on a preset hash function, it specifically performs the following operations:

[0177] Calculate at least one reference hash value corresponding to each training document in the initial corpus training data based on a preset hash function;

[0178] Each training document is sorted by at least one reference hash value, and at least one target hash value is determined based on the sorting order of the reference hash values.

[0179] In one embodiment, when the processor 701 executes the step of performing data deduplication on the initial corpus training data based on the target deduplication method (which is a global deduplication method) to obtain the corpus training data, the processor 701 specifically performs the following operations:

[0180] Based on the initial corpus training data, data grouping processing is performed to obtain multiple target data blocks;

[0181] A first target data block and a second target data block are determined in the target data block, wherein the second target data block is a data block in the target data block other than the first target data block;

[0182] Calculate the document similarity between the second document in each second target data block and the first document in each first target data block, and perform document deduplication processing on the first target data block based on the document similarity.

[0183] Determine whether the number of the second target data blocks is the preset number;

[0184] If not, then update the second target data block to the target data block and execute the steps of determining the first target data block and the second target data block in each of the target data blocks, calculating the document similarity between the second document in each second target data block and the first document in the first target data block, and performing document deduplication processing on the first target data block based on the document similarity, until the number of the second target data blocks is a preset number.

[0185] In one embodiment, when the processor 701 performs the step of determining the target deduplication level corresponding to the initial corpus training data, it specifically performs the following operations:

[0186] Obtain the model training configuration parameters corresponding to the large language model training scenario;

[0187] Obtain the training corpus metadata of the initial corpus training data;

[0188] The target deduplication level corresponding to the initial corpus training data is determined based on the model training configuration parameters and the training corpus metadata.

[0189] In one embodiment, when the processor 701 performs the step of determining the target deduplication level corresponding to the initial corpus training data based on the model training configuration parameters and the training corpus metadata, it specifically performs the following operations:

[0190] The model training configuration parameters and the training corpus metadata are input into the deduplication level recognition model to obtain the target deduplication level corresponding to the initial corpus training data. The deduplication level recognition model is obtained after training with the sample model training configuration parameters and sample training corpus metadata labeled with deduplication level.

[0191] This application also provides a computer-readable storage medium storing at least one instruction that is executed by a processor to implement the data deduplication method as described in the above embodiments.

[0192] This application also provides a computer program product that stores at least one instruction, which is loaded and executed by the processor to implement the data deduplication method described in the above embodiments.

[0193] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0194] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data deduplication method, characterized in that, The method includes: In the context of large language model training, initial corpus training data is obtained; Determine the target deduplication level corresponding to the initial corpus training data, obtain the target deduplication method corresponding to the target deduplication level, and perform data deduplication processing on the initial corpus training data based on the target deduplication method to obtain corpus training data. The target deduplication method includes substring deduplication method, document deduplication method and global deduplication method. Wherein, when the target deduplication method is the substring deduplication method, the step of performing data deduplication processing on the initial corpus training data based on the target deduplication method to obtain corpus training data includes: determining multiple reference repeating characters corresponding to the initial corpus training data based on the frequency of occurrence of each character in the initial corpus training data; obtaining a preset character filtering template, the preset character filtering template containing multiple filtering characters, performing character filtering processing on the multiple reference repeating characters according to the preset character filtering template, filtering out entity names present in the reference repeating characters, and obtaining at least one target repeating character corresponding to the initial corpus training data; and removing characters that appear repeatedly in sentences in the initial corpus training data according to each target repeating character, thereby obtaining corpus training data; When the target deduplication method is the global document deduplication method, the step of performing data deduplication processing on the initial corpus training data based on the target deduplication method to obtain corpus training data includes: performing data grouping processing on the initial corpus training data to obtain multiple target data blocks; determining a first target data block and a second target data block in the target data blocks, wherein the second target data block is a data block other than the first target data block in the target data blocks; calculating the document similarity between the second document in each second target data block and the first document in the first target data block, and performing document deduplication processing on the first target data block based on the document similarity.

2. The method according to claim 1, characterized in that, The step of removing recurring characters from sentences in the initial corpus training data according to each of the target recurring characters to obtain the corpus training data includes: Generate a directed acyclic graph of characters based on the target repeating characters; The initial corpus training data is obtained by deduplicating the data based on the character directed acyclic graph.

3. The method according to claim 1, characterized in that, When the target deduplication method is a document deduplication method, the step of performing data deduplication processing on the initial corpus training data based on the target deduplication method to obtain the corpus training data includes: Calculate at least one target hash value corresponding to each training document in the initial corpus training data based on a preset hash function; The training documents are allocated to buckets based on the at least one target hash value; The similarity between each target training document is determined based on the reference hash value corresponding to each target training document in the storage bucket. Based on the similarity, the target training documents in the storage bucket are deduplicated to obtain the corpus training data.

4. The method according to claim 3, characterized in that, The step of calculating at least one target hash value corresponding to each training document in the initial corpus training data based on a preset hash function includes: Calculate at least one reference hash value corresponding to each training document in the initial corpus training data based on a preset hash function; Each training document is sorted by at least one reference hash value, and at least one target hash value is determined based on the sorting order of the reference hash values.

5. The method according to claim 1, characterized in that, After performing document deduplication on the first target data block based on the document similarity, the process further includes: Determine whether the number of the second target data blocks is the preset number; If not, then update the second target data block to the target data block and execute the steps of determining the first target data block and the second target data block in each of the target data blocks, calculating the document similarity between the second document in each second target data block and the first document in the first target data block, and performing document deduplication processing on the first target data block based on the document similarity, until the number of the second target data blocks is a preset number.

6. The method according to claim 1, characterized in that, Determining the target deduplication level corresponding to the initial corpus training data includes: Obtain the model training configuration parameters corresponding to the large language model training scenario; Obtain the training corpus metadata of the initial corpus training data; The target deduplication level corresponding to the initial corpus training data is determined based on the model training configuration parameters and the training corpus metadata.

7. The method according to claim 6, characterized in that, The step of determining the target deduplication level corresponding to the initial corpus training data based on the model training configuration parameters and the training corpus metadata includes: The model training configuration parameters and the training corpus metadata are input into the deduplication level recognition model to obtain the target deduplication level corresponding to the initial corpus training data. The deduplication level recognition model is obtained after training with the sample model training configuration parameters and sample training corpus metadata labeled with deduplication level.

8. A data deduplication device, characterized in that, The device includes: The data acquisition module is used to acquire initial corpus training data in large language model training scenarios; The data processing module is used to determine the target deduplication level corresponding to the initial corpus training data, obtain the target deduplication method corresponding to the target deduplication level, and perform data deduplication processing on the initial corpus training data based on the target deduplication method to obtain corpus training data. The target deduplication method includes substring deduplication method, document deduplication method and global deduplication method. Wherein, when the target deduplication method is the substring deduplication method, the step of performing data deduplication processing on the initial corpus training data based on the target deduplication method to obtain corpus training data includes: determining multiple reference repeating characters corresponding to the initial corpus training data based on the frequency of occurrence of each character in the initial corpus training data; obtaining a preset character filtering template, the preset character filtering template containing multiple filtering characters, performing character filtering processing on the multiple reference repeating characters according to the preset character filtering template, filtering out entity names present in the reference repeating characters, and obtaining at least one target repeating character corresponding to the initial corpus training data; and removing characters that appear repeatedly in sentences in the initial corpus training data according to each target repeating character, thereby obtaining corpus training data; When the target deduplication method is the global document deduplication method, the step of performing data deduplication processing on the initial corpus training data based on the target deduplication method to obtain corpus training data includes: performing data grouping processing on the initial corpus training data to obtain multiple target data blocks; determining a first target data block and a second target data block in the target data blocks, wherein the second target data block is a data block other than the first target data block in the target data blocks; calculating the document similarity between the second document in each second target data block and the first document in the first target data block, and performing document deduplication processing on the first target data block based on the document similarity.

9. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are adapted to be loaded by a processor and executed as method steps as claimed in any one of claims 1 to 7.

10. A terminal, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Data processing method, electronic equipment and computer readable storage medium

    CN110610450A

  • Text duplicate removal method and device, electronic equipment and storage medium

    CN114282511A