A large model training data deduplication method based on semantic clustering

Abstract

The application discloses a kind of big model training data deduplication method based on semantic clustering.The method is first filtered by Hash matching and MinHash structure to completely repeated and locally repeated text;Subsequently, a pre-training semantic encoding model is used to generate a deep semantic vector, and a density clustering method is used to construct a semantic cluster, converting global high-complexity comparison into local retrieval within the cluster;Then, combined with the efficient neighbor search based on FAISS, the semantic similar samples within the cluster are grouped and deduplicated;Finally, an information entropy sensing mechanism is introduced, and representative samples are selected according to the content complexity of the samples, to significantly reduce redundancy while maintaining corpus diversity.The application can effectively reduce the risk of model memory caused by data duplication, improve training efficiency, and reduce data processing cost, suitable for various large-scale corpus construction and privacy-sensitive scenarios.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data processing technology, specifically to a method for deduplicating large model training data based on semantic clustering. Background Technology

[0002] With the widespread application of Large Language Models (LLMs), their pre-training process typically requires massive amounts of text data from diverse sources and with complex structures. This type of data is often crawled from the internet or pieced together from multiple source datasets, naturally containing a large amount of repetitive, near-repetitive, and semantically redundant text. For example, the same news article may be reprinted on different websites, templated responses may exist in multi-turn dialogues, and highly similar paragraph structures may exist among a large number of web page contents. This redundant data leads to significant waste of computational resources during training, increases data cleaning costs, and reduces overall training efficiency.

[0003] At the model training level, duplicate samples can amplify the model's attention to specific texts, causing bias during parameter updates and leading to a homogenized output pattern. Simultaneously, the model may memorize the specific content of duplicate samples, resulting in text repetition during inference, a phenomenon known as "memory leakage," which threatens the security of private data or sensitive text. Therefore, effectively deduplicating the dataset before training large models is a crucial step in ensuring model quality, security, and training efficiency.

[0004] Existing data deduplication technologies mainly fall into three categories: First, precise deduplication methods based on string matching, such as hash comparison, can quickly remove completely identical text, but cannot identify paragraphs with the same meaning but different expressions; second, approximate deduplication based on minimum hash signature (MinHash) or locality-sensitive hashing (LSH), which can handle slightly modified text, but is insufficient for complex semantic equivalence cases; and third, methods that have emerged in recent years based on semantic comparison using embeddings generated by pre-trained models, which can identify deep semantic repetitions, but require high-complexity similarity retrieval across the entire vector space. When the data scale reaches billions of tokens or hundreds of gigabytes, the memory and computational overhead of vector retrieval increases exponentially, making it impractical for real-world engineering scenarios.

[0005] Furthermore, existing methods mostly focus on "removing duplicates" while paying less attention to "what to retain." Without a hierarchical structure or content complexity evaluation mechanism, the deduplication process is prone to mistakenly deleting high-value, high-information-entropy samples, thus reducing the linguistic diversity and knowledge coverage of the corpus and hindering the improvement of model generalization ability. Some studies have attempted to reduce retrieval overhead through clustering, but most methods rely on algorithms such as K-means (unsupervised learning clustering algorithm) that require a pre-set number of clusters, making them unsuitable for large-scale corpora with complex structures and uneven distributions.

[0006] Therefore, when faced with large-scale, multi-distributed, and strongly semantically related training data, a new deduplication technology is needed that can simultaneously: identify different types of multi-level repeated samples, maintain the semantic diversity and knowledge structure of the corpus, reduce the computational burden of semantic retrieval and clustering processes, and reduce the risk of memory leakage during model training. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for deduplicating large-scale model training data based on semantic clustering. This method constructs a multi-level, multi-stage semantic deduplication system and introduces semantic clustering and information entropy evaluation mechanisms. Its aim is to maintain deduplication accuracy while reducing computational costs, improving training data quality, and mitigating model memory risks.

[0008] To achieve the above objectives, this invention provides a method for deduplicating large model training data based on semantic clustering, comprising the following steps: Obtain the original text dataset D Then, it is preprocessed to obtain a text set. D 0; Use a hash algorithm to process the text collection. D 0. The text set obtained after removing completely duplicate text. D 1; For text collections D 1. Perform n-gram segmentation to construct the text's Shingle set and generate a MinHash feature signature; after mapping using a Locality Sensitive Hash (LSH) structure, perform Jaccard similarity screening within the same bucket, removing texts with Jaccard similarity greater than a first threshold to obtain the text set. D 2; Text collection D2. Input the pre-trained semantic encoding model to generate corresponding semantic vectors. Perform preliminary screening based on vector similarity to obtain a set of semantic vectors corresponding to the text. Input the set of semantic vectors into a density-based clustering algorithm to generate several semantic clusters and noise points. For each semantic cluster, use the FAISS tool to build a vector retrieval index within the cluster, perform a top-k nearest neighbor search, and compare neighboring texts based on cosine similarity. Similar texts with a similarity greater than a second threshold are identified as semantically redundant samples and grouped into the same repetition group. In each repetition group, only the texts with the strongest semantic representativeness or the highest information content are retained, and redundant samples are removed to obtain the text set. D 3; For text collections D In step 3, the content information entropy of each text is calculated based on its token distribution. Within the same semantic cluster, hierarchical retention is performed according to the entropy value to obtain the deduplicated training corpus.

[0009] Furthermore, for text sets D For each text in set 0, a hash value is calculated using a hash algorithm. The hash values ​​are then compared; if they match, the text is considered a complete duplicate, and only one text is retained, resulting in the text set. D 1.

[0010] Furthermore, the hash algorithm is either the SHA-256 algorithm or the xxHash64 algorithm.

[0011] Furthermore, for text sets D The text in step 1 is segmented into n-grams to construct a Shingle set, and a MinHash signature is generated based on the Shingle set. By constructing a Locality Sensitive Hash (LSH) structure, texts with similar MinHash signatures are mapped to the same or adjacent buckets. Within the same bucket, Jaccard similarity screening is performed on the text samples, and texts with a Jaccard similarity greater than a first threshold are removed, resulting in the text set. D 2.

[0012] Furthermore, the text collection D In step 2, the text input is pre-trained into a semantic encoding model to generate corresponding semantic vectors. Based on the generated semantic vectors, the vector similarity between texts is calculated, and texts with vector similarity greater than the third threshold are initially screened to obtain the set of semantic vectors corresponding to the texts.

[0013] Furthermore, the semantic encoding model is selected from one or more of the SimCSE, SBERT, E5-large, or M3E-base models, and the output vector dimension is 768, 1024, or 2048.

[0014] Furthermore, the FAISS tool is an index structure based on clustering + bucketing retrieval or a graph-based index structure.

[0015] To achieve the above objectives, the present invention also provides a deduplication device for large model training data based on semantic clustering, comprising one or more processors for implementing the above-described deduplication method for large model training data based on semantic clustering.

[0016] To achieve the above objectives, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described method for deduplication of large model training data based on semantic clustering.

[0017] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for deduplication of large model training data based on semantic clustering.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) Significantly improve deduplication efficiency: By using "clustering divide and conquer" and local vector retrieval, the global O(n²) semantic comparison complexity is significantly reduced, which is suitable for processing ultra-large-scale corpora; (2) Improve deduplication quality and semantic recognition capability: The multi-level deduplication structure covers three types of samples: complete repetition, local repetition and deep semantic repetition, making the deduplication results more comprehensive and accurate; (3) Enhance the diversity and coverage of the corpus: The proposed information entropy-aware sample selection mechanism can avoid excessive deduplication, effectively retain high-value content, and improve the generalization ability of the model. (4) Reduce the risk of model memory leakage: By reducing high-frequency repeated samples, the overfitting of the model to specific texts during training is effectively reduced, thereby improving the overall privacy and security. (5) Strong engineering feasibility: The clustering algorithm, vector retrieval structure and semantic coding model adopted in this invention can be stably deployed in large-scale data scenarios and have good scalability. Attached Figure Description

[0019] Figure 1 A schematic diagram of the overall process of the method for deduplication of large model training data based on semantic clustering provided in an embodiment of the present invention; Figure 2 A flowchart of text sample preprocessing provided in an embodiment of the present invention; Figure 3 A flowchart illustrating the generation of semantic vectors by a pre-trained semantic encoding model provided in this embodiment of the invention; Figure 4This is a flowchart illustrating the collaborative workflow of semantic cluster construction and intra-cluster retrieval provided in this embodiment of the invention. Figure 5 A schematic diagram illustrating the integration of the present invention as a data preprocessing module within the large model training framework provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the device structure provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0021] To facilitate a better understanding of the technical solutions of this invention by those skilled in the art, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following embodiments are preferred implementations given under the overall concept of this invention, used to illustrate the technical principles and implementation process of this invention, and do not constitute a limitation on the scope of protection of this invention. Equivalent substitutions or modifications made by those skilled in the art without departing from the concept of this invention should all fall within the scope of protection of this invention.

[0022] Example 1

[0023] This embodiment provides a method for deduplicating large model training data based on semantic clustering. It is applicable to ultra-large-scale text datasets consisting of Internet crawled corpora, open-source datasets, multi-turn dialogue logs, technical documents, or structured text conversion data, for example, for the pre-training or fine-tuning stage of GPT-type autoregressive language models.

[0024] Unlike existing deduplication methods that rely solely on string matching or simple similarity assessment, this embodiment employs a multi-level deduplication + semantic clustering + fine-grained intra-cluster deduplication approach, effectively reducing redundancy while avoiding semantic information loss. See also... Figure 1 The method includes the following steps: (1) Input data preprocessing.

[0025] Obtain the original text dataset D The dataset can be derived from web scraping corpora, forum Q&A, technical blogs, ebooks, or chat logs. In practical applications, the raw data often suffers from inconsistent encoding, chaotic formatting, or contains a large number of noisy characters.

[0026] like Figure 2 As shown, this embodiment first performs a unified normalization process on the original text, including but not limited to: normalizing the text to Unicode to eliminate the differences caused by different encoding forms; deleting non-text content such as HTML tags and script tags; deleting special symbols or control characters that do not contribute to semantics; unifying punctuation and whitespace characters; and performing language detection to filter text samples that are inconsistent with the target training language.

[0027] Through the above processing, a text set with uniform structure and standardized content is obtained. D 0 provides a stable input for subsequent deduplication steps.

[0028] This step can reduce the interference of format noise on subsequent hash calculations, n-gram segmentation, and semantic encoding, thereby improving the stability and accuracy of subsequent steps.

[0029] (2) Precise deduplication.

[0030] For text collections D For each text in 0, a hash value is calculated using a hash algorithm, such as SHA-256 or xxHash64, which offer high collision security. By comparing the hash values, text samples with identical hash values ​​are identified as completely duplicate text, and only one of them is retained. This quickly eliminates explicit duplicate data in the initial stage of the deduplication process, reducing subsequent computation and overall processing costs. For example, for completely duplicate text in web scraping corpora caused by mirror sites, repeated reposting, etc., a large proportion of data compression can be quickly achieved. After this precise deduplication step, a text set is obtained. D 1.

[0031] It should be noted that this embodiment uses precise deduplication as the first screening step, rather than directly performing complex semantic calculations, which can significantly reduce the computational load of the subsequent semantic model.

[0032] (3) Approximate deduplication (MinHash+LSH).

[0033] Even after precise deduplication, there may still be a large number of texts with highly similar structures but subtle differences, such as templated news, rewritten advertising copy, or text spliced ​​together with content.

[0034] Therefore, this step involves the text collection. DThe text in step 1 is segmented into n-grams (e.g., 3-grams or 5-grams) to construct a Shingle set for the text, and a MinHash signature is generated based on this set. Subsequently, a Locality Sensitive Hashing (LSH) structure is constructed to map texts with similar MinHash signatures to the same or adjacent buckets. Within the same bucket, a rapid Jaccard similarity screening is performed on the text samples to remove texts with high structural repetition (e.g., Jaccard similarity greater than 0.8), resulting in the text set. D 2.

[0035] By performing similarity assessments on text samples within the same bucket, texts that are structurally highly similar but not completely identical can be efficiently identified. This further eliminates shallow repetitions or templated text, providing a more concise data foundation for subsequent semantic-level processing. For example, in news corpora or product description corpora, it can effectively eliminate duplicate text that is "different in title but not in content" or "partially replaced with words." Moreover, this step eliminates shallow repetitions in advance without introducing high-dimensional semantic computation, reducing the sample size in the semantic vector generation stage.

[0036] (4) Generate semantic vectors.

[0037] See Figure 3 For text collections D The text samples in step 2 are input into a pre-trained semantic encoding model to generate corresponding semantic vector representations. The semantic encoding model can be one or more models such as SimCSE, SBERT, E5-large, or M3E-base, and its output vector dimension can be 768, 1024, or 2048. Based on the generated semantic vectors, the vector similarity between text samples is calculated to initially screen text samples with high semantic similarity (e.g., vector similarity greater than 0.95), identifying data with significant semantic redundancy and providing a vector basis for subsequent clustering operations. In practical applications, such as question-and-answer or dialogue corpus processing, semantic vectors can capture the semantic equivalence relationships of texts, rather than relying solely on literal similarity.

[0038] Through the above processing, we obtain the semantic vector set E corresponding to the text:

[0039] Compared to traditional methods, this embodiment utilizes deep semantic representation instead of relying solely on TF-IDF (term frequency-inverse document frequency) or term frequency features as the basis for subsequent clustering and deduplication. Deep semantic representation maps text to a semantic space through a neural network model, enabling semantically similar texts to have higher similarity in the vector space, thereby overcoming the limitations of word-based matching methods.

[0040] (5) HDBSCAN semantic clustering.

[0041] See Figure 4 The semantic vector set E is input into a density-based clustering algorithm (HDBSCAN model). The HDBSCAN model automatically generates several semantic clusters C1…C1 based on density reachability. r And noise points. Unlike K-means, which requires a preset number of clusters, HDBSCAN can adaptively determine the number of clusters and cluster boundaries, making it particularly suitable for large-scale text data with uneven semantic distribution. This clustering method does not require a pre-set number of clusters; it can adaptively determine the cluster structure and boundaries based on the vector distribution, breaking down the original global semantic similarity judgment problem into multiple problems within local semantic clusters. The main function of clustering is not to directly remove duplicates, but to divide the global semantic space into multiple local semantic regions so that high-precision deduplication can be performed within the clusters later.

[0042] It should be noted that noise point samples are not easy to form stable clusters with other texts, and usually represent semantically unique content, so they are retained as independent samples.

[0043] Through semantic clustering, the original The global similarity judgment problem is transformed into multiple local subproblems, which significantly reduces computational complexity.

[0044] (6) High-precision semantic deduplication within the cluster.

[0045] For each semantic cluster C obtained by HDBSCAN clustering i A vector retrieval index is constructed within clusters using FAISS (with an index structure of IVF-FLAT or HNSW). FAISS is an artificial intelligence similarity search tool, IVF-FLAT is an index structure based on clustering and bucketing, and HNSW is a graph-based index structure. FAISS performs a top-k nearest neighbor search on the vectors within the clusters and performs a refined comparison of nearest neighbor texts based on cosine similarity. Similar texts with a similarity greater than a preset threshold (e.g., 0.75–0.90) are identified as semantically redundant samples and grouped into the same repetition group.

[0046] In each repeated group, only the text sample with the strongest semantic representativeness or the highest information content is retained, and the remaining redundant samples are removed, thus obtaining the deduplicated text set. D 3.

[0047] Compared to the global retrieval method, this step improves overall processing efficiency while ensuring deduplication accuracy by limiting the retrieval scope to a single semantic cluster.

[0048] (7) Information entropy perception sample retention.

[0049] To avoid accidentally deleting high-information-density text during the deduplication process, this embodiment further introduces information entropy as a criterion for sample selection. After identifying semantically redundant samples within a cluster, the content information entropy of text samples within the same semantic cluster is calculated to measure the amount of information and expressive diversity contained in the text.

[0050] Specifically, for text collections D In step 3, the content information entropy of each text sample is calculated based on its token distribution to measure the information richness of the text. Within the same semantic cluster, a hierarchical preservation strategy is implemented based on the entropy value: Execute within the cluster based on entropy value:

[0051] Wherein, parameter H is the text information entropy value, parameter Let be the probability of the i-th token appearing in the current text sample.

[0052] If the sample is high-entropy (information-rich), it will be retained first; if it is medium-entropy, it will be retained according to weight; if it is low-entropy (template / repetition), the number of samples retained will be minimized.

[0053] The text samples retained from each semantic cluster after the above steps are summarized and merged to form a deduplicated text dataset for training large models, ultimately yielding high-quality deduplicated training corpus. This dataset has low semantic redundancy while maintaining high information density and content diversity, making it suitable for pre-training or fine-tuning stages of large-scale language models.

[0054] Figure 5 The diagram illustrates the integration of this invention as a data preprocessing module within the large model training framework provided in this embodiment, demonstrating the application of this invention in the complete training process.

[0055] Example 2

[0056] This embodiment, based on the method described in Embodiment 1, further provides a specific implementation of training data deduplication based on semantic vector generation and density clustering to address the problem that relying solely on surface similarity makes it difficult to identify semantically equivalent text. The purpose of this embodiment is to: map text samples to a unified vector space; utilize the distance relationship between semantic vectors to achieve semantic-level similarity modeling; narrow the comparison range of similar texts through clustering; and achieve high-precision semantic deduplication while ensuring computational efficiency.

[0057] (1) Multi-model parallel distribution encoding: The text set after preprocessing and initial deduplication is distributed. DEach text in section 2 is simultaneously input into multiple pre-trained semantic encoding models for parallel encoding, and the encoding models include at least: First semantic coding model: BERT coding model trained based on the SimCSE method (SimCSE-BERT-base). Second semantic encoding model: Text embedding model trained by contrastive learning (E5-large-v2); The third semantic encoding model: an embedding model optimized for Chinese semantic matching (M3E-base).

[0058] For the same text The corresponding semantic vectors are obtained respectively. : , ,

[0059] The output vector dimensions of each model may differ. For example, the output vector dimension of SimCSE-BERT-base is 768; the output vector dimension of E5-large-v2 is 1024; and the output vector dimension of M3E-base is 768.

[0060] (2) Dimension alignment and normalization of heterogeneous vectors: To address the issue of inconsistent dimensions of output vectors from different models, a linear projection layer is constructed for each encoding model. :

[0061] in, For the first The original output dimensions of each model; To ensure a consistent target dimension, a value of 1024 is preferred. For the first The original output vector space of the model (dimension 1) ), To unify the target vector space (dimension 1) ).

[0062] The parameters of the linear projection layer can be obtained in one of the following ways: random initialization followed by fixing during system operation; unsupervised alignment training using a small number of samples; or orthogonal initialization to maintain the vector space structure.

[0063] Dimension alignment and normalization: Since different models output different dimensions (e.g., 768 dimensions and 1024 dimensions), a linear projection layer is used to unify the dimensions to 1024 dimensions, and then... L 2. Normalization.

[0064] (3) Dynamic weighted fusion: At the dataset level or batch level, the domain characteristics of the statistical text are analyzed, including but not limited to: the proportion of professional terms; the density of numbers and symbols; the complexity of sentences; and the identification of text sources.

[0065] Based on domain characteristics, the fusion weight parameters α, β, and γ of the three models are determined, and the original output vector is... Where α + β + γ = 1, the weights are determined by methods including: preset rule mapping (e.g., increasing model weights when technical terms account for a high proportion); fixed configuration based on historical validation set statistical results; and weight values ​​output based on a simple regression model. α, β, and γ can be taken as 0.4, 0.3, and 0.3, respectively.

[0066] Calculate the fusion vector e The formula is:

[0067] (4) Enhanced Clustering Stability: The fused vector set is input into a density-based clustering model for clustering processing. For each text sample, its corresponding soft clustering confidence score is obtained to indicate the reliability of the sample being assigned to the current cluster. When the clustering confidence of a sample is lower than a preset threshold (e.g., 0.5–0.7), the sample is marked as a low-confidence sample and is separately classified into an independent category or a noise set to avoid it interfering with the main semantic cluster.

[0068] Through the above-mentioned multi-model heterogeneous feature extraction and fusion mechanism, the accuracy of semantic repetition recognition in professional domain texts (medical, legal, etc.) has been improved; the consistency of multimodal transcribed texts (Optical Character Recognition (OCR) and Automatic Speech Recognition (ASR)) in the semantic space has been enhanced; and the stability and robustness of the clustering process have been strengthened.

[0069] Example 3

[0070] Because training corpora inevitably contain private information such as personal identification information, contact information, and addresses; semantically repetitive private texts are easily memorized by the model with high probability during multiple training sessions; and traditional deduplication methods do not differentiate private samples.

[0071] Therefore, this embodiment achieves fine-grained control over privacy-sensitive text through privacy recognition, independent modeling, and high-threshold deduplication.

[0072] (1) Implementation of privacy-sensitive text recognition.

[0073] A predefined set P of privacy-sensitive information types is included, including but not limited to: identity information (name, ID card number, passport number); contact information (phone number, email address); location information (home address, detailed geographic location); and account information (account number, bank card number).

[0074] The text samples are subjected to rule-based detection, which includes: using regular expressions to match fixed-format fields; performing length and check digit checks on numeric fields; and performing format checks on email addresses, phone numbers, etc.

[0075] Successfully matched text is marked as candidate privacy text. Further named entity recognition processing is performed on the candidate privacy text: a pre-trained entity recognition model is used to identify entities such as names of people, locations, and organizations in the text; when the recognition result matches any type in the privacy type set P, the text is confirmed as privacy-sensitive text. The confirmed text is added to the privacy text set. D p Set the privacy flag field privacy-flag=1 for this text; add the remaining text to the plain text collection. D n And set privacy-flag=0.

[0076] The privacy-flag flag represents a privacy-sensitive flag in the text, used to mark whether the text contains privacy information.

[0077] (2) Independent semantic modeling process for privacy text.

[0078] Based on the above privacy identification results, for text samples with privacy-flag=1, their corresponding texts have been classified into the privacy text set. .

[0079] To avoid semantic mixing between privacy information and ordinary text during semantic modeling, a privacy text set is used. The specific process for performing independent semantic modeling is as follows: For private text collections Each text in The vector representation is generated using a semantic coding model that is the same as or independently configured as in Example 2:

[0080] After semantic encoding is completed, all privacy text vectors are constructed into an independent privacy vector pool. :

[0081] in, Represents a collection of private texts The first in Sample text; This represents a semantic encoding model; Representing text The corresponding semantic vector representation; Indicates the number of samples in the privacy text set; This represents an independent privacy vector pool.

[0082] To avoid cross-category semantic interference, privacy-preserving text vectors are not jointly modeled or uniformly clustered with ordinary text vectors. Privacy Vector Pool It is used only for semantic similarity calculation and duplicate detection between private texts, and does not participate in the semantic clustering process of ordinary texts, thereby avoiding the absorption of private samples by ordinary semantic clusters.

[0083] (3) High-threshold semantic clustering and deduplication of privacy texts.

[0084] Privacy Vector Pool E p Clustering is performed with the following parameters set: a larger minimum cluster size threshold; stricter similarity constraints; and lower noise tolerance.

[0085] Within the privacy semantic cluster, compute any two privacy texts. The semantic similarity between them, when they satisfy:

[0086] in, For privacy purposes, the threshold should preferably be set to a value greater than 0.9.

[0087] For groups of privacy texts identified as duplicates, the following policy is applied: only one representative text is retained; the remaining texts are marked for deletion or anonymization. This setting ensures that privacy texts are only aggregated if they are semantically highly consistent.

[0088] For example, number masking (123→XXX), address obfuscation (xx city xx district), and initialization of names can effectively reduce the probability of model "memorization".

[0089] Compared with deduplication methods that do not distinguish between private text and other private text, this embodiment significantly reduces the number of times private text appears repeatedly in the training corpus; reduces the risk of reproducing the original private text during the model generation stage; and improves the compliance and security of training data without significantly affecting model performance.

[0090] Corresponding to the aforementioned embodiments of the large model training data deduplication method based on semantic clustering, the present invention also provides embodiments of a large model training data deduplication device based on semantic clustering.

[0091] See Figure 6 The deduplication device for large model training data based on semantic clustering provided in this embodiment of the invention includes one or more processors for implementing the deduplication method for large model training data based on semantic clustering in the above embodiment.

[0092] The embodiment of the large-scale model training data deduplication device based on semantic clustering of this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6 The diagram shown is a hardware structure diagram of any data processing-capable device, including the large model training data deduplication device based on semantic clustering of this invention. (Except for...) Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0093] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0094] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0095] Corresponding to the aforementioned embodiments of the large model training data deduplication method based on semantic clustering, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the large model training data deduplication method based on semantic clustering as described above. Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities, used for the deduplication method of large model training data based on semantic clustering provided in this application embodiment. (Except for...) Figure 7In addition to the processor, memory, DMA controller, disk, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0096] Corresponding to the aforementioned embodiments of the large model training data deduplication method based on semantic clustering, this embodiment of the invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the large model training data deduplication method based on semantic clustering in the above embodiments.

[0097] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0099] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for deduplicating training data of large models based on semantic clustering, characterized in that, Includes the following steps: Obtain the original text dataset D Then, it is preprocessed to obtain a text set. D 0; Use a hash algorithm to process the text collection. D 0. The text set obtained after removing completely duplicate text. D 1; For text collections D 1. Perform n-gram segmentation, construct the Shingle set of the text, and generate MinHash feature signature; After mapping using the Locality Sensitive Hash (LSH) structure, Jaccard similarity screening is performed within the same bucket to remove texts with a Jaccard similarity greater than a first threshold, resulting in a text set. D 2; Text collection D 2. Input the pre-trained semantic encoding model to generate corresponding semantic vectors, perform preliminary screening based on vector similarity to obtain the semantic vector set corresponding to the text; input the semantic vector set into a density-based clustering algorithm to generate several semantic clusters and noise points; For each semantic cluster, a vector retrieval index is built within the cluster using the FAISS tool. A top-k nearest neighbor search is performed, and neighboring texts are compared based on cosine similarity. Texts with a similarity greater than a second threshold are identified as semantically redundant samples and grouped into the same repetition group. Within each repetition group, only the texts with the strongest semantic representativeness or highest information content are retained, and redundant samples are removed to obtain the text set. D 3; For text collections D In step 3, the content information entropy of each text is calculated based on its token distribution. Within the same semantic cluster, hierarchical retention is performed according to the entropy value to obtain the deduplicated training corpus.

2. The method for deduplicating large model training data based on semantic clustering according to claim 1, characterized in that, For text collections D For each text in set 0, a hash value is calculated using a hash algorithm. The hash values ​​are then compared; if they match, the text is considered a complete duplicate, and only one text is retained, resulting in the text set. D 1.

3. The method for deduplicating large model training data based on semantic clustering according to claim 1, characterized in that, The hash algorithm is either SHA-256 or xxHash64.

4. The method for deduplicating large model training data based on semantic clustering according to claim 1, characterized in that, For text collections D The text in step 1 is segmented into n-grams to construct a Shingle set, and a MinHash signature is generated based on the Shingle set. By constructing a Locality Sensitive Hash (LSH) structure, texts with similar MinHash signatures are mapped to the same or adjacent buckets. Within the same bucket, Jaccard similarity screening is performed on the text samples, and texts with a Jaccard similarity greater than a first threshold are removed, resulting in the text set. D 2.

5. The method for deduplicating large model training data based on semantic clustering according to claim 1, characterized in that, Text collection D In step 2, the text input is pre-trained into a semantic encoding model to generate corresponding semantic vectors. Based on the generated semantic vectors, the vector similarity between texts is calculated, and texts with vector similarity greater than the third threshold are initially screened to obtain the set of semantic vectors corresponding to the texts.

6. The method for deduplicating large model training data based on semantic clustering according to claim 1, characterized in that, The semantic encoding model is selected from one or more of the SimCSE, SBERT, E5-large or M3E-base models, and the output vector dimension is 768, 1024 or 2048.

7. The method for deduplicating large model training data based on semantic clustering according to claim 1, characterized in that, The FAISS tool is an index structure based on clustering and bucketing retrieval or a graph-based index structure.

8. A deduplication device for large model training data based on semantic clustering, characterized in that, It includes one or more processors for implementing the method for deduplication of large model training data based on semantic clustering as described in any one of claims 1-7.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the semantic clustering-based deduplication method for large model training data as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method for deduplication of large model training data based on semantic clustering as described in any one of claims 1-7.

Images