High-similarity duplicated data cleaning method, system and equipment and medium

Abstract

The invention provides a high-similarity duplicated data cleaning method, system and device and a medium, and relates to the technical field of high-similarity duplicated data cleaning.The method comprises the steps that a text data set is read; performing hash processing and cluster division on the text data set to obtain similar duplicated data clusters and non-duplicated data clusters; executing duplicate checking operation and duplicate removal operation to obtain a duplicate data set and a data set after duplicate removal; and respectively carrying out sampling check on the repeated data set and the data set after deduplication, correcting erroneous judgment data and dynamically optimizing parameters of the hash function. According to the method, high-similarity repeated data in the text data set can be accurately recognized, efficient cleaning of the large-scale text data set is achieved, meanwhile, dynamic optimization can be achieved, and adaptability is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of high similarity repetitive data processing, and particularly relates to a high similarity repetitive data cleaning method, system, device and medium. BACKGROUND

[0002] With the development of big data and artificial intelligence technology, large models (i.e. large-scale pre-training models) have been widely applied in various fields. The training and optimization of large models require a large amount of high-quality data as support. However, there are often a large amount of high similarity repetitive data in the text data set used to train large models. High similarity repetitive data refers to data records with extremely similar or almost identical content, which can be completely repeated, partially repeated, or semantically repeated. These high similarity repetitive data not only occupy a large amount of storage resources, but also can cause bias and overfitting in the training process of large models, reducing the efficiency and accuracy of large model training. Therefore, it is of great practical significance to clean the high similarity repetitive data in the text data set of large models.

[0003] Currently, the methods for cleaning high similarity repetitive data in the text data set of large models mainly include rule-based cleaning methods and machine learning-based cleaning methods. Among them, the rule-based cleaning method is to identify and delete high similarity repetitive data according to human pre-set rules; and the machine learning-based method is to train a classification model to automatically identify and delete repetitive data.

[0004] However, the existing rule-based high similarity repetitive data cleaning method can only identify completely identical or very similar text data, and it is difficult to accurately identify semantically repetitive data, i.e. data with similar semantics but different expressions, and cannot achieve accurate processing of high similarity repetitive data. Although the machine learning-based method can process a certain degree of similarity data, when processing large-scale text data sets, it needs to calculate high-dimensional feature vectors for each data and compare the similarity of each pair of data, resulting in exponential growth of computational complexity, low efficiency, and the accuracy of the model is easily affected by data noise and feature selection. In addition, most of the existing methods lack dynamic optimization mechanism and are difficult to adapt to the dynamic changes of data in the text data set. SUMMARY

[0005] In view of the technical problems that the rule-based method in the existing high-similarity repeated data cleaning method is difficult to accurately process high-similarity repeated data, the method based on machine learning has high computational complexity, low efficiency and low accuracy when processing large-scale text data sets, and the existing method lacks a dynamic optimization mechanism, the present application provides a high-similarity repeated data cleaning method, system, device and medium, which can accurately identify high-similarity repeated data in a text data set, efficiently clean a large-scale text data set, and dynamically optimize and adapt.

[0006] In a first aspect, the present application provides a high-similarity repeated data cleaning method, comprising the following steps: S1. reading a text data set to be processed; S2. performing hash processing and cluster division on the text data set, comprising: extracting n-gram sequences from each text data in the text data set, and encoding the text content into a byte stream; applying k different hash functions to the byte stream to generate k hash values, k >= 3, and taking the minimum value of all hash values as the minimum hash value of the data item, and collecting all data items to form a new data set with minimum hash values; constructing an undirected graph based on the minimum hash values of the data items, running a parallel connected component algorithm, grouping data items with the same minimum hash value into the same connected subgraph, and forming a similar repeated data cluster, and grouping isolated vertices into a non-repeated data cluster; S3. performing a duplicate checking operation and a duplicate removal operation, wherein: the duplicate checking operation is to classify and save the data in each repeated data cluster, and generate an independently stored repeated data set; the duplicate removal operation is to retain all data of the non-repeated data cluster and unique representative data in each repeated data cluster, and form a deduplicated data set; S4. sampling and checking the repeated data set and the deduplicated data set respectively, correcting misjudgment data, and dynamically optimizing the parameters of the hash function.

[0007] It should be further explained that in step S1, the format of the text data set includes at least one of JSON, PARQUET and CSV.

[0008] It should be further explained that in step S2, the text content is encoded into a byte stream using UTF-8 encoding.

[0009] It should be further explained that in step S2, when extracting n-gram sequences: the value of n is in the range of 3-5, and n=3 is preferred; for text items with a length less than n, special characters are filled to a length of n before processing; The extracted n-gram sequence needs to be filtered by a stop word list containing punctuation marks and common words.

[0010] It should be further explained that the common words include functional words, pronouns, high-frequency low-information words, and domain general words, wherein the functional words include auxiliary words, prepositions, and conjunctions.

[0011] It should be further explained that in step S2, the k different hash functions are generated by the following method: Select a reference hash function Generate its variant function through prime modulus operation , The formula is:

[0012] In the formula, is the original data item The initial hash value generated by processing the reference hash function is a large prime number of 1024 bits or more; represents the modulus operation, that is, the remainder; is a random integer, ; is a random integer, .

[0013] Preferably, the reference hash function is MurmurHash3.

[0014] It should be further explained that in step S2, when constructing the undirected graph based on the minimum hash value of the data item, the Apache Spark GraphFrames graph processing library is used to run the parallel connected component algorithm, and the hash calculation task of each data item is distributed to the cluster nodes for parallel execution.

[0015] It should be further explained that step S3 uses a big data storage and computing base for data storage and calculation.

[0016] It should be further explained that in step S3, in each cluster of duplicate data, the selection rule for uniquely representing data is to select the data item with the smallest Hamming distance of the minimum hash value in the duplicate data cluster.

[0017] It should be further explained that the sampling and checking step of step S4 includes: S401. Randomly sample the verification sample set from the text data set according to a predetermined proportion to form a verification data set, use the verification data set to perform the operations of steps S2-S3, and obtain a duplicate verification data set after deduplication of the verification data set. S402. Artificially checking the duplicate checking data set and the deduplicated checking data set respectively to screen false positive data and false negative data, wherein the false positive data are data in the duplicate checking data set that are actually not duplicated with other data, and the false negative data are data in the deduplicated checking data set that are duplicated with other data; S403. Storing the false positive data from the duplicate checking data set to the deduplicated checking data set; and for each group of false negative data, only retaining unique representative data in the deduplicated checking data set, and storing the rest of the false negative data to the duplicate checking data set; S404. Based on the duplicate checking data set and the deduplicated checking data set obtained in step S403, using an iterative optimization algorithm to optimize the parameters of the hash function, and applying the optimized hash function to the processing of subsequent incremental data through a parameter update interface.

[0018] In a second aspect, the present application provides a high-similarity duplicate data cleaning system for implementing the high-similarity duplicate data cleaning method described above, comprising: A text data set reading module for reading a text data set to be processed; A hashing processing and cluster division module for performing hashing processing and cluster division on the text data set to obtain a similar duplicate data cluster and a non-duplicate data cluster; A duplicate checking and deduplication module for performing duplicate checking and deduplication operations to obtain a duplicate data set and a deduplicated data set; A sampling verification module for performing sampling verification on the duplicate data set and the deduplicated data set respectively, correcting misjudgment data, and dynamically optimizing the parameters of the hash function.

[0019] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor is configured to implement the steps of the high-similarity duplicate data cleaning method described above when executing the computer program.

[0020] In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the high-similarity duplicate data cleaning method described above.

[0021] The present application has the following advantages: 1. The high-similarity duplicate data cleaning method provided by the present application can effectively identify semantically duplicated data by extracting n-gram sequences and generating minimum hash values, and can accurately divide high-similarity duplicate data into duplicate data clusters even if the expressions of these data are different, thereby achieving comprehensive and accurate processing of high-similarity duplicate data and improving the accuracy of high-similarity duplicate data cleaning.

[0022] 2. The present invention performs clustering by constructing an undirected graph connected set based on the minimum hash value, transforming the duplicate data identification problem into the connected component calculation problem in graph theory. Combined with the parallel graph processing algorithm, it realizes distributed parallel computing of data similarity comparison tasks, thereby improving the processing efficiency of similarity comparison. Duplicate checking and deduplication operations are performed based on the data clustering results, which greatly reduces unnecessary similarity calculations in the duplicate checking and deduplication process, further improving the efficiency of processing large-scale data.

[0023] 3. This invention uses dynamic multi-hash function generation technology to generate k ≥ 3 independent hash functions through prime number modular operations for joint judgment. It focuses on the intrinsic relationship between data and does not rely on specific feature selection. It reduces the interference of data noise on data processing accuracy, allowing the model to maintain high accuracy in complex data environments.

[0024] 4. After completing the deduplication and duplicate checking operations, the present invention performs sampling verification on the deduplication data set and the duplicate data set, which can timely discover misjudgment data and dynamically adjust the parameters of the hash function based on the iterative optimization algorithm, so that the hash function cluster can adapt to changes in data distribution characteristics. This enables the model to be continuously optimized as the data in the text data set changes dynamically, continuously maintaining good data processing effects, with strong adaptability, and the misjudgment rate can decrease exponentially with the increase in iteration rounds.

[0025] 5. This invention employs a dual-channel output mechanism for independently stored duplicate datasets and deduplicated datasets. This ensures data cleansing integrity while providing data selection options of varying granularity for subsequent model training. The duplicate datasets provide a traceability basis for data quality analysis, while the deduplicated datasets significantly reduce storage space usage by retaining unique representative data for each cluster, effectively addressing the problem of storage resource waste caused by redundant data. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 The present invention is a flowchart of a method for cleaning duplicate data with high similarity in one embodiment.

[0028] Figure 2 It is a schematic block diagram of a high-similarity duplicate data cleaning system in one embodiment of the present invention.

[0029] Figure 3Fig. 1 is a schematic diagram of a hardware structure of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objectives, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the present application will be clearly and completely described below in combination with the drawings in the specific embodiments. Obviously, the following described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0031] The high similarity repetitive data cleaning method involved in the present application is mainly directed to the technical field of high similarity repetitive data cleaning. Through the technical means of extracting n-gram sequences and generating minimum hash values, the similarity judgment of text content is converted into the equivalence comparison of hash values, which can effectively identify semantically repetitive data. Even if the expressions of these data are different, the high similarity repetitive data can be accurately divided into repetitive data clusters, so as to realize comprehensive and accurate processing of high similarity repetitive data, and the accuracy of high similarity repetitive data cleaning is high. Through the construction of an undirected graph connected set based on minimum hash values for cluster division, the repetitive data identification problem is converted into the connected component calculation problem in graph theory, and the distributed parallel computing of data similarity comparison tasks is realized by combining a parallel graph processing algorithm, which improves the processing efficiency of similarity comparison. Based on the data cluster division result, the duplicate checking and removal operation is performed, which greatly reduces unnecessary similarity calculation in the duplicate checking and removal process, and further improves the efficiency of processing large-scale data. Through the multi-hash function dynamic generation technology, k≥3 independent hash functions are generated through prime modulus operation for joint judgment, the internal relationship between data is focused, the model still maintains high accuracy in a complex data environment without relying on specific feature selection, the interference of data noise on data processing accuracy is reduced, the model can be continuously optimized with the dynamic change of data in the text data set, and good data processing effect is continuously maintained, which has strong adaptability and the misjudgment rate can decrease exponentially with the increase of iteration rounds. Through the design of the double-path output mechanism of the independently stored repetitive data set and the deduplicated data set, the data cleaning integrity is ensured, and different granularity data selection schemes are provided for subsequent model training, wherein the repetitive data set can provide traceability basis for data quality analysis, the deduplicated data set significantly reduces the storage space occupation by retaining the unique representative data of each cluster, and effectively solves the problem of waste of storage resources caused by redundant data.

[0032] The high similarity repetitive data cleaning method related to the present application is mainly aimed at the technical problems that the rule-based method in the existing high similarity repetitive data cleaning method is difficult to accurately process high similarity repetitive data, the method based on machine learning has high computational complexity, low efficiency and low accuracy when processing large-scale text data sets, and the existing method lacks a dynamic optimization mechanism.

[0033] The high similarity repetitive data cleaning method related to the present application will be described in detail below. For the purpose of illustration but not for the purpose of limitation, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0034] In the high similarity repetitive data cleaning method related to the present application, the term "comprising" indicates the presence of the described features, whole, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof. The terms "comprising", "including", "having" and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0035] In order to clearly describe the technical solutions of the present application, the terms "first", "second", etc. are used to distinguish the same items or similar items with basically the same function and role. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. do not necessarily mean different.

[0036] The phrase "one embodiment" or "some embodiments" appearing in the present application means that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" appearing in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized.

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0038] The high-similarity repetitive data cleaning method provided by the embodiment of the present application is executed by a computer device, and accordingly, the high-similarity repetitive data cleaning system runs in the computer device.

[0039] Figure 1 The flowchart of the high-similarity repetitive data cleaning method of one embodiment of the present application is shown in FIG. 1. In the flowchart, Figure 1 The execution subject can be a high-similarity repetitive data cleaning system. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0040] As shown in FIG. 2, the high-similarity repetitive data cleaning method includes the following steps. Figure 1 Step S1, reading a text data set to be processed.

[0041] The step of reading the text data set can ensure that the data can be completely and accurately acquired, avoid data loss or errors caused by data reading problems, and provide protection for subsequent data-based operations.

[0042] In some specific embodiments, the format of the data set includes at least one of JSON, PARQUET, and CSV.

[0043] In actual application, different business scenarios and data sources will produce different formats of data. By specifying the optional formats of the text data set, it is ensured that the high-similarity repetitive data cleaning method can be compatible with multiple common data formats, so that users can directly use the cleaning method without complex conversion of the data format, thereby reducing the data processing threshold and cost and expanding the application range of the method.

[0044] Step S2, performing hash processing and cluster division on the text data set, including: extracting an n-gram sequence for each text data in the text data set, and encoding the text content into a byte stream; applying k different hash functions to the byte stream to generate k hash values, k≥3, taking the minimum value of all hash values as the minimum hash value of the data item, and collecting all data items to form a new data set with the minimum hash value; constructing an undirected graph based on the minimum hash value of the data item, running a parallel connected component algorithm, grouping data items with the same minimum hash value into the same connected subgraph, forming a similar repetitive data cluster, and grouping isolated vertices into a non-repetitive data cluster.

[0045] The fingerprint features generated by the multiple hash functions are used for similarity modeling, and the distributed calculation of data relationships is realized by the undirected graph connected component algorithm, which greatly reduces the time and space complexity of the traditional clustering algorithm while ensuring the accuracy of similar data classification, and adapts to the massive data processing demand.

[0046] ​In some embodiments, the text content is encoded into a byte stream using UTF-8 encoding.

[0047] By eliminating parsing errors caused by character set differences through mandatory UTF-8 encoding, ensuring consistency in cross-language text processing, avoiding hash calculation bias caused by encoding confusion, and improving data representation stability.

[0048] In some embodiments, when extracting n-gram sequences: n is in the range of 3-5, preferably n=3; For text items with length less than n, fill special characters to length n and process; The extracted n-gram sequence needs to be filtered by a stop word list, which includes punctuation marks and common words.

[0049] Through the dynamic padding mechanism to solve the problem of short text processing, combined with stop word filtering to effectively extract core semantic features, it can balance the calculation efficiency and semantic capture ability in a 3-5 character window, and improve the accuracy of similarity judgment.

[0050] In some embodiments, common words include function words, pronouns, high-frequency low-information words, and domain general words, where function words include auxiliary words, prepositions, conjunctions; Among them, auxiliary words include structural auxiliary words (such as de, di, and de), dynamic auxiliary words (such as le, zhe, and guo), and mood auxiliary words (such as ma, ne, and ba); Prepositions include spatial relationship prepositions (such as on, and towards), time relationship prepositions (such as from, to, and when), and logical relationship prepositions (such as about and for); Conjunctions include parallel conjunctions (such as and, with, and and), transition conjunctions (such as but, however), and causal conjunctions (such as because and therefore); Pronouns include personal pronouns (such as I, you, he, they, and myself), demonstrative pronouns (such as this, that, and here), interrogative pronouns (such as who, where, and how), and indefinite pronouns (such as someone, some, and anything).

[0051] A multi-level stop word filtering system can accurately remove grammatical function words and high-frequency interference words, focus on text features with distinguishing degrees, strengthen the semantic relevance of similarity judgment, and reduce false positives caused by lexical surface similarity.

[0052] In some embodiments, k different hash functions are generated as follows: Select a reference hash function Generate its variant function through prime modulus operation , The formula is:

[0053] In the formula, is the original data item is the initial hash value generated by processing the reference hash function is a large prime number greater than 1024 bits; represents a modulo operation, that is, a remainder calculation; is a random integer, ; is a random integer, .

[0054] The prime number modulo operation is used to construct a hash function family with strong randomness, which can ensure uniform distribution of the hash space, effectively reduce the probability of hash collision, and maintain controllability of function generation, thereby providing a reliable mathematical foundation for similarity detection.

[0055] In some specific embodiments, the reference hash function is MurmurHash3.

[0056] MurmurHash3 is selected as the reference hash function, which can fully utilize its advantages of high efficiency and low collision rate, improve processing speed under the premise of ensuring hash quality, and is particularly suitable for streaming data and real-time processing scenarios.

[0057] In some specific embodiments, when a directed graph is constructed based on the minimum hash value of each data item, the Apache Spark GraphFrames graph processing library is used to run a parallel connected component algorithm, and the hash calculation task of each data item is distributed to cluster nodes for parallel execution.

[0058] The distributed graph computing framework is used to implement parallel processing of large-scale data relationships, which can break through the size limit of traditional single-machine algorithms and significantly improve the efficiency of connected component calculation through task distribution of cluster nodes.

[0059] Step S3, performing a duplicate detection operation and a deduplication operation, wherein: The duplicate detection operation is to classify and save the data in each duplicate data cluster to generate an independently stored duplicate data set; The deduplication operation is to retain all data of the non-duplicate data cluster and unique representative data in each duplicate data cluster to form a deduplicated data set.

[0060] A hierarchical data processing mechanism is established, which retains data traceability through independent storage of duplicate data sets, maintains data integrity during deduplication, and balances cleaning efficiency and data security, thereby providing a high-quality basic data set for subsequent data analysis.

[0061] In some embodiments, step S3 uses a big data storage and computing base for data storage and calculation.

[0062] The big data storage and computing base has strong storage and computing capabilities, can handle large-scale data, and can provide efficient data read-write and computing performance when performing duplicate detection and deduplication operations, ensuring data processing speed and stability. It also supports distributed computing and parallel processing, can fully utilize cluster computing resources, speed up data cleaning processes, and adapt to the characteristics of large model data volume.

[0063] In each duplicate data cluster, the selection rule for the unique representative data is to select the data item with the smallest hash value and the smallest Hamming distance in the duplicate data cluster.

[0064] Step S4, respectively, sample and check the duplicate data set and the deduplicated data set, correct misjudgment data and dynamically optimize the parameters of the hash function.

[0065] By respectively sampling and checking the deduplicated data set and the duplicate data set, misjudgments that may exist in previous steps can be found and corrected in a timely manner, further improving the accuracy and reliability of the data. Based on the results of the sampling and checking, the parameters of the hash function are dynamically optimized to better adapt to the characteristics and changes of the data, improve the performance and accuracy of the classification. At the same time, continuously optimizing the parameters of the hash function helps to improve the adaptability of the entire data processing system to different types and characteristics of text data sets, enhances the stability and universality of the system, and makes it better able to meet the data processing needs in various practical application scenarios.

[0066] In some embodiments, the sampling and checking step includes: S401. Randomly sample a verification sample set from the text data set according to a predetermined proportion to form a verification data set, and use the verification data set to perform the operations of steps S2-S3 to obtain a duplicate verification data set and a deduplicated verification data set; S402. Manually check the duplicate verification data set and the deduplicated verification data set, and screen false positive data and false negative data, wherein the false positive data are data in the duplicate verification data set that are actually not duplicated with other data, and the false negative data are data in the deduplicated verification data set that are duplicated with other data; S403. Store the false positive data from the duplicate verification data set to the deduplicated verification data set; for each group of false negative data, only keep the unique representative data in the deduplicated verification data set, and store the rest of the false negative data to the duplicate verification data set; S404. Based on the duplicate verification data set and the deduplicated verification data set obtained in step S403, use an iterative optimization algorithm to optimize the parameters of the hash function, and apply the optimized hash function to the processing of subsequent incremental data through a parameter update interface.

[0067] Randomly extracting a verification sample set from the text data set in a preset proportion to form a verification data set and performing related operations can check the data cleaning result without processing the full amount of data, reduce the workload of manual checking, and improve the verification efficiency; manually checking the verification data set, screening false positive data and false negative data, and adjusting them can correct the misjudgment in the data cleaning process and improve the accuracy of data cleaning; using the adjusted re-verification data set and the deduplicated verification data set to train a high-similarity repetitive data identification model, optimizing the model parameters, so that the model can learn the real features and repetitive patterns of the data, improve the identification ability of the model in subsequent high-similarity repetitive data cleaning tasks, realize incremental learning, and continuously improve the data cleaning effect.

[0068] In one specific embodiment, the high-similarity repetitive data cleaning method comprises: Step S1, reading the text data set to be processed, the format of the text data set being one of JSON, PARQUET, and CSV; Step S2, performing hash processing and cluster division on the text data set, comprising: extracting an n-gram sequence for each text data in the text data set, and using UTF-8 encoding to encode the text content into a byte stream; When extracting the n-gram sequence: n is 3; For text items with a length less than n, fill in special characters to a length of n for processing; The extracted n-gram sequence needs to be filtered by a stop word list, which contains punctuation marks and common words, including function words, pronouns, high-frequency low-information words, and domain general words, wherein the function words include auxiliary words, prepositions, and conjunctions; Apply k different hash functions to the byte stream to generate k hash values, k≥3, and take the minimum value of all hash values as the minimum hash value of the data item, and aggregate all data items to form a new data set with the minimum hash value; The k different hash functions are generated by the following method: Select a reference hash function The reference hash function is MurmurHash3, and its variant function is generated by prime modulus operation The formula is:

[0069] In the formula, is the original data item is the initial hash value generated by the reference hash function is a large prime number of 1024 bits or more; representative modular operation, i.e. calculating the remainder; is a random integer, ; is a random integer, ; a directed graph is constructed based on the minimum hash values of the data items, a parallel connected component algorithm is run using the GraphFrames graph processing library of Apache Spark, the hash calculation task of each data item is distributed to the cluster nodes for parallel execution, the data items with the same minimum hash value are grouped into the same connected subgraph to form a similar duplicate data cluster, and isolated vertices are grouped into a non-duplicate data cluster; Step S3, performing a duplicate checking operation and a deduplication operation, wherein: the duplicate checking operation is to classify and save the data in each duplicate data cluster to generate an independently stored duplicate data set; the deduplication operation is to retain all data of the non-duplicate data cluster and unique representative data in each duplicate data cluster to form a deduplicated data set; In each duplicate data cluster, the selection rule of the unique representative data is to select the data with the smallest minimum hash value and the smallest Hamming distance in the duplicate data cluster; During the process, a big data storage and computing base is used for data storage and calculation Step S4, sampling and checking the duplicate data set and the deduplicated data set respectively, correcting misjudgment data and dynamically optimizing the parameters of the hash function, the steps comprising: S401. A verification sample set is randomly extracted from the text data set according to a preset proportion to form a verification data set, the verification data set is used to perform the operations of steps S2-S3 to obtain a duplicate verification data set and a deduplicated verification data set; S402. The duplicate verification data set and the deduplicated verification data set are manually checked to screen false positive data and false negative data, wherein the false positive data is data in the duplicate verification data set that is actually not duplicated with other data, and the false negative data is data in the deduplicated verification data set that is duplicated with other data; S403. The false positive data is transferred from the duplicate verification data set to the deduplicated verification data set; for each group of false negative data, only the unique representative data is retained in the deduplicated verification data set, and the remaining false negative data is transferred to the duplicate verification data set; S404. Based on the duplicate verification data set and the deduplicated verification data set obtained in step S403, the parameters of the hash function are optimized using an iterative optimization algorithm and , and the optimized hash function is applied to the processing of subsequent incremental data through a parameter update interface; The iterative optimization algorithm is a supervised learning algorithm based on binary cross-entropy loss, and the loss function is calculated by a back propagation algorithm The gradient of the hash function parameters and is calculated, and the hash function parameters and are updated by using a gradient descent method to minimize the misjudgment probability, and the loss function is:

[0070] In the formula, is the total number of samples in the training set; is a data pair; is a binary label manually labeled, when the data pair is actually repeated, when the data pair is not actually repeated, ; is the data pair repetition probability calculated by the hash function, which is specifically defined as:

[0071] is a sigmoid function, ; ) is an indicator function, which is 1 when , otherwise 0.

[0072] The following is an embodiment of the high similarity repetitive data cleaning system provided by the embodiment of the present disclosure. The active load shedding optimization system and the high similarity repetitive data cleaning method of each embodiment described above belong to the same inventive concept. Details not described in the embodiment of the high similarity repetitive data cleaning system can be referred to the embodiment of the high similarity repetitive data cleaning method.

[0073] A mobile terminal implementing various embodiments of the present application will now be described with reference to the accompanying drawings. In the following description, the suffix "module" or "part" used for an element is merely intended for facilitating description of the present application, and does not have specific meaning or function. Therefore, "module" and "part" can be used interchangeably.

[0074] As shown in Figure 2 , the high similarity repetitive data cleaning system includes: a text data set reading module for reading a text data set to be processed; The hashing processing and cluster division module is configured to perform hashing processing and cluster division on the text data set to obtain a similar repeated data cluster and a non-repeated data cluster. The duplicate checking and removal module is configured to perform duplicate checking and removal operations to obtain a repeated data set and a data set after removal of duplicates. The sampling verification module is configured to perform sampling verification on the repeated data set and the data set after removal of duplicates, correct misjudgment data, and dynamically optimize parameters of the hash function.

[0075] The high-similarity repeated data cleaning system of the embodiment is configured to implement a high-similarity repeated data cleaning method, and the steps include: S1. reading a text data set to be processed; S2. performing hashing processing and cluster division on the text data set, including: extracting an n-gram sequence from each piece of text data in the text data set to encode the text content into a byte stream; applying k different hash functions to the byte stream to generate k hash values, k≥3, taking the minimum value of all hash values as the minimum hash value of the data item, and collecting all data items to form a new data set with minimum hash values; constructing an undirected graph based on the minimum hash values of the data items, running a parallel connected component algorithm, grouping data items with the same minimum hash value into the same connected subgraph, forming a similar repeated data cluster, and grouping isolated vertices into a non-repeated data cluster; S3. performing duplicate checking and removal operations, wherein: the duplicate checking operation is to classify and save data in each repeated data cluster to generate an independently stored repeated data set; the removal operation is to retain all data of the non-repeated data cluster and unique representative data in each repeated data cluster to form a data set after removal of duplicates; S4. performing sampling verification on the repeated data set and the data set after removal of duplicates, correcting misjudgment data, and dynamically optimizing parameters of the hash function.

[0076] The application also provides an electronic device for implementing various embodiments of the application, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0077] Those skilled in the art can understand that the electronic device structure involved in the embodiments of the application does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the illustration, or combine certain components, or different component arrangements.

[0078] Figure 3 A hardware structure schematic diagram of an electronic device for implementing various embodiments of the application.

[0079] The electronic device includes, but is not limited to, a processor, a memory, and the like. Those skilled in the art can understand that the electronic device structure involved in the embodiments of the present application does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0080] In the embodiments of the present application, the electronic device includes, but is not limited to, a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present application as described and / or claimed herein.

[0081] In the embodiments of the present application, the processor can be implemented by using at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a processor, a controller, a microcontroller, a microprocessor, an electronic unit designed to perform the functions described herein, and in some cases, such an implementation can be implemented in a controller. For software implementation, such as process or function implementation, can be implemented with a separate software module allowing at least one function or operation to be performed. The software code can be implemented by a software application (or program) written in any appropriate programming language, which can be stored in a memory and executed by a controller.

[0082] In addition, the electronic device includes some functional modules that are not shown and will not be described here.

[0083] Those skilled in the art can understand that various aspects of the present application provide an electronic device can be implemented as a system, a method or a program product. Therefore, various aspects of the present disclosure can be embodied as a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0084] The present application also provides a storage medium storing a program product capable of implementing a method for cleaning duplicate data with high similarity. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code that, when executed on a terminal device, causes the terminal device to execute the steps described in the "Exemplary Methods" section above according to various exemplary implementations of the present disclosure.

[0085] The storage medium can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0086] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for cleaning duplicate data with high similarity, characterized in that the steps include: S1. Read the text dataset to be processed; S2. Hashing and clustering the text dataset, including: Extract n-gram sequences from each text data item in the text dataset and encode the text content into a byte stream; Apply k different hash functions to the byte stream and perform hash transformation to generate k hash values, k ≥ 3. Take the minimum value of all hash values ​​as the minimum hash value of the data item, and aggregate all data items to form a new data set with the minimum hash value. An undirected graph is constructed based on the minimum hash value of the data items. The parallel connected component algorithm is run to group the data items with the same minimum hash value into the same connected subgraph to form clusters of similar duplicate data and isolated vertices into clusters of non-duplicate data. S3. Perform duplicate checking and deduplication operations, including: The duplicate checking operation is to classify and save the data in each duplicate data cluster to generate an independently stored duplicate data set; The deduplication operation is to retain all the data in the non-duplicate data cluster and the unique representative data in each duplicate data cluster to form the deduplicated data set; S4. Perform sampling verification on duplicate datasets and deduplicated datasets, correct misjudged data, and dynamically optimize the parameters of the hash function.

2. The method for cleaning duplicate data with high similarity according to claim 1, wherein: In step S1, the format of the text dataset includes at least one of JSON, PARQUET, and CSV.

3. The method for cleaning duplicate data with high similarity according to claim 1, wherein: In step S2, when extracting n-gram sequences: The value range of n is 3-5; For text items with a length less than n, special characters are added to the length n before processing; The extracted n-gram sequence needs to be filtered out of the stop word list, which includes punctuation marks and common words.

4. The method for cleaning duplicate data with high similarity according to claim 1, wherein: k different hash functions are generated by: selecting a benchmark hash function , generate its variant function through prime number modular operation , , the formula is: Where, For the original data item The initial hash value generated by the benchmark hash function It is a large prime number with more than 1024 digits; represents modular operation; is a random integer, ; is a random integer, .

5. The method for cleaning duplicate data with high similarity according to claim 1, wherein: In step S2, when constructing an undirected graph based on the minimum hash value of the data items, the GraphFrames graph processing library of Apache Spark is used to run the parallel connected component algorithm, and the hash calculation task of each data item is distributed to the cluster nodes for parallel execution.

6. The method for cleaning duplicate data with high similarity according to claim 1, wherein: Step S3 uses the big data storage and computing base for data storage and computing.

7. The method for cleaning duplicate data with high similarity according to claim 1, wherein: The sampling verification step of step S4 includes: S401 randomly extracts verification samples from the text dataset according to a preset ratio to form a verification dataset, and performs steps S2-S3 using the verification dataset to obtain a duplicate verification dataset after deduplication; S402. Manually check the duplicate verification data set and the deduplication verification data set to screen out false positive data and false negative data, where false positive data is data that is not actually duplicated with other data in the duplicate verification data set, and false negative data is data that is duplicated with other data in the deduplication verification data set; S403. The false positive data is transferred from the duplicate verification data set to the deduplication verification data set; for each set of false negative data, only the only representative data is retained in the deduplication verification data set, and the remaining false negative data is transferred to the duplicate verification data set; S404. Based on the duplicate check dataset and the deduplication check dataset obtained in step S403, the parameters of the hash function are optimized using an iterative optimization algorithm, and the optimized hash function is applied to subsequent incremental data processing through a parameter update interface.

8. A high-similarity duplicate data cleaning system, characterized in that: A method for cleaning duplicate data with high similarity according to any one of claims 1 to 7, comprising: A text dataset reading module is used to read the text dataset to be processed; The hashing and clustering module is used to hash and cluster the text dataset to obtain clusters of similar duplicate data and clusters of non-duplicate data; The duplicate checking and deduplication module performs duplicate checking and deduplication operations to obtain duplicate data sets and deduplication data sets; The sampling verification module performs sampling verification on duplicate data sets and deduplicated data sets respectively, corrects misjudged data and dynamically optimizes the parameters of the hash function.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the steps of the high-similarity duplicate data cleaning method as described in any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the high-similarity duplicate data cleaning method according to any one of claims 1 to 7 are implemented.

Images