Spark-based large-scale data global deduplication method, electronic device and medium

Abstract

The application provides a large-scale data deduplication method based on Spark, an electronic device and a storage medium. The large-scale corpus data is preprocessed, the first processed document obtained after preprocessing is stored in different storage partitions, grouping is performed on the first processed document in each storage partition, a large number of completely irrelevant documents are excluded, similarity detection is performed to obtain a similar pair of each first processed document, and the similar pairs are merged in three granularities of document grouping, storage partition and global. The similar pairs are efficiently merged in the granularity of document grouping and storage partition through a distributed parallel running method, the calculation amount of merging in the global granularity of the system is greatly reduced, and high-efficiency fuzzy deduplication of large-scale data is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a large-scale data global deduplication method based on Spark, an electronic device and a storage medium BACKGROUND

[0002] When training a large-scale model, a large amount of corpus data is needed as a training sample. These training samples are often obtained by crawling various data sources. On the Internet, there are often a large number of repeated corpus data in different data sources or different partitions of the same data source, which are similar in actual content and only differ in expression. Since these corpus data are not completely repeated, it is difficult to directly determine whether they express the same content through traditional deduplication methods. When training a model, if this part of corpus data cannot be removed, it is easy to cause the model to over-learn these repeated corpus data, resulting in poor training effect of the model. For a large model, the scale of corpus data required for training is very large. It is often very inefficient to deduplicate large-scale corpus data through traditional single-machine methods. Therefore, how to efficiently deduplicate large-scale similar corpus data is a problem that needs to be solved at present. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a large-scale data global deduplication method based on Spark, an electronic device and a computer readable storage medium. The corpus data is merged and deduplicated at three different granularities of grouping, partitioning and global, and efficient fuzzy deduplication of large-scale corpus data is realized.

[0004] The first aspect of the embodiments of the present application provides a large-scale data global deduplication method based on Spark, comprising:

[0005] Preprocessing the large-scale data to obtain a plurality of first processed documents, and storing the first processed documents in a plurality of storage partitions, respectively;

[0006] Grouping the first processed documents in each storage partition to obtain a plurality of document groups;

[0007] Detecting the similarity of the first processed documents in each document group to obtain a plurality of similar pairs;

[0008] Determining the minimum merging similar pair of each similar pair according to the preconfigured global number of the first processed document, and merging all similar pairs pointing to the same minimum merging similar pair to obtain a grouped similar set;

[0009] determining a minimum merge group set of each of the group similarity sets according to the global number of the first processing document, and merging all group similarity sets pointing to the same minimum merge group set to obtain a partition similarity set;

[0010] determining a minimum merge partition set of each of the partition similarity sets according to the global number of the first processing document, and merging all partition similarity sets pointing to the same minimum merge partition set to obtain a global similarity set, and recording the global number of all the first processing documents in each of the global similarity sets through a second processing document;

[0011] removing the first global number in each of the global similarity sets from the second processing document, and collecting all the remaining global numbers in all the global similarity sets to obtain a to-be-eliminated document number set, and broadcasting the to-be-eliminated document number set to each of the storage partitions;

[0012] performing a filtering operation on the first processing document according to the to-be-eliminated document number set in each of the storage partitions.

[0013] In some embodiments, the determining a minimum merge similarity pair of each of the similarity pairs according to the global number of the first processing document, and merging all similarity pairs pointing to the same minimum merge similarity pair to obtain a group similarity set comprises:

[0014] numbering the similarity pairs, and constructing a first hash table according to the global number of the first processing document and the number of the similarity pairs, wherein the key value of the first hash table is the global number of the first processing document, and the true value of the first hash table is an ordered linked list comprising the number of all similarity pairs of the first processing document corresponding to the key value;

[0015] constructing a first dynamic array and assigning the first dynamic array according to the first hash table, determining a minimum similarity pair of each of the similarity pairs according to the assignment result, and merging multiple similarity pairs pointing to the same minimum similarity pair in the same document group to obtain a group similarity set.

[0016] In some embodiments, the determining a minimum merge group set of each of the group similarity sets according to the global number of the first processing document, and merging all group similarity sets pointing to the same minimum merge group set to obtain a partition similarity set comprises:

[0017] numbering the group similarity sets, and constructing a second hash table according to the global numbers and the numbering of the group similarity sets, wherein the key value of the second hash table is the global number of the first processed document, and the true value of the second hash table is an ordered linked list composed of the numbers of all the group similarity sets corresponding to the key value;

[0018] constructing a second dynamic array and assigning values to the second dynamic array according to the second hash table, determining the minimum similarity set to which each of the group similarity sets is merged according to the assigned values, and merging multiple group similarity sets pointing to the same minimum similarity set in the same storage partition to obtain a partition similarity set.

[0019] In some embodiments, the minimum merging partition set to which each of the partition similarity sets is merged is determined according to the global number of the first processed document, and all similarity pairs pointing to the same minimum merging partition set are merged to obtain a global similarity set, and the global number of each of the first processed documents included in each of the global similarity sets is recorded by a second processed document, including:

[0020] numbering the partition similarity sets, and constructing a third hash table according to the global numbers and the numbering of the partition similarity sets, wherein the key value of the third hash table is the global number of the first processed document, and the true value of the third hash table is an ordered linked list composed of the numbers of all the partition similarity sets corresponding to the key value;

[0021] constructing a third dynamic array and assigning values to the third dynamic array according to the third hash table, determining the minimum global set to which each of the partition similarity sets is merged according to the assigned values, merging multiple partition similarity sets pointing to the same minimum global set to obtain a global similarity set, and recording the global number of all the first processed documents in each of the global similarity sets by a second processed document.

[0022] In some embodiments, the first dynamic array is constructed and assigned values according to the first hash table, and the minimum similarity pair to which each of the similarity pairs is merged is determined according to the assigned values, and multiple similarity pairs pointing to the same minimum similarity pair in the same document group are merged to obtain a group similarity set, including:

[0023] constructing and initializing the first dynamic array;

[0024] scanning the first hash table to assign values to the first dynamic array, and the index value of each item of the first dynamic array after the assignment is the number of the similarity pair, and the initial value stored by each item is the number of the similarity pair located in the next item of the index value in the true value of the first hash table;

[0025] For the i-th item of the first dynamic array, a first storage value of the i-th item is obtained, and in a case where the first storage value is a first preset value, the first storage value is set as i, where i is a positive integer;

[0026] In a case where the first storage value is not the first preset value, a second storage value of an item in the first dynamic array with the first storage value as an index value is obtained, and in a case where the second storage value is not the first preset value, the second storage value is taken as a new first storage value, and a new second storage value is obtained with the new first storage value as an index value, until the second storage value is the first preset value;

[0027] In a case where the second storage value is the first preset value, it is determined that the first storage value is the number of the smallest similar pair in the merging of similar pairs with the number i;

[0028] According to the index values of all items with the same storage value in the first dynamic array, a plurality of similar pairs corresponding to each of the document groups are merged to obtain the grouped similar set.

[0029] In some embodiments, the construction of the second dynamic array and the assignment of values to the second dynamic array according to the second hash table, the determination of the smallest similar set in the merging of each of the grouped similar sets according to the assignment result, and the merging of a plurality of grouped similar sets pointing to the same smallest similar set in the same storage partition to obtain a partition similar set, comprise:

[0030] The second dynamic array is constructed and initialized;

[0031] The second hash table is scanned to assign values to the second dynamic array, and the index value of each item of the second dynamic array after the assignment is the number of the grouped similar set, and the initial value stored by each item is the number of the grouped similar set of the next item of the hash table located after the index value;

[0032] For the j-th item of the second dynamic array, a third storage value of the j-th item is obtained, and in a case where the third storage value is a second preset value, the third storage value is set as j, where j is a positive integer;

[0033] In a case where the third storage value is not the second preset value, a fourth storage value of an item in the second dynamic array with the third storage value as an index value is obtained, and in a case where the fourth storage value is not the second preset value, the fourth storage value is taken as a new third storage value, and a new fourth storage value is obtained with the new third storage value as an index value, until the fourth storage value is the second preset value;

[0034] In the case that the fourth stored value is the second preset value, it is determined that the third stored value is the number of the smallest similarity set of the merging of the group similarity set numbered i;

[0035] The group similarity sets corresponding to the index values of all items with the same stored value in the second dynamic array are merged to obtain the partition similarity set.

[0036] In some embodiments, the third dynamic array is constructed and the third dynamic array is assigned values according to the third hash table, and the smallest global set of the merging of each partition similarity set is determined according to the assignment result, the global similarity sets are obtained by merging the partition similarity sets pointing to the same smallest global set, and the global numbers of all the first processing documents included in each global similarity set are recorded by the second processing document, including:

[0037] The third dynamic array is constructed and initialized;

[0038] The third hash table is scanned to assign values to the third dynamic array, and the index value of each item of the third dynamic array after assignment is the number of the partition similarity set, and the initial value stored by each item is the number of the partition similarity set of the next item of the index value in the true value of the third hash table;

[0039] For the kth item of the third dynamic array, the fifth stored value of the kth item is obtained, and in the case that the fifth stored value is a third preset value, the fifth stored value is k, wherein k is a positive integer;

[0040] In the case that the fifth stored value is not the third preset value, the sixth stored value of the item in the third dynamic array with the fifth stored value as the index value is obtained, and in the case that the sixth stored value is not the third preset value, the sixth stored value is taken as a new fifth stored value, and a new sixth stored value is obtained with the new fifth stored value as the index value, until the sixth stored value is the third preset value;

[0041] In the case that the sixth stored value is the third preset value, it is determined that the fifth stored value is the number of the global similarity set of the merging of the partition similarity set numbered j;

[0042] The partition similarity sets corresponding to the index values of all items with the same stored value in the third dynamic array are merged to obtain the global similarity set;

[0043] The global numbers of the first processing documents included in each global similarity set are recorded by the second processing document.

[0044] In some embodiments, the preprocessing of the large-scale data to obtain a plurality of first processing documents and storing the first processing documents into a plurality of storage partitions respectively comprises:

[0045] extracting a plurality of original input documents from the large-scale data and numbering the original input documents to obtain a global number of each original input document;

[0046] performing word segmentation processing on all the original input documents to convert the original input documents into a word set comprising a plurality of words;

[0047] calculating a hash code of each word;

[0048] generating the first processing document corresponding to each original input document according to the hash code and the global number and storing all the first processing documents into respective storage partitions.

[0049] In some embodiments, the grouping of the first processing documents in the respective storage partitions to obtain a plurality of document groups comprises:

[0050] calculating a word frequency of each word in the first processing document and sorting the words in each first processing document according to the word frequency, and performing prefix pruning processing on the sorted first processing document to obtain a prefix array of each first processing document;

[0051] In each storage partition, the first processing documents comprising at least one same word in the prefix array are grouped into the same group to obtain a plurality of document groups.

[0052] In some embodiments, the similarity detection of the first processing documents in each document group to obtain a plurality of similar pairs comprises:

[0053] calculating a document fingerprint of each first processing document by a preset algorithm;

[0054] determining a Hamming distance between the first processing documents in each document group according to the document fingerprint;

[0055] grouping the first processing documents with a Hamming distance less than a fourth preset value as similar pairs.

[0056] A second aspect of the embodiments of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the large-scale data global deduplication method based on Spark as described in any one of the first aspect embodiments when executing the computer program.

[0057] The third aspect of the embodiment of the present application provides a computer readable storage medium, characterized in that the computer readable storage medium stores one or more programs, and the one or more programs can be run by one or more processors to implement the Spark-based large-scale data global deduplication method as any one of the first aspect embodiments.

[0058] The embodiment of the present application provides a Spark-based large-scale data global deduplication method, an electronic device and a storage medium, by storing large-scale corpus data into different storage partitions, grouping the corpus data in each storage partition to eliminate a large number of irrelevant corpus data, and then detecting the similarity of the first processing document in each document group, determining whether each document in the same group is similar, and then merging the similar pairs in the document group, the storage partition and the system global three different granularities in turn, so that the distributed parallel processing method is adopted in the document group and the storage partition two granularities to preliminarily merge the similar corpus data twice, and then the preliminarily merged corpus data is merged in the system global, so that a large amount of time can be saved when deduplicating large-scale data, and efficient fuzzy deduplication of large-scale corpus is realized.

[0059] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be achieved and obtained by the structure particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of a Spark-based large-scale data global deduplication method provided by the embodiment;

[0061] Figure 2 is Figure 1 the flowchart of step S400 in

[0062] Figure 3 is Figure 2 the flowchart of step S220 in

[0063] Figure 4 is Figure 1 the flowchart of step S500 in

[0064] Figure 5 is Figure 4 the flowchart of step S520 in

[0065] Figure 6 is Figure 4 the flowchart of step S600 in

[0066] Figure 7 is Figure 6 the flowchart of step S620 in the method;

[0067] Figure 8 is a structural schematic diagram of an electronic device provided by the embodiment;

[0068] The accompanying drawings are included to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification is for the purpose of describing the embodiments of the present application only and is not intended to limit the present application.

[0071] In addition, the described features, structures or characteristics can be combined in any suitable way in one or more embodiments. In the following description, many specific details are provided to give a full understanding of the embodiments of the present disclosure. However, one skilled in the art will realize that the technical scheme of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0072] The flowchart shown in the drawings is only an exemplary description, and does not necessarily include all contents and operations / steps, nor does it necessarily run in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so the actual running order may be changed according to the actual situation.

[0073] With reference to Figure 1 , the first aspect of the embodiment of the present application proposes a large-scale data global deduplication method based on Spark, comprising steps S100 to S800:

[0074] Step S100, pre-processing the large-scale data to obtain a plurality of first processed documents, and storing the first processed documents into a plurality of storage partitions respectively;

[0075] In some embodiments, the large-scale data can be a large amount of text data crawled from various web pages and the like data sources, and the text data constitutes a plurality of documents. The preprocessing step of the large-scale data can include extracting a plurality of original input documents from the large-scale data. In some embodiments, the corpus data is large, which can reach tens of TB, and cannot be directly stored in the memory, but needs to be stored through external storage. The corpus data is read from the external storage, and each original input document is recorded by a resilient distributed dataset (RDD). Then, all original input documents are numbered to obtain the global number of each original input document in the system. Specifically, the global number of each original input document can be obtained by performing a ZipWithIndex operation on the original input document, and each numbered original input document is stored in each storage partition by a MemoryAndDisk operation. In some embodiments, the preprocessing step of the large-scale data further includes performing word segmentation processing on each original input document. Specifically, a conversion operation can be performed by a Map operator, and a word segmentation tool is used to segment the original input document into a set of multiple words. In some embodiments, a word that appears repeatedly in the same original input document is recorded only once, so as to avoid data redundancy caused by repeatedly recording the same word in the same document, thereby improving the deduplication efficiency. In some embodiments, the preprocessing step of the large-scale data further includes calculating the hash code of each word. Specifically, each word can be converted from a string type to a 64-bit hash value by a Murmur hash algorithm, thereby obtaining the hash code of each word.

[0076] Step S200, grouping the first processing document in each storage partition to obtain a plurality of document groups.

[0077] In some embodiments, step S200 comprises: calculating the word frequency of each word in the first processed document, and sorting the words in each first processed document according to the word frequency, performing prefix pruning processing on the sorted first processed document to obtain a prefix array of each first processed document; and in each storage partition, grouping the first processed documents including at least one same word in the prefix array into the same group to obtain a plurality of document groups. In the present embodiment, all first processed documents in the same storage partition are first expanded by a FlatMap operator, and the word frequency of each word is initialized to 1. At this time, the hash code of each word and the corresponding word frequency can be recorded through a first word table, and the words with the same hash code are then merged. The word frequency of each word is calculated according to the number of merging, thereby obtaining the word frequency in each storage partition. The first word tables of the various storage partitions are then aggregated to form a second word table. The word frequencies corresponding to the words with the same hash code in the second word table are added, and thus the global word frequency of each word is obtained. Specifically, the ReduceByKey operation can be performed on the first word table to merge the words with the same hash code and accumulate the word frequencies. The CollectAsMap operation can be used to aggregate the first word table to form the second word table. It can be understood that since a repeated word is recorded only once in each first processed document, the word frequency can represent the number of first processed documents in which a certain word appears simultaneously. In some embodiments, the second word table can also be broadcast to each storage partition to facilitate subsequent sorting of the words in the first processed document in each storage partition.

[0078] In some embodiments, step S200 further comprises sorting the words in the first processed document according to the word frequency. Specifically, the words in the document are sorted in ascending order of word frequency. The prefix length of each first processed document is then calculated by a preset prefix length formula. The prefix array is cut from the first processed document after sorting processing. The first processed documents including at least one word with the same hash code in the prefix array are grouped into the same group. Specifically, each word in the prefix array can be traversed and expanded by a FlatMap operation in the form of (hash code, fingerprint) to form a third processed document. The GroupByKey operation is then performed on the third processed document to realize document grouping. Thus, a large number of irrelevant documents are excluded, effectively reducing the computational load in subsequent similarity detection. Specifically, the prefix length can be calculated by a preset prefix length formula as follows:

[0079]

[0080] wherein x represents the global number, j represents the prefix length, σ xrepresents the sum of the IDF weights of all words in the third processed document, i is an integer ranging between 1 and j, x[i] represents the ith word in the third processed document, weight(x[i]) represents the IDF weight of the ith word in the third processed document, and t represents a preset similarity threshold; wherein the IDF weight is determined according to the word frequency of each word and the first processed document, and specifically, refer to the following formula:

[0081]

[0082] wherein idf i represents the IDF weight of the ith word, |D| represents the total amount of the first processed document, |{j: t∈dj} represents the word frequency of the ith word. j

[0083] It can be understood that if the same word appears in a large number of documents, it means that the word is a more common word, such as the common words "of" and "first", which are actually less relevant to the content of the document and are difficult to reflect the characteristics of the document content. When the word frequency of a certain word is low and only appears in a small number of documents, it means that the word is highly related to the content of a certain field, such as the words "concrete" and "concrete", which generally only appear in documents in the construction field. Such words are highly relevant to the content of the document and can better reflect the characteristics of the document content. Based on this, in the embodiment, the words in the first processed document are sorted in reverse order and then pruned, thereby discarding a large number of common words and retaining only professional words that are highly relevant to the content of the document. The first processed documents that still include the same hash code after prefix pruning are grouped into the same document group. By grouping the first processed documents, a large number of irrelevant documents are filtered, thereby effectively reducing the computational complexity of subsequent similarity detection.

[0084] Step S300, performing similarity detection on the first processed documents in each document group to obtain a plurality of similar pairs;

[0085] ​In some embodiments, the document fingerprint of each first processing document can be calculated by a preset algorithm, and in the present embodiment, the document fingerprint of each first processing document is calculated by a simhash algorithm. Then, similarity checking is performed on each first processing document within each document group, specifically, the Hamming distance between each first processing document is calculated, and the first processing documents with a Hamming distance less than a fourth preset value are classified as a similar pair. Specifically, the Map operation can be performed on the third processing document after grouping processing, and the similarity verification is performed in each group according to the fingerprint using the Hamming distance, and in the present embodiment, the fourth preset value can be 3. Thus, the similarity detection of the first processing document is realized, and at this time, the fourth processing document can be used to record the similar pairs in each group and the numbers of the first processing documents included in each similar pair.

[0086] It can be understood that in some embodiments, the step of calculating the document fingerprint of each first processing document by a preset algorithm can be performed in advance, for example, the document fingerprint of each first processing document is calculated after the word segmentation processing of the first processing document and the calculation of the word frequency of each word, and the document fingerprint of each first processing document is also recorded in the first processing document. Or the global number and the fingerprint of each first processing document are recorded in the fifth processing document, so that when the Hamming checking and the division of similar pairs are performed subsequently, the global number and the document fingerprint of each first processing document can be directly read from the fifth processing document, without the need to read and write the corresponding first processing document multiple times for subsequent steps, thereby saving the communication time and reducing the time waste caused by reading and writing memory.

[0087] In step S400, the minimum merging similar pair of each similar pair is determined according to the preconfigured global number of the first processing document, and all similar pairs pointing to the same minimum merging similar pair are merged to obtain a grouped similar set.

[0088] In the present embodiment, all similar pairs including the same first processing document within each document group are regarded as similar pairs that can be merged with each other, and are merged, specifically, referring to Figure 2 Step S400 includes but is not limited to the following steps S210 to S220:

[0089] In step S210, the similar pairs are numbered, and a first hash table is constructed according to the global number of the first processing document and the number of the similar pairs, wherein the key value of the first hash table is the global number of the first processing document, and the true value of the first hash table is an ordered linked list composed of the numbers of all similar pairs of the first processing document corresponding to the key value;

[0090] It can be understood that the similar pairs in the same document group are numbered from 1, and the global number of the first processed document is taken as the key value of the first hash table, and the numbers of all similar pairs including the first processed document form an ordered linked list, and the ordered linked list is taken as the true value corresponding to the key value. For example, the similar pairs numbered 1, 3, 7, and 10 all include the first processed document with the global number 1, and the true value stored in the item with the key value 1 in the first hash table is the ordered linked list formed by 10, 7, 3, and 1.

[0091] In step S220, the first dynamic array is constructed and the first dynamic array is valued according to the first hash table, and the minimum similar pair to which each similar pair can be merged is determined according to the valuation result, and the multiple similar pairs in the same document group pointing to the same minimum similar pair are merged to obtain the grouped similar set.

[0092] The minimum similar pair to which each similar pair can be merged is determined according to the first dynamic array after valuation, such as in the above embodiment, the similar pairs numbered 10, 7, and 3 can all be merged into the similar pair numbered 1, thereby obtaining the grouped similar set.

[0093] Reference Figure 3 In some embodiments, step S220 can include but is not limited to steps S310 to S360:

[0094] In step S310, the first dynamic array is constructed and initialized.

[0095] It can be understood that the initial stored value of each item in the first dynamic array is 0

[0096] In step S320, the first hash table is scanned to value the first dynamic array, and the index value of each item of the first dynamic array after valuation is the number of the similar pair, and the initial value stored in each item is the number of the similar pair located in the next item of the index value in the true value of the first hash table.

[0097] In this embodiment, the valuation of the first dynamic array is realized by scanning the true value of the hash table. Specifically, the true value of the hash table is an ordered linked list formed by the numbers of multiple similar pairs including the same first processed document, and the two values successively adjacent in the ordered linked list are taken as the index value and the stored value of the first dynamic array to value the first dynamic array.

[0098] In step S330, for the i-th item of the first dynamic array, the first stored value of the i-th item is obtained, and in the case that the first stored value is the first preset value, the first stored value is i, where i is a positive integer.

[0099] In the embodiment, each initial stored value of the first dynamic array is 0 before the first dynamic array is assigned by scanning the first hash table, based on which the first preset value is 0. For the i-th item of the first dynamic array, the stored value B[i] is obtained. If B[i] is 0, it indicates that i is the rightmost node of the ordered linked list in the value of the hash table, and i is the smallest number in the similar pair with the similar pair of the number i containing all similar pairs of the same first processing document. Therefore, the similar pair with the number i can be regarded as the smallest similar pair which can be merged, and B[i] is set to i.

[0100] In step S340, if the first stored value is not the first preset value, the second stored value of the item in the first dynamic array with the first stored value as the index value is obtained. If the second stored value is not the first preset value, the second stored value is taken as a new first stored value, and a new second stored value is obtained with the new first stored value as the index value, until the second stored value is the first preset value.

[0101] In step S350, if the second stored value is the first preset value, it is determined that the first stored value is the number of the smallest similar pair of the similar pair with the number i.

[0102] In some embodiments, if B[i] is not 0, it indicates that the right node of i in the corresponding ordered linked list stores a smaller value than i, and i is not the number of the smallest similar pair which can be merged. At this time, the stored value of B[i] is taken as a new index value, B[B[i]] is found, and B[i] is set to B[B[i]], until B[B[i]] = 0, which indicates that the stored value of B[i] is the number of the smallest similar pair which can be merged.

[0103] In step S360, according to the index values of all items with the same stored value in the first dynamic array, the corresponding multiple similar pairs in each document group are merged to obtain a grouped similar set.

[0104] After the above steps, the minimum similar pair to which each similar pair can be merged is found, and then the first dynamic array is traversed, and all similar pairs with the same value, i.e., which can be merged into the same minimum similar pair, are merged, so that multiple grouped similar sets are obtained. It can be understood that, since the minimum similar pair to which each similar pair can be merged in the document group is unique, there is no intersection between the multiple grouped similar sets obtained in the same document group. Based on this, in the implementation of merging the similar pairs in each document group, the numbers of all first processed documents in each grouped similar set can be recorded by the sixth processing document, so as to facilitate the subsequent merging of the grouped similar sets at the granularity of each storage partition. In addition, steps S310 to S360 can be distributed and run in parallel among the various document groups. The total time required and the maximum amount of data required for the document group that takes the longest time to execute the above steps are the same. Since only the documents including the same word in the prefix array are grouped into the same document group when grouping the documents, the amount of data in each document group is greatly reduced compared to the total amount of documents stored in the storage partition. Therefore, the time required to perform the above merging steps is greatly reduced, and the merging efficiency is significantly improved.

[0105] In step S500, the minimum merging group set of each grouped similar set is determined according to the global number of the first processed document, and all grouped similar sets pointing to the same minimum merging group set are merged to obtain a partition similar set.

[0106] In this embodiment, after merging the similar pairs in each document group, the grouped similar sets in different document groups need to be further merged. It can be understood that, since the first processed documents including the same word are grouped into the same document group when grouping the first processed documents in each storage partition, and since there are multiple words between each first processed document, the same first processed document will also be grouped into multiple document groups. Based on this, in this embodiment, multiple grouped similar sets including the same first processed document can also be merged at the level of the storage partition. Specifically, in this embodiment, step S500 includes but is not limited to steps S410 to S420:

[0107] In step S410, the grouped similar sets are numbered, and a second hash table is constructed according to the global number and the number of the grouped similar set, wherein the key value of the second hash table is the global number of the first processed document, and the true value of the second hash table is an ordered linked list composed of the numbers of all grouped similar sets including the first processed document corresponding to the key value.

[0108] In some embodiments, all the documents in a storage partition can be grouped together for operation by performing a MapPartition operation on the sixth processing document, and the grouped similar sets in the same storage partition are numbered from 1, and the global number of the first processing document is taken as the key value of the second hash table, and the numbers of all the grouped similar sets including the first processing document form an ordered linked list, and the ordered linked list is taken as the true value corresponding to the key value. For example, the grouped similar sets numbered 1, 3, 7 and 10 all include the first processing document with the global number 1, and the true value stored in the item with the key value 1 in the second hash table is the ordered linked list formed by 10, 7, 3 and 1.

[0109] In step S420, a second dynamic array is constructed and the second dynamic array is valued according to the second hash table, and the minimum similar set to which each grouped similar set is merged is determined according to the valuation result, and the grouped similar sets in the same storage partition pointing to the same minimum similar set are merged to obtain a partition similar set.

[0110] Reference Figure 5 In some embodiments, step S420 includes but is not limited to steps S510 to S560

[0111] In step S510, a second dynamic array is constructed and initialized.

[0112] It can be understood that the initial storage value of each item in the second dynamic array is 0

[0113] In step S520, the second hash table is scanned to value the second dynamic array, and the index value of each item in the second dynamic array after valuation is the number of a grouped similar set, and the initial value stored in each item is the number of a grouped similar set located after the index value in the true value of the second hash table.

[0114] In this embodiment, the second dynamic array is valued by scanning the true value of the hash table, and specifically, the true value of the hash table is an ordered linked list formed by the numbers of multiple grouped similar sets including the same first processing document, and the linked list can be an inverted list, and two values successively adjacent in the ordered linked list are taken as the index value and the storage value of the second dynamic array to value the second dynamic array.

[0115] In step S530, the third storage value of the jth item of the second dynamic array is obtained, and in the case that the third storage value is the second preset value, the third storage value is set to j, where j is a positive integer.

[0116] In the embodiment, each initial stored value of the second dynamic array is 0 before the second dynamic array is assigned by scanning the second hash table, and the second preset value is 0 based on this. For the jth item of the second dynamic array, the stored value B[j] is obtained. If B[j] is 0, it indicates that, in the value of the hash table, j is the node at the right end of the ordered list, and the group similar set with the number j contains all group similar sets of the same first processing document, and j is the smallest number. Therefore, the group similar set with the number j can be regarded as the smallest merging group set which can be merged by itself, and B[j] is set to j.

[0117] In step S540, if the third stored value is not the second preset value, the fourth stored value of the item in the second dynamic array with the third stored value as the index value is obtained. If the fourth stored value is not the second preset value, the fourth stored value is taken as a new third stored value, and a new fourth stored value is obtained with the new third stored value as the index value, until the fourth stored value is the second preset value.

[0118] In step S550, if the fourth stored value is the second preset value, it is determined that the third stored value is the number of the smallest merging group set of the group similar set with the number j.

[0119] In some embodiments, if B[j] is not 0, it indicates that, in the corresponding ordered list, the right node of j stores a smaller value than j, and j is not the number of the smallest merging group set of the group similar set which can be merged. At this time, the value stored in B[j] is taken as a new index value, B[B[j]] is searched, and B[j] is set to B[B[j]], until B[B[j]] is 0. It indicates that the stored value of B[j] is the number of the smallest merging group set of the group similar set j which can be merged.

[0120] In step S560, according to the index values of all items with the same stored value in the second dynamic array, the corresponding multiple group similar sets are merged in each storage partition to obtain a partition similar set.

[0121] After the above steps, the minimum merging group set to which each group similar set can be merged is found, and the second dynamic array is traversed to merge all group similar sets with the same value, i.e., can be merged into the same minimum merging group set, so that multiple partition similar sets can be obtained. It can be understood that, since the minimum merging group set to which each group similar set can be merged in the storage partition is unique, there is no intersection between the multiple partition similar sets obtained in the same storage partition. Based on this, in the implementation of merging the group similar sets in each storage partition, the numbers of all first processing documents in each partition similar set can be recorded by the seventh processing document, so as to facilitate the subsequent merging of the partition similar sets at the granularity of the system as a whole; in addition, the steps S510 to S560 can be distributed and run in parallel among the storage partitions, and the total time and the maximum amount of data required for the storage partition that spends the same time in executing the above steps. Since the similar pairs have been preliminarily merged at the document grouping granularity, the amount of data to be processed at the storage partition granularity is greatly reduced relative to the total amount of first processing documents, and thus the time required for executing the above merging steps is also greatly reduced, and the merging efficiency is obviously improved.

[0122] Step S600, determining the minimum merging partition set of each partition similar set according to the global number of the first processing document, and merging all partition similar sets pointing to the same minimum merging partition set to obtain a global similar set, and recording the global number of all first processing documents in each global similar set by a second processing document;

[0123] After the merging of the group similar sets at the storage partition granularity is completed, the partition similar sets can be further merged at the granularity of the system as a whole. Referring to the above embodiment, first, the seventh processing document is subjected to a Collect operation to collect all partition similar sets to the Master node, and the Master node is referred to Figure 6 , and the step S600 includes but is not limited to the following steps S610 to S620.

[0124] Step S610, numbering the partition similar sets, and constructing a third hash table according to the global number and the number of the partition similar set, wherein the key value of the third hash table is the global number of the first processing document, and the true value of the third hash table is an ordered linked list composed of the numbers of all partition similar sets including the first processing document corresponding to the key value;

[0125] The partition similar sets are numbered from 1, and the first global number of the processed document is taken as the key value of the second hash table, and the numbers of all the grouped similar sets including the first processed document are taken to form an ordered linked list, and the ordered linked list is taken as the true value corresponding to the key value. For example, the grouped similar sets numbered 1, 3, 7 and 10 all include the first processed document with the global number 1, and the true value stored by the item with the key value 1 in the third hash table is the ordered linked list formed by 10, 7, 3 and 1.

[0126] In step S620, the third dynamic array is constructed and the third dynamic array is valued according to the third hash table, and the minimum global set to which each partition similar set is merged is determined according to the valuation result, the multiple partition similar sets pointing to the same minimum global set are merged to obtain a global similar set, and the global numbers of all the first processed documents in each global similar set are recorded through the second processed document.

[0127] Reference Figure 7 In some embodiments, step S620 includes but is not limited to steps S710 to S770

[0128] In step S710, the third dynamic array is constructed and initialized.

[0129] It can be understood that the initial storage value of each item in the third dynamic array is 0

[0130] In step S720, the third hash table is scanned to value the third dynamic array, and the index value of each item of the third dynamic array after the valuation is the number of the partition similar set, and the initial value stored by each item is the number of the partition similar set located at the next item of the index value in the true value of the third hash table.

[0131] In this embodiment, the valuation of the third dynamic array is realized by scanning the true value of the hash table. Specifically, the true value of the hash table is an ordered linked list formed by the numbers of multiple partition similar sets including the same first processed document, and the linked list can be an inverted list. Two values successively adjacent in the ordered linked list are taken as the index value and the storage value of the third dynamic array to value the third dynamic array.

[0132] In step S730, the fifth storage value of the kth item of the third dynamic array is obtained, and in the case that the fifth storage value is the third preset value, the fifth storage value is k, where k is a positive integer.

[0133] In the embodiment, each item of the third dynamic array initially stores a value of 0 before the third dynamic array is assigned by scanning the third hash table, and based on this, the third preset value is 0. For the kth item of the third dynamic array, the stored value B[k] is obtained. If B[k] is 0, it indicates that, in the value of the hash table, k is the node at the right end of the ordered list, and the partition similar set with the number k contains all the partition similar sets of the same first processing document in the similar set, and k is the smallest number. Therefore, the partition similar set with the number k can be regarded as the smallest merging partition set which can be merged by itself, and B[k] is set to k.

[0134] In step S740, if the fifth stored value is not the third preset value, the sixth stored value of the item in the third dynamic array with the fifth stored value as the index value is obtained. If the sixth stored value is not the third preset value, the sixth stored value is taken as a new fifth stored value, and a new sixth stored value is obtained with the new fifth stored value as the index value, until the sixth stored value is the third preset value.

[0135] In step S750, if the sixth stored value is the third preset value, it is determined that the fifth stored value is the number of the smallest merging partition set of the partition similar set with the number k.

[0136] In some embodiments, if B[k] is not 0, it indicates that, in the corresponding ordered list, the right node of k stores a value smaller than k, and k is not the number of the smallest merging partition set of the partition similar set which can be merged. At this time, the value stored in B[k] is taken as a new index value, B[B[k]] is found, and B[k] is set to B[B[k]], until B[B[k]] is 0, which indicates that the stored value of B[k] is the number of the smallest merging partition set of the partition similar set k which can be merged.

[0137] In step S760, all the items in the third dynamic array with the same stored value are merged according to the index values to obtain a global similar set.

[0138] After the smallest merging partition set of each partition similar set is found through the above steps, the third dynamic array is traversed, and all the partition similar sets with the same stored value, i.e., which can be merged into the same smallest merging partition set, are merged to obtain a plurality of global similar sets. It can be understood that, since the smallest merging partition set of each partition similar set is unique, there is no intersection between the plurality of global similar sets.

[0139] In step S770, the global number of the first processing document included in each global similar set is recorded by the second processing document.

[0140] After the Master node collects the similar sets of partitions in each storage partition and merges the similar sets of partitions to obtain a plurality of global similar sets, the second processing document records the numbers of all the first processing documents in each global similar set.

[0141] In this embodiment, the global Master node collects the similar sets of partitions in each storage partition and merges the similar sets of partitions. Since the first processing documents have been preliminarily merged at the granularity of document grouping and storage partition in this process, the amount of data collected to the global Master node for processing has been greatly reduced compared to the original number of first processing documents, and the time required for merging processing is also reduced, and the merging efficiency is obviously improved.

[0142] Step S700, removes the first global number in each global similar set from the second processing document, and collects all the remaining global numbers in all the global similar sets to obtain a set of document numbers to be eliminated, and broadcasts the set of document numbers to be eliminated to each storage partition.

[0143] Step S800, in each storage partition, performs a filtering operation on the first processing document according to the set of document numbers to be eliminated.

[0144] It can be understood that the second processing document records the global numbers of all the first processing documents in each global similar set, and the plurality of first processing documents in the same global similar set are similar, and only one needs to be retained, and the remaining first processing documents can be regarded as repeated redundant data, which needs to be de-duplicated. Therefore, in this embodiment, the global number of the first first processing document in each global similar set is removed, and the remaining global numbers are collected to obtain a set of global numbers of documents to be eliminated, and the set of document numbers to be eliminated is broadcast to each storage partition. Then, in each storage partition, a filtering operation is performed on the first processing document according to the set of document numbers to be eliminated, and the first processing document whose global number appears in the set of document numbers to be eliminated is filtered, so as to complete the de-duplication. It can be understood that step S800 is distributed and run in parallel in each storage partition. Compared with unified filtering at the global level, parallel running in the storage partition can effectively reduce the filtering amount of each storage partition and improve the de-duplication efficiency.

[0145] In some embodiments, the first processing document remaining after filtering can also be saved to the file system from each storage partition, so as to save the high-quality corpus data after de-duplication.

[0146] The embodiment of the present application provides a large-scale data deduplication method based on Spark, and realizes a distributed similar pair elimination process from local to global. After a large number of similar pairs are obtained by performing Hamming verification, the similar pairs are globally merged from three granularities. In the first stage, in each group, all similar pair results are scanned, each similar pair is numbered, and an inverted list recording the document number-similar pair number is dynamically constructed, the inverted list records in which similar pairs each document appears, and each list corresponding to the document is incrementally sorted, and in addition, an array B is used to record the minimum similar pair number that can be merged for each similar pair. After the inverted list is constructed, each list in the inverted list is scanned item by item, assuming that the first item number of each list is F, and B[F] is recorded as Min(B[F], F), and thereafter, for each item number i of the list, B[i] is recorded as Min(B[i], B[F]). After scanning, the B array will obtain the minimum similar pair number that each similar pair can be merged to, and the similar pairs with the same B array value are merged together by traversing the B array, to obtain the merged similar set of each group, and the similar sets do not intersect each other, and the number of sets is also obviously reduced compared with the number of similar pairs. The second stage is storage partition granularity merging. Based on the similar set obtained from each group, an inverted list and a B array are constructed in the same way, and the group similar set is taken as an object, and the process in the group is similar to the process in the partition granularity, to obtain the non-intersecting similar sets at the partition level. In the third stage, the results of all partitions are aggregated to the Master node of Spark, and the same processing as in the partition is performed, and the non-intersecting similar sets are further merged, to finally obtain the global similar set, and finally, only one document in each global similar set is retained, to obtain the global deduplication result. When the merging is performed at the document grouping and storage partition granularities, parallel processing is performed in a distributed environment, the number of sets aggregated to the Master node is already small, and large-scale data sets can be easily processed. Based on the large-scale data global deduplication method based on Spark provided in the embodiment, the present application effectively solves the computing bottleneck problem of global fuzzy deduplication merging, greatly reduces the time required for fuzzy deduplication of large-scale data, and improves deduplication efficiency.

[0147] Reference Figure 8 The embodiment of the present application also provides an electronic device 800, comprising:

[0148] at least one processor, and

[0149] a memory in communication connection with the at least one processor; wherein

[0150] The memory stores instructions, and the instructions are executed by the at least one processor, so that the at least one processor implements the method in any one of the embodiments of the present application when the instructions are executed.

[0151] The following describes the hardware structure of the electronic device in detail. Figure 8 The electronic device includes a processor 810, a memory 820, an input / output interface 830, a communication interface 840, and a bus 850.

[0152] The processor 810 can be implemented in a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0153] The memory 820 can be implemented in a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 820 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 820 and are called and executed by the processor 810 to implement the metaphor recognition method of the embodiments of the present application.

[0154] The input / output interface 830 is configured to implement information input and output.

[0155] The communication interface 840 is configured to implement the communication interaction between the device and other devices. The communication can be implemented in a wired manner (for example, USB, network cable, etc.) or in a wireless manner (for example, mobile network, WIFI, Bluetooth, etc.).

[0156] The bus 850 is configured to transmit information between various components (for example, the processor 810, the memory 820, the input / output interface 830, and the communication interface 840) of the device.

[0157] The processor 810, the memory 820, the input / output interface 830, and the communication interface 840 are connected to each other through the bus 850 for internal communication connection in the device.

[0158] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, and stores computer executable instructions for causing a computer to execute the metaphor recognition method of the embodiments of the present application.

[0159] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include memory that is remotely located with respect to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0161] Those skilled in the art can understand that, Figures 1 to 8 The technical solutions shown in the above figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer components than shown, or combine certain components, or different components.

[0162] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases of only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0163] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, which do not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A global deduplication method for large-scale data based on Spark, characterized in that, The method includes: Large-scale data is preprocessed to obtain multiple first-processed documents, and the first-processed documents are stored in multiple storage partitions respectively; The first processed document is grouped in each storage partition to obtain multiple document groups; Within each of the document groups, the first processed document is subjected to similarity detection to obtain multiple similar pairs; The minimum merged similarity pair for each similarity pair is determined based on the pre-configured global number of the first processed document, and all similarity pairs pointing to the same minimum merged similarity pair are merged to obtain a grouped similarity set; The minimum merged group set of each group similarity set is determined based on the global number of the first processed document, and all group similarity sets pointing to the same minimum merged group set are merged to obtain the partition similarity set; The minimum merged partition set of each partition similarity set is determined based on the global number of the first processed document, and all partition similarity sets pointing to the same minimum merged partition set are merged to obtain a global similarity set. The global number of all the first processed documents in each global similarity set is recorded through the second processed document. Remove the first global number from each of the global similarity sets from the second processed document, and summarize all the remaining global numbers in all the global similarity sets to obtain a set of document numbers to be eliminated, and broadcast the set of document numbers to be eliminated to each of the storage partitions; In each of the storage partitions, the first processed document is filtered according to the set of document numbers to be eliminated.

2. The method according to claim 1, characterized in that, The step of determining the minimum merged similarity pair for each similarity pair based on the pre-configured global number of the first processed document, and merging all similarity pairs pointing to the same minimum merged similarity pair to obtain a grouped similarity set, includes: The similar pairs are numbered, and a first hash table is constructed based on the global number of the first processed document and the number of the similar pairs. The key of the first hash table is the global number of the first processed document, and the truth value of the first hash table is an ordered linked list consisting of the numbers of all similar pairs of the first processed document corresponding to the key. A first dynamic array is constructed and assigned values ​​according to the first hash table. The minimum similarity pair for merging each similar pair is determined based on the assignment results. Multiple similar pairs pointing to the same minimum similarity pair within the same document group are merged to obtain a grouped similarity set.

3. The method according to claim 1, characterized in that, The step of determining the minimum merged group set for each group similarity set based on the global ID of the first processed document, and merging all group similarity sets pointing to the same minimum merged group set to obtain a partition similarity set, includes: The group similarity sets are numbered, and a second hash table is constructed based on the global number and the number of the group similarity sets. The key value of the second hash table is the global number of the first processed document, and the true value of the second hash table is an ordered linked list consisting of the numbers of all group similarity sets of the first processed document corresponding to the key value. A second dynamic array is constructed and assigned values ​​according to the second hash table. The minimum similarity set for merging each group similarity set is determined based on the assignment results. Multiple group similarity sets pointing to the same minimum similarity set within the same storage partition are merged to obtain a partition similarity set.

4. The method according to claim 1, characterized in that, The step of determining the minimum merge partition set for each partition similarity set based on the global ID of the first processed document, merging all similar pairs pointing to the same minimum merge partition set to obtain a global similarity set, and recording the global ID of each global similarity set including all the first processed documents through the second processed document includes: The partition similarity sets are numbered, and a third hash table is constructed based on the global number and the number of the partition similarity sets. The key value of the third hash table is the global number of the first processed document, and the true value of the third hash table is an ordered linked list consisting of the numbers of all partition similarity sets of the first processed document corresponding to the key value. A third dynamic array is constructed and assigned values ​​according to the third hash table. The minimum global set for merging each partition similarity set is determined based on the assignment results. Multiple partition similarity sets pointing to the same minimum global set are merged to obtain a global similarity set. The global number of all the first processing documents in each global similarity set is recorded through the second processing document.

5. The method according to claim 2, characterized in that, The process of constructing a first dynamic array and assigning values ​​to the first dynamic array according to the first hash table, determining the minimum similarity pair for merging each similarity pair based on the assignment results, and merging multiple similarity pairs pointing to the same minimum similarity pair within the same document group to obtain a grouped similarity set includes: Construct and initialize the first dynamic array; The first hash table is scanned to assign values ​​to the first dynamic array. The index value of each item in the first dynamic array after assignment is the number of the similar pair. The initial value stored in each item is the number of the similar pair of the item following the index value in the true value of the first hash table. For the i-th item of the first dynamic array, obtain the first stored value of the i-th item. If the first stored value is a first preset value, let the first stored value be i, where i is a positive integer. If the first stored value is not the first preset value, obtain the second stored value of the item in the first dynamic array that uses the first stored value as the index value. If the second stored value is not the first preset value, use the second stored value as the new first stored value and obtain the new second stored value using the new first stored value as the index value, until the second stored value is the first preset value. If the second stored value is the first preset value, the first stored value is determined to be the number of the smallest similar pair merged from similar pairs with similar pair number i; Based on the index values ​​of all items with the same value stored in the first dynamic array, the corresponding similar pairs in each document group are merged to obtain the group similarity set.

6. The method according to claim 3, characterized in that, The process of constructing a second dynamic array and assigning values ​​to the second dynamic array according to the second hash table, determining the minimum similarity set for merging each group similarity set based on the assignment results, and merging multiple group similarity sets pointing to the same minimum similarity set within the same storage partition to obtain a partition similarity set includes: Construct and initialize the second dynamic array; The second hash table is scanned to assign values ​​to the second dynamic array. The index value of each item in the second dynamic array after assignment is the number of the group similarity set. The initial value stored in each item is the number of the group similarity set located after the index value in the true value of the second hash table. For the j-th item of the second dynamic array, obtain the third stored value of the j-th item. If the third stored value is a second preset value, set the third stored value to j, where j is a positive integer. If the third stored value is not the second preset value, obtain the fourth stored value of the item in the second dynamic array that uses the third stored value as the index value. If the fourth stored value is not the second preset value, use the fourth stored value as the new third stored value, and use the new third stored value as the index value to obtain the new fourth stored value, until the fourth stored value is the second preset value. If the fourth stored value is the second preset value, then the third stored value is determined to be the number of the smallest similar set merged from the group similar set numbered i. The partition similarity set is obtained by merging the corresponding multiple group similarity sets according to the index values ​​of all items with the same value stored in the second dynamic array.

7. The method according to claim 4, characterized in that, The process involves constructing a third dynamic array and assigning values ​​to it according to the third hash table. Based on the assignment results, a minimum global set is determined for merging each partition similarity set. Multiple partition similarity sets pointing to the same minimum global set are merged to obtain a global similarity set. A second processing document records the global IDs of all first processing documents included in each global similarity set, including: Construct and initialize the third dynamic array; The third hash table is scanned to assign values ​​to the third dynamic array. The index value of each item in the third dynamic array after assignment is the number of the partition similarity set. The initial value stored in each item is the number of the partition similarity set located after the index value in the true value of the third hash table. For the k-th item of the third dynamic array, obtain the fifth stored value of the k-th item. If the fifth stored value is the third preset value, let the fifth stored value be k, where k is a positive integer. If the fifth stored value is not the third preset value, obtain the sixth stored value of the item in the third dynamic array that uses the fifth stored value as the index value. If the sixth stored value is not the third preset value, use the sixth stored value as the new fifth stored value, and use the new fifth stored value as the index value to obtain the new sixth stored value, until the sixth stored value is the third preset value. When the sixth stored value is the third preset value, the fifth stored value is determined to be the number of the global similarity set merged from the partition similarity set number j. The global similarity set is obtained by merging the corresponding partition similarity sets according to the index values ​​of all items with the same value stored in the third dynamic array; The global number of the first processed document included in each of the global similarity sets is recorded through the second processed document.

8. The method according to claim 1, characterized in that, The process of preprocessing large-scale data to obtain multiple first processed documents and storing these documents in multiple storage partitions includes: Multiple original input documents are extracted from the large-scale data, and the original input documents are numbered to obtain a global number for each original input document; All the original input documents are segmented into words to transform them into a set of words containing multiple words. Calculate the hash code for each word; The first processed document corresponding to each of the original input documents is generated based on the hash code and the global number, and all the first processed documents are stored in their respective storage partitions.

9. The method according to claim 1, characterized in that, The first processed document is grouped in each storage partition to obtain multiple document groups, including: Calculate the word frequency of each word in the first processed document, sort the words in each first processed document according to the word frequency, perform prefix pruning on the sorted first processed documents, and obtain a prefix array for each first processed document; In each of the storage partitions, the first processed documents that include at least one of the same words in the prefix array are grouped into the same group, resulting in multiple document groups.

10. The method according to claim 1, characterized in that, The step of performing similarity detection on the first processed document within each of the document groups to obtain multiple similar pairs includes: The document fingerprint of each of the first processed documents is calculated using a preset algorithm; The Hamming distance between the first processed documents within each document group is determined based on the document fingerprint; The first processed documents whose Hamming distance is less than the fourth preset value are classified as similar pairs.

11. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the Spark-based global deduplication method for large-scale data as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the Spark-based global deduplication method for large-scale data as described in any one of claims 1 to 10.

Images