Multi-level batch text parallel deduplication method, system and equipment and storage medium

Abstract

The invention discloses a multi-level batch text parallel deduplication method, system and device and a storage medium, and the method comprises the steps: carrying out the preprocessing of text data, and carrying out the similarity text representation and content deduplication of sensitive hashing at a chapter level, a paragraph level and a statement level; adding a priority score and a sorting rule in a duplicate removal process to eliminate duplicate removal randomness; during content deduplication, a graph structure is created according to similarity text representation of sensitive Hash, a plurality of connected components are established on the graph structure by utilizing combination and separation of union sets, and a target text is selected and reserved in each connected component by adopting a dynamic Hash threshold value; in the process, data frame objects are respectively constructed for chapter-level, paragraph-level and statement-level texts, unique identifiers are allocated, and the time complexity and the required memory space are optimized by adopting a parallel deduplication strategy of block delay calculation and pointer storage. According to the method, the duplicate removal speed and accuracy can be improved, and meanwhile, the problem of memory shortage faced by mass duplicate removal is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer text data processing, and specifically relates to a multi-level batch text parallel deduplication method, system, device and storage medium. Background Art

[0002] With the rise and widespread development of large language model technology in recent years, the number of training samples used has continued to rise. The number of samples has shown a significant explosive growth with the increase in the number of model parameters and the amount of calculation, and has gradually tended to trillions of data or more. In the process of using actual training samples as corpus for large model training, it is necessary to deduplicate a large amount of text data in order to remove a large amount of identical or similar data in order to reduce unbalanced and repeated data while also improving the quality of samples and meeting the required standards and specifications, ultimately achieving high-quality and accurate text data pre-training and fine-tuning tasks, giving full play to and mobilizing the potential of large models to meet more realistic tasks and needs.

[0003] The existing technology mainly includes three types of algorithms for deduplication of text: the first type is an algorithm based on cosine similarity, which performs standardization and word segmentation on the text data to be processed, groups the text data according to the results of word segmentation, sorts the obtained word set according to the same word segmentation and counts the word frequency, and then calculates the cosine similarity based on the vector obtained by the word frequency statistics, and directly discards the text with high similarity to achieve the purpose of deduplication. This type of algorithm takes into account the use of vector cosine similarity to achieve the purpose of deduplication, which improves the accuracy of deduplication to a certain extent, but the efficiency is not high and cannot reasonably cope with the deduplication of large batches of samples. The second type of algorithm is based on the first type of algorithm, and constructs a feature vector library. By storing the feature vectors obtained by some fixed word segments or paragraphs and articles with a fixed algorithm in a fixed area in advance, there is no need to calculate the text to be compared each time when performing the deduplication operation, but the features can be directly obtained from the feature vector library and the redundancy removal can be achieved by calculating the similarity with the calculated deduplication text features. Compared with the first type, this type of algorithm has improved efficiency and simplified the process of text deduplication. However, this type of method still faces the challenge of low deduplication accuracy and requires additional data storage space for efficient reading. The third type of algorithm is an algorithm based on local sensitive hashing. By comparing each piece of text data with a designed hash function and calculating their distance, documents with similar distances are directly removed. The processing accuracy of this type of method is much higher than the first two solutions, and the speed is also improved. However, in the batch processing of large model training samples, the final deduplication effect is not good because the traditional deduplication algorithm only considers the deduplication between documents and ignores the deduplication at the paragraph and sentence levels within the document. At the same time, the selection of duplicate text to be removed is random, lacks reasonable judgment criteria, and the similarity threshold is a fixed value, resulting in the accidental removal of duplicate text data that you want to retain, or the incorrect retention of a small amount of duplicate text. In addition, there is still a need to further improve the processing speed at the level of trillion-level data, and there are problems such as insufficient memory when processing large amounts of data that need to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a multi-level batch text parallel deduplication method, system, device and storage medium to improve the speed and accuracy of deduplication and solve the memory shortage problem faced by large-scale deduplication.

[0005] In order to achieve the above object, the present invention has the following technical solutions:

[0006] In a first aspect, a multi-level batch text parallel deduplication method is provided, comprising:

[0007] Preprocess the text data and perform sensitive hashing on similarity text representation and content deduplication at the chapter level, paragraph level, and sentence level. Priority scores and sorting rules are added to the deduplication process to eliminate randomness in deduplication.

[0008] When deduplicating content, a graph structure is created based on the similarity text representation of sensitive hashing. Multiple connected components are established on the graph structure by using the union-find method to merge and separate. Within each connected component, a dynamic hash threshold is used to select the target text to be retained.

[0009] In the above process, data frame objects are constructed for the chapter-level, paragraph-level and sentence-level texts respectively, and unique identifiers are assigned. A parallel deduplication strategy of block delay calculation and pointer storage is used to optimize the time complexity and required memory space.

[0010] As a preferred solution, the step of preprocessing the text data includes preliminary cleaning and quality inspection of the text data, format standardization of the text content, removal of special characters, chart recognition and conversion, and information desensitization, and conversion into a unified json or jsonl format.

[0011] As a preferred solution, the steps of performing similarity text representation of sensitive hashing at the chapter level, paragraph level, and sentence level respectively include:

[0012] The local sensitive hashing algorithm is used to calculate the minimum hash value of the text between chapters, paragraphs or sentences as the feature representation of the corresponding text, and several groups of different hash standard functions are used for hash coding, and the hash string groups with fixed byte lengths are concatenated to serve as the dimensionality reduction feature representation of the corresponding chapters, paragraphs or sentences. After the hash code groups of all chapters, paragraphs or sentences are obtained, the hash code groups are divided into several intervals, and the similarity of the hash code strings within the intervals is calculated according to the following formula:

[0013]

[0014] Where X and Y represent two hash code strings whose similarity needs to be calculated, |X∩Y| represents the common code of the two hash code strings, and |X∪Y| represents all codes of the two hash code strings.

[0015] As a preferred solution, the step of creating a graph structure based on the similarity text representation of sensitive hashing includes: taking the hash code groups within the interval as nodes, and connecting the nodes whose similarity is greater than the initial basic threshold to form an edge to form multiple connected components, thereby completing a rough division of the connected components;

[0016] The steps of establishing multiple connected components by merging and separating the union-find set on the graph structure, and selecting the target text to be retained by using the dynamic hash threshold in each connected component include: determining the dynamic similarity threshold according to the text content, the dynamic similarity threshold is related to the information volume and priority score of all the texts; expressing the text information volume in the form of entropy, and the priority score is a value between 0 and 1 given to the text in advance;

[0017] The calculation expression of the dynamic similarity threshold C is:

[0018]

[0019] In the formula, c is the initial basic threshold, n is the number of texts, and h is i is the information entropy of the i-th text, h max is the maximum value of information entropy, s i is the priority score of the i-th text, α and β are the weight ratios of information content and priority score respectively;

[0020] The calculation formula of information entropy is:

[0021]

[0022] In the formula, X i is the i-th text, x j is the jth token of the i-th text, p(x j ) represents the frequency of the jth word in all words, and information entropy represents the specific amount of information in a text;

[0023] After obtaining the dynamic similarity threshold C, query and merge the connected components for the constructed graph structure, including starting the query from two different nodes. If the corresponding similarity is greater than or equal to the dynamic similarity threshold C, they are merged into the same connected component, otherwise continue to traverse; select the target text in each connected component to retain, and remove the remaining text.

[0024] As a preferred solution, the step of adding priority scores and sorting rules to the deduplication process to eliminate randomness in deduplication includes:

[0025] Deduplication is performed according to the article-level deduplication rules, paragraph-level and sentence-level deduplication rules, and cross-level deduplication rules.

[0026] Article-level deduplication rule: directly use the priority score to sort, that is, in each connected component, only the article with the highest priority score is retained, and the rest of the articles are directly removed; if there are two or more articles with the highest and equal priority scores, first remove the article with a score less than the highest priority score, and then perform deduplication according to the paragraph and sentence-level deduplication rules;

[0027] Paragraph and sentence level deduplication rules: First, sort by calculating the amount of information. By calculating the amount of information in the text and sorting it in descending order, the paragraph or sentence with the largest amount of information is retained; if there are paragraphs, sentences or chapters with the same amount of information, the deduplication operation is completed according to the cross-level deduplication rules;

[0028] Cross-level deduplication rules: select the text with the largest number of tokens to retain and remove the rest of the text; for texts with the same number of tokens, sort them in order of appearance and retain the first text while excluding the rest to complete the deduplication operation.

[0029] As a preferred solution, the steps of constructing data frame objects for chapter-level, paragraph-level and sentence-level texts respectively and assigning unique identifiers include:

[0030] A doc_id is assigned to each document at the chapter level as the unique identifier of the corresponding chapter, and the paragraphs within the chapter have the paragraph sec_id added after the document id, that is, the doc_id_sec_id format is used as the unique identifier of the paragraph; the sentence-level unique identifier will add the sentence stc_id after the paragraph representation, that is, doc_id_sec_id_stc_id.

[0031] As a preferred solution, the steps of optimizing the time complexity by adopting the parallel deduplication strategy of block delay calculation include:

[0032] The number of blocks is determined according to the hardware and software conditions of the operating environment, and the existing data is divided into different blocks for distributed parallel processing. The blocks are divided according to the priority order, and the chapters are classified according to different priorities according to the priority scores. For paragraph-level and sentence-level texts, the inherent properties of the texts are used to divide the blocks, so that the texts belonging to the same chapter or the same paragraph are divided into the same block, and the different blocks are synchronously and parallelly processed in a distributed manner;

[0033] To deduplicate the data within each block, lazy calculation functions are used. Each incoming data frame object is treated as a lazy object, and all non-explicitly declared operations on the lazy object are delayed. These operations are saved in the form of calculation graphs. All operations are executed in the form of saved graph calculations only when the function that needs to be calculated is explicitly called.

[0034] As a preferred solution, the steps of optimizing the required memory space using a parallel deduplication strategy using pointer storage include:

[0035] Record the head node address of each data block, and load the address of the corresponding data block into the memory at the computing node where the data block is to be called in the computational graph; before explicitly calling the computational function, only the pointer to the specific operation data block is stored in the memory, and it is loaded into the memory through addressing when explicitly called, and allocated and optimized according to different computational and resource requirements during the computational call.

[0036] In a second aspect, a multi-level batch text parallel deduplication system is provided, comprising:

[0037] The multi-level deduplication module is used to pre-process the text data and perform sensitive hashing on similarity text representation and content deduplication at the chapter level, paragraph level, and sentence level. Priority scores and sorting rules are added to the deduplication process to eliminate randomness in deduplication.

[0038] The graph structure selection module is used to create a graph structure based on the similarity text representation of sensitive hash when deduplicating content. Multiple connected components are established on the graph structure by using the union-find method to merge and separate. Within each connected component, a dynamic hash threshold is used to select the target text to be retained.

[0039] The parallel deduplication optimization module is used to construct data frame objects for chapter-level, paragraph-level and sentence-level texts in the above process, assign unique identifiers, and use a parallel deduplication strategy of block delay calculation and pointer storage to optimize the time complexity and required memory space.

[0040] As a preferred solution, the multi-level deduplication module uses a local sensitive hashing algorithm to calculate the minimum hash value of the text between chapters, paragraphs or sentences as the feature representation of the corresponding text, uses several groups of different hash standard functions for hash encoding, and splices to obtain a hash string group with a fixed byte length as the dimensionality reduction feature representation of the corresponding chapter, paragraph or sentence text; after obtaining the hash code group of all chapters, paragraphs or sentences, the hash code group is divided into several intervals, and the similarity of the hash code string within the interval is calculated according to the following formula:

[0041]

[0042] Where X and Y represent two hash code strings whose similarity needs to be calculated, |X∩Y| represents the common code of the two hash code strings, and |X∪Y| represents all codes of the two hash code strings.

[0043] As a preferred solution, when the graph structure selection module creates a graph structure based on the similarity text representation of sensitive hash, the hash code groups within the interval are taken as nodes, and the nodes with similarity greater than the initial basic threshold are connected into an edge to form multiple connected components, completing the rough division of the connected components; the graph structure selection module uses the merging and separation of the union-find set on the graph structure to establish multiple connected components, and when the dynamic hash threshold is used to select the target text to be retained within each connected component, the dynamic similarity threshold is determined according to the text content, and the dynamic similarity threshold is related to the information volume and priority score of all texts; the text information volume is represented in the form of entropy, and the priority score is a value between 0 and 1 that is given to the text in advance;

[0044] The calculation expression of the dynamic similarity threshold C is:

[0045]

[0046] In the formula, c is the initial basic threshold, n is the number of texts, and h is i is the information entropy of the i-th text, h max is the maximum value of information entropy, s i is the priority score of the i-th text, α and β are the weight ratios of information content and priority score respectively;

[0047] The calculation formula of information entropy is:

[0048]

[0049] In the formula, X i is the i-th text, x j is the jth token of the i-th text, p(x j ) represents the frequency of the jth word in all words, and information entropy represents the specific amount of information in a text;

[0050] After obtaining the dynamic similarity threshold C, query and merge the connected components for the constructed graph structure, including starting the query from two different nodes. If the corresponding similarity is greater than or equal to the dynamic similarity threshold C, they are merged into the same connected component, otherwise continue to traverse; select the target text in each connected component to retain, and remove the remaining text.

[0051] As a preferred solution, the parallel deduplication optimization module constructs data frame objects for chapter-level, paragraph-level and sentence-level texts respectively, and the steps of assigning unique identifiers include:

[0052] Each document at the chapter level is assigned a doc_id as the unique identifier of the corresponding chapter, and the paragraphs within the chapter are marked with the paragraph sec_id after the document id, that is, the doc_id_sec_id format is used as the unique identifier of the paragraph; the sentence-level unique identifier is marked with the sentence stc_id after the paragraph representation, that is, doc_id_sec_id_stc_id;

[0053] The steps of optimizing the time complexity by adopting the parallel deduplication strategy of block delay calculation in the parallel deduplication optimization module include:

[0054] The number of blocks is determined according to the hardware and software conditions of the operating environment, and the existing data is divided into different blocks for distributed parallel processing. The blocks are divided according to the priority order, and the chapters are classified according to different priorities according to the priority scores. For paragraph-level and sentence-level texts, the inherent properties of the texts are used to divide the blocks, so that the texts belonging to the same chapter or the same paragraph are divided into the same block, and the different blocks are synchronously and parallelly processed in a distributed manner;

[0055] For deduplication of data within each block, lazy calculation functions are used to treat each incoming data frame object as a lazy object, delaying all non-explicitly declared operations on lazy objects. These operations are saved in the form of calculation graphs. All operations are executed in the form of saved graph calculations only when the function that needs to be calculated is explicitly called.

[0056] The steps of optimizing the memory space required by the parallel deduplication optimization module using the parallel deduplication strategy stored by pointers include: recording the head node address of each data block, and calling the address of the corresponding data block into the memory at the computing node where the data block is to be called in the computing graph; before explicitly calling the computing function, only the pointer to the specific operation data block is stored in the memory, and it is only called into the memory through addressing when it is explicitly called, and allocation and optimization are performed according to different computing and resource requirements when the computing call is made.

[0057] In a third aspect, an electronic device is provided, comprising a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the multi-level batch text parallel deduplication method.

[0058] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the multi-level batch text parallel deduplication method is implemented.

[0059] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects:

[0060] In view of the single level of existing large-scale text deduplication methods, and the randomness and limited threshold of deduplication text, the method of the present invention respectively performs sensitive hashing similarity text representation and content deduplication by designing a multi-level mode at the chapter level, paragraph level and sentence level, and adds priority scores and sorting rules to the deduplication process to eliminate the randomness of deduplication, and by creating a graph structure, multiple connected components are established by using the merging and separation of the union-find set, and the text removal of the node is adaptively completed by using the dynamic hash threshold scheme inside each connected component, and finally a complete and efficient text deduplication effect is achieved. At the same time, the method of the present invention takes into account that large quantities of text data cannot be put into a small memory in batches and the calculation efficiency is low. For large quantities of text, a block delay calculation scheme is designed, and the calculation is performed lazily before the final deduplication is performed, thereby improving the calculation efficiency. In addition, a special pointer storage strategy is adopted to process the text duplication detection task in parallel while accepting the long text processing that is much larger than the memory, so that the memory shortage problem faced by large-scale deduplication of large model training samples can be solved in a real sense.

[0061] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0063] Figure 1 Flow chart of a multi-level batch text parallel deduplication method according to an embodiment of the present invention;

[0064] Figure 2 A flow chart of a union-find query based on a dynamic similarity threshold according to an embodiment of the present invention;

[0065] Figure 3 The embodiment of the present invention adopts a parallel deduplication flow chart of block delay calculation;

[0066] Figure 4 A schematic diagram of a calculation graph structure using block delay calculation in an embodiment of the present invention;

[0067] Figure 5 A schematic diagram of storing block pointers for deduplicated text according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0069] The embodiment of the present invention proposes a multi-level batch text parallel deduplication method based on priority rules and dynamic hash thresholds. On the basis of the traditional local sensitive hash algorithm for chapter-level deduplication, an additional design is made for content deduplication at the paragraph level and sentence level within each article, so as to realize a multi-level efficient deduplication mode for large quantities of text. On this basis, priority scores are given to important chapters, paragraphs and sentences, and after calculating the similarity, a graph structure is constructed to represent them, in which the nodes represent these hash values, and the edges are represented by the similarity between the hash values. The priority score will exist as the attribute value of each node. Subsequently, a graph search algorithm of union-find is adopted, and a dynamic similarity hash threshold is designed for calculation based on the similarity distribution of the documents. The graph is divided into multiple connected components according to the threshold, that is, each connected component represents repeated text, and the deduplication operation is completed according to the priority score and specific sorting rules. The method of the embodiment of the present invention additionally adds parallel object construction, realizes block delay calculation to improve the computational efficiency of deduplication processing, and only stores pointers to these data blocks in the memory, thereby allowing the amount of data to be much larger than the memory, so as to fully cope with the large batch sample deduplication work required for large model training.

[0070] See also Figure 1 The multi-level batch text parallel deduplication method of the embodiment of the present invention mainly includes:

[0071] Preprocess the text data and perform sensitive hashing on similarity text representation and content deduplication at the chapter level, paragraph level, and sentence level. Priority scores and sorting rules are added to the deduplication process to eliminate randomness in deduplication.

[0072] When deduplicating content, a graph structure is created based on the similarity text representation of sensitive hashing. Multiple connected components are established on the graph structure by using the union-find method to merge and separate. Within each connected component, a dynamic hash threshold is used to select the target text to be retained.

[0073] In the above process, data frame objects are constructed for the chapter-level, paragraph-level and sentence-level texts respectively, and unique identifiers are assigned. A parallel deduplication strategy of block delay calculation and pointer storage is used to optimize the time complexity and required memory space.

[0074] In a possible implementation, the step of preprocessing text data in the embodiment of the present invention includes: preliminary cleaning and quality inspection of text data, format standardization of text content, removal of special characters, chart recognition and conversion, information desensitization, and conversion into a unified json or jsonl format. On this basis, the embodiment of the present invention adopts a three-level deduplication strategy for text data, that is, multi-level deduplication is performed at the chapter level, paragraph level, and sentence level. Among them, the following steps are consistent with the three-level deduplication process unless otherwise specified, and can be specifically described as:

[0075] The local sensitive hashing algorithm is used between each article to calculate the minimum hash value of the text as the feature representation of the text. The embodiment of the present invention uses 250 groups of different hash standard functions for hash coding, that is, texts of any length will be hash coded after fixed grouping, and then spliced ​​to obtain 250 hash string groups of fixed byte length. This hash string group will be used as the dimensionality reduction feature representation of the text of the chapter. This step completes the traversal of each document, so the time complexity is O(n). After obtaining the hash code group of all chapters, it is divided into 25 intervals, so that each interval has 10 hash code strings, and the similarity calculation is performed within these 10 hash code strings. The similarity Sim calculation formula is as follows:

[0076]

[0077] In the formula, X and Y represent the two hash code strings whose similarity needs to be calculated, |X∩Y| represents the common code of the two hash code strings, and |X∪Y| represents all the codes of the two hash code strings. The calculation of similarity here is accompanied by the creation of a deduplication graph structure. The creation process uses the 25 hash code groups of each document as nodes, and the nodes with a similarity greater than the initial threshold will be connected to form an edge. The initial threshold in the embodiment of the present invention is set to 0.85, thus forming multiple connected components. Because only 10 hash code strings of a node in each connected component need to be calculated for similarity each time, there is no need for pairwise comparison calculations, and the time complexity of this step is increased from O(n 2 ) is optimized to O(n).

[0078] Because the calculation in the previous step uses the similarity of the initial threshold for rough grouping, the accuracy of the connected component division needs to be improved. Therefore, after completing the construction of the nodes and edges of the entire graph in the previous step, the embodiment of the present invention designs a dynamic similarity threshold union-find strategy to adaptively further refine the repeated clusters. The process is shown in the attached figure. Figure 2 As shown, specifically:

[0079] First, the dynamic threshold is determined according to the text content. The dynamic threshold is closely related to the information volume and priority score of all texts. The text information volume is expressed in the form of entropy, and the priority score is a value between 0 and 1 that is manually given in advance for each text. The strategy adopted by the dynamic threshold is: the greater the amount of text information, the higher the priority level, and the more relaxed the similarity threshold will be, so as to retain more valuable text information; conversely, when the amount of text information is small or the priority is low, it means that the quality or value of this part of the text is not high, so a stricter similarity threshold is given to filter more duplicate texts. Therefore, the calculation formula of the dynamic similarity threshold C is as follows:

[0080]

[0081] In the formula, c represents the initial basic threshold, n represents the number of texts, and h i represents the information entropy of the i-th text, h max Indicates the maximum value of information entropy, that is, the value is a fixed value log2n, s i Represents the priority score of the ith text, α and β represent the weighted proportions of the information volume and priority score, respectively. In the formula, the information entropy and the priority score are actually normalized and then weighted summed to obtain the final dynamic threshold. In addition, the embodiment of the present invention sets the initial basic threshold c to 0.5, and the hyperparameters α and β to 0.1 and 0.25, respectively, so that the adaptive dynamic threshold required for deduplication of this batch of texts can be calculated. It is worth noting that in the process of deduplication at the paragraph level and sentence level, it is better than the lack of priority scores, so the calculation formula of the dynamic threshold only retains the first two items, where the initial basic threshold c remains unchanged, and the value of α is changed to 0.2, so as to meet the requirements of text deduplication at a more detailed level. In particular, the calculation formula for information entropy is:

[0082]

[0083] In the formula, X i represents the i-th text, x j represents the jth token of the i-th text. A token refers to a text segmentation unit in an article, which is generally a sequence of words or subwords. j ) represents the frequency of the jth word in all words. The calculation of information entropy represents the specific amount of information in a text, and its value range is [0, log2n]. The larger the amount of information, the more information the text has that can be learned.

[0084] After obtaining the dynamic threshold, the next step is to use the idea of ​​union-find to query and merge connected components for the constructed graph structure, that is, start the query from two different nodes. If the corresponding similarity is greater than or equal to the dynamic threshold, they are merged into the same connected component, otherwise continue to traverse. After this step, multiple connected components that are considered to have duplicate or highly similar texts are finally obtained. The time complexity of this step is O(logn), where n is the number of nodes. Then enter the last step of the deduplication algorithm, select the best text in each connected component to retain, and remove the remaining texts. In this step, a sorting rule based on priority score is designed. Different from the random text removal method of other current deduplication algorithms, the embodiment of the present invention adopts a specifically designed rule for multi-level deduplication as follows:

[0085] (1) Article-level deduplication rules: Article-level text uses a priori priority scores to represent the importance of documents. Therefore, the deduplication rules directly use priority scores for sorting. That is, only the articles with the highest priority scores in each connected component are retained, and the rest of the articles are directly removed. The special case is that there are two or more articles with the highest and equal priority scores. For this special case, first remove the articles with a score less than the highest priority score, and then enter the paragraph and sentence-level deduplication rules of rule (2) for further deduplication operations.

[0086] (2) Paragraph-level and sentence-level deduplication rules: Due to the lack of priority scores, paragraph-level and sentence-level deduplication first uses the information volume calculation method to sort. By calculating the information volume of the text and sorting it in descending order, the paragraph or sentence with the largest information volume is retained. For multiple chapters with the same priority in rule (1), this step calculates the information volume among these retained chapters and retains the chapter with the largest information entropy. Similarly, if there are paragraphs, sentences or special chapters with the same information entropy, the deduplication rules of rule (3) are entered to complete the final deduplication operation.

[0087] (3) Cross-level deduplication rule: This rule is only applicable to a small number of paragraphs or sentences with the same amount of information, and a very small number of chapters with the same priority and the same amount of information. Under this rule, the text with the largest number of tokens will be retained, and the rest of the text will be removed. This rule generally believes that texts with more tokens have more learnable information. After the final deduplication rule is filtered, there may be very few texts with the same number of tokens. In actual operation, this basically does not exist, but it is still possible from a theoretical analysis. Therefore, for these extremely special cases, the final deduplication operation is completed by sorting in order of appearance, retaining the first text and excluding other texts.

[0088] Although the above sorting rules involve sorting, the actual algorithm implementation only requires one traversal of each node within each connected component. The relevant scores, amount of information, and number of words can be retained in the form of attributes during the traversal process and the maximum value can be recorded, so the final time complexity is also O(n).

[0089] After screening by the above-mentioned sorting rules, the overall deduplication process is completed. The large-volume text deduplication operation finally retains the text with the highest quality and the largest amount of information through various strategies designed by the embodiment of the present invention, thereby completing accurate deduplication. In addition, the time complexity of the overall deduplication algorithm is O(n)+O(logn)+O(n)=O(n), so the deduplication algorithm and strategy adopted by the embodiment of the present invention only need to be completed within the time complexity of O(n), and at the same time have the characteristics of efficient and fast deduplication.

[0090] In the specific implementation of the embodiment of the present invention, because there are hundreds of millions of large quantities of texts to be deduplicated, even under the time complexity of O(n), it still takes a certain amount of time to complete the deduplication operation. At the same time, a large memory space is required to store hash-coded nodes and similarity edges as well as various attributes and relationships, which is also a large overhead. Therefore, the embodiment of the present invention first designs a block delay calculation solution. The process of this solution is shown in the attached figure. Figure 3 shown.

[0091] Specifically, after cleaning and quality inspection of the input batch text data, data frame objects are constructed for the chapter-level, paragraph-level and sentence-level texts. At the chapter level, each document is assigned a doc_id as the unique identifier of the chapter, and the paragraphs within the article are added with the paragraph sec_id after the document id, that is, the doc_id_sec_id format is used as the unique identifier of the paragraph. Similarly, the unique identifier of the sentence will be added with the sentence stc_id after the paragraph representation, that is, doc_id_sec_id_stc_id. The purpose of adding the identification strategy in the process of constructing the data frame object is to facilitate the subsequent block delay calculation and pointer storage. After the data frame object is constructed, the next step is to block the batch text data, that is, determine the number of blocks K according to the hardware and software conditions of the operating environment, and then divide the existing data into different chunks for distributed parallel processing, thereby greatly improving the processing speed and operating efficiency. The block division strategy here adopts priority order block division, because the scheme designed in the embodiment of the present invention gives a priority score for the chapter level, so the chapters can be roughly classified according to different priorities by judging the size of the score. There is no need for accurate sorting here, but in the case of preliminary random grouping, the sampling method is used for exchange to ensure that the priorities of most texts in an interval are roughly similar, so it only needs to be completed within the time complexity of O(logn). As for paragraph-level and sentence-level texts, they do not have priority scores, but they can be divided into blocks using their own inherent properties, that is, texts belonging to the same document or the same paragraph are divided into the same block as much as possible, and synchronous and parallel distributed processing is performed on different blocks. The above scheme takes into account the priority of the chapter and ensures the continuity and coherence of the processed text.

[0092] In addition, based on the block division, the embodiment of the present invention introduces a delayed calculation mode. For the deduplication processing and operation of the internal data of each block, all lazy calculation functions are used, that is, this strategy considers each incoming data frame object as an lazy object, and all non-explicitly declared operations on the object will be delayed, and these operations will be saved in the form of a calculation graph. Only when the function that needs to be calculated is explicitly called, will all operations be executed in the form of saved graph calculation. Specific examples are shown in the attached Figure 4 As shown in the figure, the related operations, computing nodes and dependencies are saved as a computing graph structure, and optimization, resource allocation and execution are performed only when the calculation is explicitly called. The advantage of this is that unified processing is finally performed by saving the computing graph, and the computing order and scheduling order can be optimized for different resources required by different computing modes, thereby improving computing and processing efficiency.

[0093] The above-mentioned block delay calculation mode proposed in the embodiment of the present invention is optimized at the time level. In addition, at the spatial level, the deduplication strategy proposed in the embodiment of the present invention does not read large batches of text directly into the memory, but records the address of the head node of each data block on the basis of block delay calculation, and calls the address of the data block into the memory at the calculation node that needs to call the data block in the calculation graph. Figure 5 As shown in the figure, this strategy is equivalent to storing only a pointer to the specific operation data block in the memory before explicitly calling the calculation function, and addressing it into the memory only at the last explicit call. However, at the last calculation call, it will be reasonably allocated and optimized according to different calculation and resource requirements, so that the deduplication operation can be completed without a large memory space.

[0094] In general, the parallel deduplication scheme of block delayed calculation and pointer storage proposed and adopted in the embodiment of the present invention significantly improves the efficiency of computing time and space when processing large quantities of text data. By dividing the deduplication task into blocks and delaying the calculation, the occurrence of redundant calculations is reduced, and pointer storage further optimizes the use of memory. In addition, the application of this solution on a multi-machine cloud platform shows extremely high adaptability and scalability, and can be easily expanded to larger-scale computing tasks, while increasing the processing speed and reducing the system's resource consumption.

[0095] Another embodiment of the present invention further provides a multi-level batch text parallel deduplication system, comprising:

[0096] The multi-level deduplication module is used to pre-process the text data and perform sensitive hashing on similarity text representation and content deduplication at the chapter level, paragraph level, and sentence level. Priority scores and sorting rules are added to the deduplication process to eliminate randomness in deduplication.

[0097] The graph structure selection module is used to create a graph structure based on the similarity text representation of sensitive hash when deduplicating content. Multiple connected components are established on the graph structure by using the union-find method to merge and separate. Within each connected component, a dynamic hash threshold is used to select the target text to be retained.

[0098] The parallel deduplication optimization module is used to construct data frame objects for chapter-level, paragraph-level and sentence-level texts in the above process, assign unique identifiers, and use a parallel deduplication strategy of block delay calculation and pointer storage to optimize the time complexity and required memory space.

[0099] In a possible implementation, the multi-level deduplication module uses a local sensitive hashing algorithm to calculate the minimum hash value of the text between chapters, paragraphs or sentences as the feature representation of the corresponding text, uses several groups of different hash standard functions for hash encoding, and concatenates a hash string group of fixed byte length as the dimensionality reduction feature representation of the corresponding chapter, paragraph or sentence text; after obtaining the hash code group of all chapters, paragraphs or sentences, the hash code group is divided into several intervals, and the similarity calculation of the hash code string within the interval is performed according to the following formula:

[0100]

[0101] Where X and Y represent two hash code strings whose similarity needs to be calculated, |X∩Y| represents the common code of the two hash code strings, and |X∪Y| represents all codes of the two hash code strings.

[0102] Furthermore, when the graph structure selection module creates a graph structure based on the similarity text representation of the sensitive hash, the hash code groups within the interval are taken as nodes, and the nodes with similarity greater than the initial basic threshold are connected into an edge to form multiple connected components, completing the rough division of the connected components; the graph structure selection module uses the merging and separation of the union-find set on the graph structure to establish multiple connected components, and when the dynamic hash threshold is used to select the target text to be retained within each connected component, the dynamic similarity threshold is determined according to the text content, and the dynamic similarity threshold is related to the information volume and priority score of all texts; the text information volume is represented in the form of entropy, and the priority score is a value between 0 and 1 given to the text in advance; the calculation expression of the dynamic similarity threshold C is:

[0103]

[0104] In the formula, c is the initial basic threshold, n is the number of texts, and h is i is the information entropy of the i-th text, h max is the maximum value of information entropy, s i is the priority score of the i-th text, α and β are the weight ratios of information content and priority score respectively;

[0105] The calculation formula of information entropy is:

[0106]

[0107] In the formula, X i is the i-th text, x j is the jth token of the i-th text, p(x j ) represents the frequency of the jth word in all words, and information entropy represents the specific amount of information in a text;

[0108] After obtaining the dynamic similarity threshold C, query and merge the connected components for the constructed graph structure, including starting the query from two different nodes. If the corresponding similarity is greater than or equal to the dynamic similarity threshold C, they are merged into the same connected component, otherwise continue to traverse; select the target text in each connected component to retain, and remove the remaining text.

[0109] In a possible implementation, the parallel deduplication optimization module constructs data frame objects for chapter-level, paragraph-level, and sentence-level texts respectively, and the step of assigning unique identifiers includes:

[0110] Each document at the chapter level is assigned a doc_id as the unique identifier of the corresponding chapter, and the paragraphs within the chapter are marked with the paragraph sec_id after the document id, that is, the doc_id_sec_id format is used as the unique identifier of the paragraph; the sentence-level unique identifier is marked with the sentence stc_id after the paragraph representation, that is, doc_id_sec_id_stc_id;

[0111] The steps of optimizing the time complexity by adopting the parallel deduplication strategy of block delay calculation in the parallel deduplication optimization module include:

[0112] The number of blocks is determined according to the hardware and software conditions of the operating environment, and the existing data is divided into different blocks for distributed parallel processing. The blocks are divided according to the priority order, and the chapters are classified according to different priorities according to the priority scores. For paragraph-level and sentence-level texts, the inherent properties of the texts are used to divide the blocks, so that the texts belonging to the same chapter or the same paragraph are divided into the same block, and the different blocks are synchronously and parallelly processed in a distributed manner;

[0113] For deduplication of data within each block, lazy calculation functions are used to treat each incoming data frame object as a lazy object, delaying all non-explicitly declared operations on lazy objects. These operations are saved in the form of calculation graphs. All operations are executed in the form of saved graph calculations only when the function that needs to be calculated is explicitly called.

[0114] The steps of optimizing the memory space required by the parallel deduplication optimization module using the parallel deduplication strategy stored by pointers include: recording the head node address of each data block, and calling the address of the corresponding data block into the memory at the computing node where the data block is to be called in the computing graph; before explicitly calling the computing function, only the pointer to the specific operation data block is stored in the memory, and it is only called into the memory through addressing when it is explicitly called, and allocation and optimization are performed according to different computing and resource requirements when the computing call is made.

[0115] Another embodiment of the present invention further proposes an electronic device, including a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the multi-level batch text parallel deduplication method.

[0116] Another embodiment of the present invention further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the multi-level batch text parallel deduplication method is implemented.

[0117] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals. For ease of explanation, the above content only shows the part related to the embodiment of the present invention. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium is non-temporary and can be stored in a storage device formed by various electronic devices, which can realize the execution process recorded in the method of the embodiment of the present invention.

[0118] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-level batch text parallel deduplication method, characterized in that: include: Preprocess the text data and perform sensitive hashing on similarity text representation and content deduplication at the chapter level, paragraph level, and sentence level. Priority scores and sorting rules are added to the deduplication process to eliminate randomness in deduplication. When deduplicating content, a graph structure is created based on the similarity text representation of sensitive hashing. Multiple connected components are established on the graph structure by using the union-find method to merge and separate. Within each connected component, a dynamic hash threshold is used to select the target text to be retained. In the above process, data frame objects are constructed for the chapter-level, paragraph-level and sentence-level texts respectively, and unique identifiers are assigned. A parallel deduplication strategy of block delay calculation and pointer storage is used to optimize the time complexity and required memory space.

2. The multi-level batch text parallel deduplication method according to claim 1 is characterized in that: The step of preprocessing the text data includes preliminary cleaning and quality inspection of the text data, format standardization of the text content, removal of special characters, chart recognition and conversion, and information desensitization, and conversion into a unified json or jsonl format.

3. The multi-level batch text parallel deduplication method according to claim 1 is characterized in that: The steps of performing sensitive hashing on similarity text representation at the chapter level, paragraph level, and sentence level include: The local sensitive hashing algorithm is used to calculate the minimum hash value of the text between chapters, paragraphs or sentences as the feature representation of the corresponding text, and several groups of different hash standard functions are used for hash coding, and the hash string groups with fixed byte lengths are concatenated to serve as the dimensionality reduction feature representation of the corresponding chapters, paragraphs or sentences. After the hash code groups of all chapters, paragraphs or sentences are obtained, the hash code groups are divided into several intervals, and the similarity of the hash code strings within the intervals is calculated according to the following formula: Where X and Y represent two hash code strings whose similarity needs to be calculated, |X∩Y| represents the common code of the two hash code strings, and |X∪Y| represents all codes of the two hash code strings.

4. The multi-level batch text parallel deduplication method according to claim 3 is characterized in that: The step of creating a graph structure based on the similarity text representation of the sensitive hash includes: taking the hash code groups within the interval as nodes, and connecting the nodes whose similarity is greater than the initial basic threshold to form an edge to form multiple connected components, thereby completing a rough division of the connected components; The steps of establishing multiple connected components by merging and separating the union-find set on the graph structure, and selecting the target text to be retained by using the dynamic hash threshold in each connected component include: determining the dynamic similarity threshold according to the text content, the dynamic similarity threshold is related to the information volume and priority score of all the texts; expressing the text information volume in the form of entropy, and the priority score is a value between 0 and 1 given to the text in advance; The calculation expression of the dynamic similarity threshold C is: In the formula, c is the initial basic threshold, n is the number of texts, and h is i is the information entropy of the i-th text, h max is the maximum value of information entropy, s i is the priority score of the i-th text, α and β are the weight ratios of information content and priority score respectively; The calculation formula of information entropy is: Where, X i is the i-th text, x j is the jth token of the i-th text, p(x j ) represents the frequency of the jth word in all words, and information entropy represents the specific amount of information in a text; After obtaining the dynamic similarity threshold C, query and merge the connected components for the constructed graph structure, including starting the query from two different nodes. If the corresponding similarity is greater than or equal to the dynamic similarity threshold C, they are merged into the same connected component, otherwise continue to traverse; select the target text in each connected component to retain, and remove the remaining text.

5. The multi-level batch text parallel deduplication method according to claim 4 is characterized in that: The step of adding priority scores and sorting rules to the deduplication process to eliminate randomness of deduplication includes: Deduplication is performed according to the article-level deduplication rules, paragraph-level and sentence-level deduplication rules, and cross-level deduplication rules. Article-level deduplication rule: directly use the priority score to sort, that is, in each connected component, only the article with the highest priority score is retained, and the rest of the articles are directly removed; if there are two or more articles with the highest and equal priority scores, first remove the article with a score less than the highest priority score, and then perform deduplication according to the paragraph and sentence-level deduplication rules; Paragraph and sentence level deduplication rules: First, sort by calculating the amount of information. By calculating the amount of information in the text and sorting it in descending order, the paragraph or sentence with the largest amount of information is retained; if there are paragraphs, sentences or chapters with the same amount of information, the deduplication operation is completed according to the cross-level deduplication rules; Cross-level deduplication rules: select the text with the largest number of tokens to retain and remove the rest of the text; for texts with the same number of tokens, sort them in order of appearance and retain the first text while excluding the rest to complete the deduplication operation.

6. The multi-level batch text parallel deduplication method according to claim 1, characterized in that: The steps of constructing data frame objects for chapter-level, paragraph-level and sentence-level texts respectively and assigning unique identifiers include: A doc_id is assigned to each document at the chapter level as the unique identifier of the corresponding chapter, and the paragraphs within the chapter have the paragraph sec_id added after the document id, that is, the doc_id_sec_id format is used as the unique identifier of the paragraph; the sentence-level unique identifier will add the sentence stc_id after the paragraph representation, that is, doc_id_sec_id_stc_id.

7. The multi-level batch text parallel deduplication method according to claim 1, characterized in that: The steps to optimize the time complexity by using the parallel deduplication strategy of block delay calculation include: The number of blocks is determined according to the hardware and software conditions of the operating environment, and the existing data is divided into different blocks for distributed parallel processing. The blocks are divided according to the priority order, and the chapters are classified according to different priorities according to the priority scores. For paragraph-level and sentence-level texts, the inherent properties of the texts are used to divide the blocks, so that the texts belonging to the same chapter or the same paragraph are divided into the same block, and the different blocks are synchronously and parallelly processed in a distributed manner; To deduplicate the data within each block, lazy calculation functions are used. Each incoming data frame object is treated as a lazy object, and all non-explicitly declared operations on the lazy object are delayed. These operations are saved in the form of calculation graphs. All operations are executed in the form of saved graph calculations only when the function that needs to be calculated is explicitly called.

8. The multi-level batch text parallel deduplication method according to claim 7, characterized in that: The steps of optimizing the memory space required by the parallel deduplication strategy using pointer storage include: Record the head node address of each data block, and load the address of the corresponding data block into the memory at the computing node where the data block is to be called in the computational graph; before explicitly calling the computational function, only the pointer to the specific operation data block is stored in the memory, and it is loaded into the memory through addressing when explicitly called, and allocated and optimized according to different computational and resource requirements during the computational call.

9. A multi-level batch text parallel deduplication system, characterized in that: include: The multi-level deduplication module is used to pre-process the text data and perform sensitive hashing on similarity text representation and content deduplication at the chapter level, paragraph level, and sentence level. Priority scores and sorting rules are added to the deduplication process to eliminate randomness in deduplication. The graph structure selection module is used to create a graph structure based on the similarity text representation of sensitive hash when deduplicating content. Multiple connected components are established on the graph structure by using the union-find method to merge and separate. Within each connected component, a dynamic hash threshold is used to select the target text to be retained. The parallel deduplication optimization module is used to construct data frame objects for chapter-level, paragraph-level and sentence-level texts in the above process, assign unique identifiers, and use a parallel deduplication strategy of block delay calculation and pointer storage to optimize the time complexity and required memory space.

10. The multi-level batch text parallel deduplication system according to claim 9, characterized in that: The multi-level deduplication module uses a local sensitive hashing algorithm to calculate the minimum hash value of the text between chapters, paragraphs or sentences as the feature representation of the corresponding text, uses several groups of different hash standard functions for hash coding, and concatenates hash string groups of fixed byte length as the dimensionality reduction feature representation of the corresponding chapter, paragraph or sentence text; after obtaining the hash code groups of all chapters, paragraphs or sentences, the hash code groups are divided into several intervals, and the similarity calculation of the hash code strings within the intervals is performed according to the following formula: Where X and Y represent two hash code strings whose similarity needs to be calculated, |X∩Y| represents the common code of the two hash code strings, and |X∪Y| represents all codes of the two hash code strings.

11. The multi-level batch text parallel deduplication system according to claim 10, characterized in that: When the graph structure selection module creates a graph structure based on the similarity text representation of the sensitive hash, the hash code groups within the interval are used as nodes, and the nodes with similarity greater than the initial basic threshold are connected into an edge to form multiple connected components, completing the rough division of the connected components; the graph structure selection module uses the merging and separation of the union-find set on the graph structure to establish multiple connected components, and when the dynamic hash threshold is used to select the target text to be retained within each connected component, the dynamic similarity threshold is determined according to the text content, and the dynamic similarity threshold is related to the information volume and priority score of all texts; the text information volume is represented in the form of entropy, and the priority score is a value between 0 and 1 that is given to the text in advance; The calculation expression of the dynamic similarity threshold C is: In the formula, c is the initial basic threshold, n is the number of texts, and h is i is the information entropy of the i-th text, h max is the maximum value of information entropy, s i is the priority score of the i-th text, α and β are the weight ratios of information content and priority score respectively; The calculation formula of information entropy is: Where, X i is the i-th text, x j is the jth token of the i-th text, p(x j ) represents the frequency of the jth word in all words, and information entropy represents the specific amount of information in a text; After obtaining the dynamic similarity threshold C, query and merge the connected components for the constructed graph structure, including starting the query from two different nodes. If the corresponding similarity is greater than or equal to the dynamic similarity threshold C, they are merged into the same connected component, otherwise continue to traverse; select the target text in each connected component to retain, and remove the remaining text.

12. The multi-level batch text parallel deduplication system according to claim 9, characterized in that: The parallel deduplication optimization module constructs data frame objects for chapter-level, paragraph-level and sentence-level texts respectively, and the steps of assigning unique identifiers include: Each document at the chapter level is assigned a doc_id as the unique identifier of the corresponding chapter, and the paragraphs within the chapter are marked with the paragraph sec_id after the document id, that is, the doc_id_sec_id format is used as the unique identifier of the paragraph; the sentence-level unique identifier is marked with the sentence stc_id after the paragraph representation, that is, doc_id_sec_id_stc_id; The steps of optimizing the time complexity by adopting the parallel deduplication strategy of block delay calculation in the parallel deduplication optimization module include: The number of blocks is determined according to the hardware and software conditions of the operating environment, and the existing data is divided into different blocks for distributed parallel processing. The blocks are divided according to the priority order, and the chapters are classified according to different priorities according to the priority scores. For paragraph-level and sentence-level texts, the inherent properties of the texts are used to divide the blocks, so that the texts belonging to the same chapter or the same paragraph are divided into the same block, and the different blocks are synchronously and parallelly processed in a distributed manner; For deduplication of data within each block, lazy calculation functions are used to treat each incoming data frame object as a lazy object, delaying all non-explicitly declared operations on lazy objects. These operations are saved in the form of calculation graphs. All operations are executed in the form of saved graph calculations only when the function that needs to be calculated is explicitly called. The steps of optimizing the memory space required by the parallel deduplication optimization module using the parallel deduplication strategy stored by pointers include: recording the head node address of each data block, and calling the address of the corresponding data block into the memory at the computing node where the data block is to be called in the computing graph; before explicitly calling the computing function, only the pointer to the specific operation data block is stored in the memory, and it is only called into the memory through addressing when it is explicitly called, and allocation and optimization are performed according to different computing and resource requirements when the computing call is made.

13. An electronic device, characterized in that: It comprises a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the multi-level batch text parallel deduplication method as described in any one of claims 1 to 8.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by the processor, it implements the multi-level batch text parallel deduplication method as described in any one of claims 1 to 8.

Images