Text corpus deduplication processing method, deduplication processing system and storage medium

Abstract

The present application provides a deduplication processing method, deduplication processing system and storage medium for text corpus. The deduplication method for text corpus includes: randomly sampling the global text corpus to be deduplicated to obtain a subset of the text corpus to be deduplicated; dividing the subset of the text corpus to be deduplicated into multiple classification sets according to a preset text classification model; hierarchically clustering the text corpus in each classification set to obtain a hierarchical clustering structure; dividing the global text corpus to be deduplicated into multiple corpus buckets according to the hierarchical clustering structure; after deduplicating all corpus buckets within the bucket, performing global deduplication to obtain the deduplicated text. In this way, the deduplication process of the text corpus is optimized, which is conducive to saving the computing resources and storage resources required for the deduplication process.

Description

Technical Field

[0001] The present invention relates to the technical field of text corpus deduplication, and in particular to a text corpus deduplication processing method, a deduplication processing system and a storage medium. Background Art

[0002] With the rapid development of the internet and information technology, the amount of information is growing exponentially, and vast quantities of text data are being created, stored, and exchanged. This process inevitably generates duplicate or highly similar content. This duplicate data not only takes up valuable storage space but can also waste computing resources. This is especially true in large-scale data analysis and machine learning model training, where duplicated data wastes training time and consumes unnecessary computing power. For applications like search engines and recommendation systems, the presence of duplicate content can negatively impact user experience, reduce the relevance and diversity of search results, and ultimately affect overall system performance.

[0003] Text deduplication technology aims to identify and remove redundant text corpora, thereby improving the quality and efficiency of datasets. It avoids overfitting caused by duplicate samples and enhances the generalization of machine learning models. Furthermore, text deduplication helps optimize the output of information retrieval systems, providing users with more diverse and relevant results. Furthermore, during the corpus construction process, text deduplication maintains data consistency and structure, providing a solid and reliable foundation for subsequent research and development.

[0004] Related technologies use a mixed approach to deduplicate text by processing text from various fields together. While this approach can cover the entire dataset, it has significant limitations when processing large-scale, multi-field corpora. Text from different fields may have little semantic similarity, and mixing them together for deduplication results in a significant amount of meaningless computation, requiring significant storage resources to maintain the intermediate files. Summary of the Invention

[0005] The present application provides a deduplication processing method, a deduplication processing system and a storage medium for text corpus to optimize the deduplication process of text corpus and save the computing resources and storage resources required for the deduplication process.

[0006] The present application provides a method for deduplicating a text corpus, comprising: randomly sampling the global text corpus to be deduplicated to obtain a subset of the text corpus to be deduplicated; dividing the subset of the text corpus to be deduplicated into multiple classification sets according to a preset text classification model; hierarchically clustering the text corpus in each classification set to obtain a hierarchical clustering structure; dividing the global text corpus to be deduplicated into multiple corpus buckets according to the hierarchical clustering structure; after deduplicating all corpus buckets within the bucket, performing global deduplication to obtain deduplicated text.

[0007] Optionally, according to a preset text classification model, the subset of text corpus to be deduplicated is divided into multiple classification sets, including: determining the attribute vector of each text corpus in the subset of text corpus to be deduplicated according to the preset text classification model; the attribute vector includes the confidence of the classification result on each classification dimension; and dividing the subset of text corpus to be deduplicated into multiple classification sets according to the attribute vector and preset rules.

[0008] Optionally, according to a preset text classification model, the attribute vector of each text corpus in the text corpus subset to be deduplicated is determined, including: according to a preset plurality of text classifiers, the text corpus in the text corpus subset to be deduplicated is classified, and the confidence level of the classification result of each text corpus on each classification dimension is obtained to form the attribute vector of each text corpus; wherein the text classifier corresponds one-to-one to the classification dimension; the text classifier includes at least one of a language classifier, a domain classifier, a subject classifier, a sentiment classifier, a writing style classifier, and a text carrier classifier.

[0009] Optionally, the multiple classification sets include a significant classification set and a fuzzy classification set, and the significant classification set is provided with at least one classification dimension; the subset of the text corpus to be deduplicated is divided into multiple classification sets according to the attribute vector and preset rules, including: if the attribute vector of the text corpus represents that the confidence of the classification result of the text corpus on at least one classification dimension is greater than or equal to the confidence threshold, then it is determined that the text corpus belongs to the significant classification set; if the attribute vector of the text corpus represents that the confidence of the classification result of the text corpus on all classification dimensions is less than the confidence threshold, then it is determined that the text corpus belongs to the fuzzy classification set.

[0010] Optionally, the multiple classification sets include multiple specific significant classification sets, at least one comprehensive significant classification set and at least one non-specific significant classification set; wherein, each specific significant classification set corresponds to at least one classification dimension; if the confidence of the classification result of the text corpus represented by the attribute vector of the text corpus on at least one classification dimension is greater than or equal to the confidence threshold, then the text corpus is determined to belong to the significant classification set, including: if the confidence of the classification result of the text corpus represented by the attribute vector of the text corpus on the classification dimension corresponding to a specific significant classification set is greater than or equal to the confidence threshold, then the text corpus is determined to belong to the specific significant classification set that meets the requirements; if the confidence of the classification result of the text corpus represented by the attribute vector of the text corpus on the classification dimension corresponding to multiple specific significant classification sets is greater than or equal to the confidence threshold, then the text corpus is determined to belong to the comprehensive significant classification set; if the confidence of the classification result of the text corpus represented by the attribute vector of the text corpus on the classification dimension corresponding to any specific significant classification set cannot be greater than or equal to the confidence threshold at the same time, then the text corpus is determined to belong to the non-specific significant classification set.

[0011] Optionally, hierarchical clustering is performed on the text corpora in each classification set to obtain a hierarchical clustering structure, including: determining the semantic vectors of the text corpora in each classification set based on a semantic vector model; hierarchically clustering each classification set according to a clustering model based on the semantic vectors of each text corpus to form multiple clusters to obtain a hierarchical clustering structure; wherein the hierarchical clustering structure includes dependency relationships between multiple clusters.

[0012] Optionally, the global text corpus to be deduplicated is divided into multiple corpus buckets according to the hierarchical clustering structure, including: marking the global text corpus to be deduplicated with cluster numbers according to the hierarchical clustering structure; and dividing the global text corpus to be deduplicated into different corpus buckets according to the cluster numbers marked on each text corpus.

[0013] Optionally, deduplication is performed within all corpus buckets, including: calculating the perplexity of each text corpus in the cluster; using a deduplication algorithm to determine the similarity between the text corpora in each corpus bucket; deduplication is performed on the text corpus group whose similarity exceeds a similarity threshold, so as to retain the text corpus with the lowest perplexity in the text corpus group and delete the other text corpora in the text corpus group.

[0014] Optionally, the deduplication method also includes: calculating the centroid of the attribute vectors of all text corpora in each cluster to obtain the attribute centroid vector of each cluster; performing global deduplication, including: if the attribute centroid vector of the cluster in the corpus bucket does not have any intersection with the attribute centroid vectors of other clusters in the world, then determining that the global deduplication of the corpus bucket is completed; if the attribute centroid vector of the cluster in the corpus bucket has an intersection with the attribute centroid vectors of other clusters, then performing inter-bucket deduplication on the corpus buckets with the intersection.

[0015] Optionally, the corpus buckets with intersection include a first corpus bucket and a second corpus bucket; deduplication is performed between the corpus buckets with intersection, including: calculating the attribute vector of each text corpus in the first corpus bucket; calculating the perplexity of each text corpus in the second corpus bucket, and the distribution parameters of the attribute vectors of all text corpora; if the attribute vector of the text corpus in the first corpus bucket meets the set distribution parameter range of the attribute vector in the second corpus bucket, deduplication is performed based on the similarity between the semantic vector of the text corpus in the first corpus bucket and the semantic vector of the text corpus in the second corpus bucket.

[0016] Optionally, deduplication is performed based on the similarity between the semantic vector of the text corpus and the semantic vector of the text corpus in the second corpus bucket, including: determining the text corpus in the second corpus bucket whose similarity with the semantic vector of the text corpus in the first corpus bucket is greater than a set similarity threshold; retaining the text corpus with the lowest perplexity among the text corpora whose similarity with the semantic vector of the text corpus in the first corpus bucket is greater than the set similarity threshold into the second corpus bucket, and deleting the text corpus in the first corpus bucket.

[0017] Optionally, the deduplication method also includes: when the text corpus is increased, the newly added incremental text corpus is divided into corpus buckets according to the hierarchical clustering structure; if the attribute vector of the incremental text corpus is an outlier relative to the attribute vector distribution of the corpus bucket, the incremental text corpus is retained; if the attribute vector of the incremental text corpus is not an outlier relative to the attribute vector distribution of the corpus bucket, deduplication is performed based on the similarity between the incremental text corpus and the text corpus in the corpus bucket to which it belongs.

[0018] The present application provides a text corpus deduplication processing system, including one or more processors for implementing the aforementioned text corpus deduplication method.

[0019] The present application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for deduplicating text corpus.

[0020] The deduplication method, deduplication processing system and computer-readable storage medium of the text corpus provided by the present application first obtain a subset of the text corpus to be deduplicated by random sampling to reduce the amount of data in some processing processes. Then, based on the subset of the text corpus to be deduplicated, the text corpus in the subset of the text corpus to be deduplicated is divided into multiple classification sets according to a preset classification model, and texts in different fields are processed separately to avoid meaningless comparisons across fields. The deduplication range is further refined through hierarchical clustering and corpus bucket division processes to reduce the amount of calculation. Finally, deduplication is performed within the bucket first, and then global deduplication is performed, thereby further improving the efficiency of deduplication. In this way, the optimization of the deduplication process of the text corpus is achieved, which is conducive to saving the computing resources and storage resources required for the deduplication process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flowchart of a method for deduplicating text corpus provided by an embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of sampling results of a global text corpus to be deduplicated provided by an embodiment of the present application;

[0023] Figure 3 1 is a flowchart of a method for deduplicating text corpus provided in another embodiment of the present application;

[0024] Figure 4 This is a schematic diagram of the classification of a subset of text corpora to be deduplicated provided by an embodiment of the present application;

[0025] Figure 5 This is a partial schematic diagram of hierarchical clustering provided by an embodiment of the present application;

[0026] Figure 6This is an overall schematic diagram of hierarchical clustering provided by an embodiment of the present application;

[0027] Figure 7 1 is a flowchart of a method for deduplicating text corpus provided in another embodiment of the present application;

[0028] Figure 8-1 is a schematic diagram of the intersection between two corpus buckets provided by an embodiment of the present application;

[0029] Figure 8-2 is a schematic diagram of the intersection between two corpus buckets provided by another embodiment of the present application;

[0030] Figure 9 1 is a flowchart of a method for deduplicating text corpus provided in another embodiment of the present application;

[0031] Figure 10 This is a schematic diagram of deduplication of incremental text corpus provided by an embodiment of the present application.

[0032] Reference numerals

[0033] 10: Global text corpus to be deduplicated; 101: Subset of text corpus to be deduplicated; 102: Remaining text corpus to be deduplicated; 201: First significant classification set; 202: Second significant classification set; 203: Third significant classification set; 204: Fourth significant classification set; 210: Fuzzy classification set; 301: First vector set; 302: First cluster; 303: Second cluster; 304: Third cluster; 305: Fourth cluster; 401: Incremental text corpus. DETAILED DESCRIPTION

[0034] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings.

[0035] Combine Figure 1 As shown, an embodiment of the present application provides a method for deduplicating text corpus, including steps S10 to S50.

[0036] Step S10: Randomly sample the global text corpus to be deduplicated to obtain a subset of the text corpus to be deduplicated.

[0037] like Figure 2 As shown, the global text corpus to be deduplicated 10 is divided into the text corpus subset to be deduplicated 101 and the remaining text corpus to be deduplicated 102 .

[0038] Text corpus can be web page text, conversations, papers, books, news, computer code, etc.

[0039] Step S20: dividing the text corpus subset to be deduplicated into multiple classification sets according to a preset text classification model.

[0040] Step S30: performing hierarchical clustering on the text corpora in each classification set to obtain a hierarchical clustering structure.

[0041] Step S40: dividing the global text corpus to be deduplicated into multiple corpus buckets according to the hierarchical clustering structure.

[0042] Step S50: After deduplication is performed on all corpus buckets, global deduplication is performed to obtain deduplicated text.

[0043] Using the text deduplication method provided in the embodiment of the present application, firstly, a subset of the text corpus to be deduplicated is obtained by random sampling to reduce the amount of data in some processing processes. Then, based on the subset of the text corpus to be deduplicated, the text corpus in the subset of the text corpus to be deduplicated is divided into multiple classification sets according to a preset classification model, and texts in different fields are processed separately to avoid meaningless comparisons across fields. The deduplication range is further refined through the hierarchical clustering and corpus bucket division process to reduce the amount of calculation. Finally, deduplication is performed within the bucket first, and then global deduplication is performed, thereby further improving the efficiency of deduplication. In this way, the deduplication process of the text corpus is optimized, which is conducive to saving the computing resources and storage resources required for the deduplication process.

[0044] Combine Figure 3 As shown, in some embodiments, the aforementioned step S20, according to a preset text classification model, divides the subset of the text corpus to be deduplicated into multiple classification sets, including steps S201 to S202.

[0045] Step S201 : determining the attribute vector of each text corpus in the text corpus subset to be deduplicated according to a preset text classification model; the attribute vector includes the confidence of the classification result in each classification dimension.

[0046] Step S202 : dividing the text corpus subset to be deduplicated into multiple classification sets according to the attribute vector and preset rules.

[0047] In this way, dividing the deduplication text corpus subsets according to the attribute vector and preset rules not only improves the efficiency of text processing, but also provides a clear classification basis for subsequent deduplication operations, which is conducive to improving the overall performance of the text corpus deduplication process.

[0048] Specifically, in some embodiments, the attribute vector of each text corpus in the subset of text corpora to be deduplicated is determined based on a preset text classification model, including: classifying the text corpus in the subset of text corpora to be deduplicated based on a plurality of preset text classifiers, obtaining the classification result confidence of each text corpus on each classification dimension, and forming the attribute vector of each text corpus. Among them, the text classifier corresponds to the classification dimension one by one. And the text classifier includes at least one of a language classifier, a domain classifier, a subject classifier, a sentiment classifier, a writing style classifier, and a text carrier classifier. In this way, the classification result confidence of a text corpus on each classification temperature can be determined based on the attribute vector of the text corpus, so as to facilitate the subsequent classification process.

[0049] Specifically, the language classifier is used to classify the language used in each text corpus, such as English, Chinese, Spanish, Polish, etc. The field classifier is used to classify the sub-fields to which the content described by the text corpus belongs within the scope of the knowledge base. The subject classifier is used to classify the subject to which the content described by the text corpus belongs within the scope of the knowledge base. The sentiment classifier is used to classify the sentiment in each text corpus, such as positive sentiment, negative sentiment, etc. The writing style classifier is used to classify the writing style of each text corpus, such as news, technology, spoken language, stories, advertisements, etc. The text carrier classifier is used to classify the carrier of each text corpus, such as web pages, papers, news, etc.

[0050] The output of a text classifier is a probability-based confidence score, such as 0.6 or 0.7. The classification dimensions corresponding to each text classifier and the confidence scores of each text classifier form the attribute vector for each text corpus. This allows for the determination of the text corpus's performance on each classification dimension based on these numerical values.

[0051] It's understandable that similar text corpora must have similar attribute vectors. If two text corpora differ significantly in any categorical dimension of their attribute vectors, then they must be unique. It's important to note that while calculating attribute vectors for text corpora using a text classifier consumes a certain amount of computing resources, this process saves more computing resources for subsequent deduplication of text corpora.

[0052] After classification, the subset of the text corpus to be deduplicated is divided into multiple classification sets. Specifically, in some embodiments, the multiple classification sets include a significant classification set and a fuzzy classification set. Among them, the significant classification set is provided with at least one classification dimension. According to the attribute vector, the subset of the text corpus to be deduplicated is divided into multiple classification sets according to preset rules, including: if the confidence level of the classification result of the text corpus represented by the attribute vector of the text corpus on at least one classification dimension is greater than or equal to the confidence threshold, then it is determined that the text corpus belongs to the significant classification set. If the confidence level of the classification result of the text corpus represented by the attribute vector of the text corpus on all classification dimensions is less than the confidence threshold, then it is determined that the text corpus belongs to the fuzzy classification set. In this way, by dividing the subset of the text corpus to be deduplicated into a significant classification set and a fuzzy classification set, it is possible to efficiently screen out text corpus with clear characteristics, and at the same time it is conducive to simplifying the subsequent processing process. Not only is the efficiency of classification and deduplication improved, but it also has strong adaptability and is suitable for a variety of text processing scenarios.

[0053] If the confidence level of a text corpus's classification result on a classification dimension is greater than or equal to a confidence threshold, such as 0.7, then the text corpus's attribute on that classification dimension is considered significant. If a text corpus's attribute on at least one classification dimension is significant, then the text corpus is considered significant and belongs to the significant classification set. If a text corpus is not significant on all classification dimensions, then the text corpus belongs to the fuzzy classification set.

[0054] In some embodiments, the aforementioned multiple classification sets include multiple specific significant classification sets, at least one comprehensive significant classification set, and at least one non-specific significant classification set. Each specific significant classification set corresponds to at least one classification dimension. If the confidence level of the classification result of the text corpus represented by the attribute vector of the text corpus on at least one classification dimension is greater than or equal to a confidence threshold, then the text corpus is determined to belong to a significant classification set, including: if the confidence level of the classification result of the text corpus represented by the attribute vector of the text corpus on the classification dimension corresponding to a specific significant classification set is greater than or equal to the confidence threshold, then the text corpus is determined to belong to the specific significant classification set that corresponds to it. If the confidence level of the classification result of the text corpus represented by the attribute vector of the text corpus on the classification dimension corresponding to multiple specific significant classification sets is greater than or equal to the confidence threshold, then the text corpus is determined to belong to a comprehensive significant classification set. If the confidence level of the classification result of the text corpus represented by the attribute vector of the text corpus on the classification dimension corresponding to any specific significant classification set is not greater than or equal to the confidence threshold at the same time, then the text corpus is determined to belong to a non-specific significant classification set. In this way, it can ensure that text corpora with significant attributes can be classified into and only into one significant classification set, avoiding repeated classification and avoiding missing text corpora.

[0055] Specifically, taking into account the situation that a text corpus may meet the classification conditions of multiple significant classification sets during the implementation process, a comprehensive significant classification set is set separately in addition to multiple specific significant classification sets to store text corpora of this type. In this way, it is possible to avoid the situation where the same text corpus is classified into multiple significant classification sets, which increases the computational burden. Furthermore, according to actual needs, the comprehensive significant classification set can be set to one or more. When set to one, all text corpora that meet the classification conditions of multiple significant classification sets are divided into the comprehensive significant classification set. When set to multiple, further detailed division can be performed according to actual conditions. For example, text corpora that meet the classification conditions of two significant classification sets at the same time are divided into one comprehensive significant classification set, and text corpora that meet the classification conditions of more than three significant classification sets at the same time are divided into another comprehensive significant classification set. It can be set according to specific needs and will not be described in detail here.

[0056] That is, if the significant classification dimension of a text corpus is consistent with all the classification dimensions corresponding to a specific significant classification set, then the text corpus is classified into the specific significant classification set. For example, a specific significant classification set related to the Chinese e-commerce field is preset, and texts with a confidence level greater than 0.7 for the language dimension of Chinese classification and the style dimension of e-commerce are classified into this category. If the significant classification dimension of a text corpus is consistent with all the classification dimensions corresponding to multiple specific significant classification sets, then the text corpus is classified into the comprehensive significant classification set. If a text corpus has significant attributes, but its significant classification dimension does not conform to the classification dimension of any specific significant classification set, and does not belong to any specific specific significant classification set, then the text corpus is classified into another significant classification set, that is, a non-specific significant classification set.

[0057] like Figure 4 As shown, through the above classification process, all text corpora are divided into multiple classification sets, for example, into a first significant classification set 201, a second significant classification set 202, a third significant classification set 203, a fourth significant classification set 204 and a fuzzy classification set 210.

[0058] In some embodiments, hierarchical clustering is performed on the text corpora in each classification set to obtain a hierarchical cluster structure, including: determining semantic vectors of the text corpora in each classification set according to a semantic vector model. Based on the semantic vectors of each text corpus, hierarchical clustering is performed on each classification set according to the clustering model to form multiple clusters to obtain a hierarchical cluster structure. The hierarchical cluster structure includes dependency relationships between the multiple clusters.

[0059] Here, a pre-trained semantic vector model is used to calculate the semantic vector for the text corpus in each significant classification set and fuzzy classification set. The semantic vector models used in different classification sets may be different. Specifically, some fields of in-depth research often have more professional semantic vector models. At least part of the specific significant classification sets set here corresponds to the professional semantic vector models related to the field. For the remaining significant classification sets that do not have professional semantic vector models, a general semantic vector model is used. The fuzzy classification set also uses a general semantic vector model. In this way, high-precision semantic understanding in specific fields is guaranteed, and the wide applicability of the general model simplifies the processing flow, improves the overall processing efficiency, and enhances the system's adaptability to different fields and languages.

[0060] After determining the semantic vectors for each text corpus, a clustering algorithm is used to perform hierarchical clustering on each classification set, targeting its semantic vector. Finally, a clustering model tree structure is determined. The clustering model tree includes multiple clusters and the dependencies between clusters. Clustering can be performed based on the distance between cluster centroids in vector space.

[0061] For a significant classification set, if the clusters obtained in the clustering process simultaneously meet the two conditions that the text corpus capacity within the cluster is less than the preset capacity threshold and the attribute vectors within the cluster converge within the preset threshold range in each dimension, then the clustering of the significant classification set is determined to be finished. Otherwise, clustering continues until these two conditions are met, or the clustering level exceeds the preset level. It should be noted that in the real-time process, the preset capacity threshold and the preset threshold range are pre-set to appropriate numbers to prevent excessive clustering. Among them, the expected value and standard deviation of the attribute vector in each cluster are calculated to obtain its distribution parameters. The distribution of the attribute vector within the cluster can be marked in the form of "expected value ± standard deviation".

[0062] For the fuzzy classification set, if the clusters obtained in the clustering process meet the condition that the text corpus capacity within the clusters is less than a preset capacity threshold, it is determined that the clustering of the fuzzy classification set is completed.

[0063] In some embodiments, the number of categories K for each clustering is set to be an exponential number of the number of vectors. For example, if the number of vectors is N, then the number of categories for clustering is int[ln(N)+1].

[0064] by Figure 5For example, the process of hierarchical clustering of a significant text set is explained. First, the first significant classification set is vectorized to obtain a first vector set 301. The first vector set 301 is clustered to obtain multiple clusters. Among the multiple clusters obtained by this clustering, the text corpus capacity of the first cluster 302 is greater than the preset capacity threshold, and it needs to be clustered again. Although the second cluster 303 obtained by this clustering meets the convergence condition in terms of text corpus capacity, that is, it is less than the preset capacity threshold, the σ1 in the distribution (u1±σ1) of its attribute vector on some classification dimensions is too large and does not converge within the preset threshold range, so it is necessary to cluster the second cluster 303 again, and finally divide it into a third cluster 304 and a fourth cluster 305. The attribute distributions of the third cluster 304 and the fourth cluster 305 both meet the convergence condition, and the hierarchical clustering is completed.

[0065] In some embodiments, the global text corpus to be deduplicated is divided into multiple corpus buckets according to the hierarchical clustering structure, including: marking the global text corpus to be deduplicated with cluster numbers according to the hierarchical clustering structure; dividing the global text corpus to be deduplicated into different corpus buckets according to the cluster numbers marked on each text corpus. After completing the hierarchical clustering, the global text corpus to be deduplicated is clustered according to the existing hierarchical clustering structure. And all clusters are numbered, and each cluster is called a corpus bucket. Figure 6 As shown in Figure 1, the global text corpus to be deduplicated is divided into multiple corpus buckets. Thus, the entire hierarchical clustering cluster corpus bucket structure is constructed.

[0066] The global text corpus to be deduplicated is clustered according to the hierarchical clustering structure, the cluster numbers are marked, the semantic vector centroid and attribute vector distribution parameters are calculated for each cluster, and the global cluster semantic centroid vector table and the global cluster attribute vector distribution table are constructed.

[0067] Specifically, after constructing the hierarchical clustering corpus bucket structure, the centroid vector of each corpus bucket is obtained by averaging the semantic vectors in each corpus bucket, and the centroid vector is saved as a global clustering cluster semantic centroid vector table. The expected value and standard deviation of the attribute vector in each corpus bucket are calculated to obtain the distribution parameters of the attribute vector of the corresponding cluster cluster, and the distribution parameters of the attribute vectors of all corpus buckets are saved in the global clustering cluster attribute vector table. Table 1 illustrates the structure of a global clustering cluster attribute vector table. Among them, the distribution parameters of the attribute vector can be marked as "expected value ± standard deviation". During the implementation process, the specific technical implementation can use a separate storage method to save the above-mentioned vector table. In this way, it is convenient to call related data directly according to the table later. In some embodiments, before deduplicating the corpus bucket within the bucket and deduplicating globally, the global clustering cluster attribute vector distribution table must first be loaded into the memory.

[0068] Table 1:

[0069]

[0070] In some embodiments, all corpus buckets are deduplicated within the bucket, including: calculating the perplexity of each text corpus in the cluster; using a deduplication algorithm to determine the similarity between the text corpora in each corpus bucket; deduplicating the text corpus group whose similarity exceeds a similarity threshold, so as to retain the text corpus with the lowest perplexity in the text corpus group and delete the other text corpora in the text corpus group. During the implementation process, the text corpus is traversed to perform deduplication within the bucket for all corpus buckets. In some embodiments, the deduplication algorithm is a local sensitive hashing algorithm (LSH). Using the local sensitive hashing algorithm, a text corpus group with a similarity exceeding a similarity threshold is obtained, and the text corpus with the lowest perplexity is retained, and the others are deleted. The similarity threshold is set to 0.9, for example.

[0071] Furthermore, in at least some embodiments, intra-bucket deduplication is performed using parallel distributed computing to reduce overall computation time. Furthermore, after deduplication is complete, the locality-sensitive hash vector pool corresponding to each corpus bucket is persisted. This allows subsequent loads to bypass the hashing process and directly load the entire hash vector pool.

[0072] In some embodiments, the deduplication method further includes step S41, calculating the centroid of the attribute vectors of all text corpora in each cluster to obtain the attribute centroid vector of each cluster. On this basis, global deduplication is performed, including: if the attribute centroid vector of the cluster in the corpus bucket does not have an intersection with the attribute centroid vectors of other clusters in the world, then it is determined that the global deduplication of the corpus bucket is completed; if the attribute centroid vector of the cluster in the corpus bucket has an intersection with the attribute centroid vector of other clusters, then the corpus buckets with the intersection are deduplicated between buckets. Specifically, combined with Figure 7 As shown, an embodiment of the present application provides a method for deduplicating text corpus, including steps S10 to S502.

[0073] Step S10: Randomly sample the global text corpus to be deduplicated to obtain a subset of the text corpus to be deduplicated.

[0074] Step S20: dividing the text corpus subset to be deduplicated into multiple classification sets according to a preset text classification model.

[0075] Step S301: Determine the semantic vectors of the text corpora in each classification set according to the semantic vector model.

[0076] Step S302 : hierarchically clustering each classification set according to the semantic vector of each text corpus and the clustering model to form multiple clusters and obtain a hierarchical clustering structure.

[0077] The hierarchical clustering structure includes the dependency relationships between multiple clusters.

[0078] Step S401 : marking the global text corpus to be deduplicated with cluster numbers according to the hierarchical clustering structure.

[0079] Step S402 : Divide the global text corpus to be deduplicated into different corpus buckets according to the cluster numbers marked on the text corpus.

[0080] Step S41 , calculating the centroid of the attribute vectors of all text corpora in each cluster to obtain the attribute centroid vector of each cluster.

[0081] Step S501: Deduplication is performed on all corpus buckets.

[0082] Step S502: If the attribute centroid vector of the cluster in the corpus bucket does not intersect with the attribute centroid vectors of other clusters globally, it is determined that the global deduplication of the corpus bucket is completed; if the attribute centroid vector of the cluster in the corpus bucket does intersect with the attribute centroid vectors of other clusters, then inter-bucket deduplication is performed on the corpus buckets with the intersection.

[0083] Traverse any pair of corpus buckets and perform deduplication between them. The first step in deduplication is to determine whether the attribute vector distributions of the two corpus buckets intersect. If there is no intersection, then deduplication between the two buckets is not required.

[0084] Figure 8-1 and 8-2 It illustrates two situations of judging the intersection of attribute distributions between corpus buckets.

[0085] like Figure 8-1 As shown in the figure, the two corpus buckets have no intersection within the preset threshold range in the classification dimension of lang_en, that is, the language is English, and the classification dimension of sentiment_positve, that is, the text sentiment is positive. Therefore, these two corpus buckets do not require deduplication between buckets.

[0086] like Figure 8-2 As shown in the figure, although the attributes of all dimensions are slightly different, they do not exceed the preset threshold range, and are judged to have an intersection. In this case, these two corpus buckets need to participate in inter-bucket deduplication.

[0087] It should be noted that actual testing has shown that, under the hierarchical clustering corpus bucket structure described above, most inter-bucket deduplication is unnecessary, which helps save computing resources.

[0088] Taking the example of a corpus bucket with an intersection, including a first corpus bucket and a second corpus bucket, deduplication is performed on the inter-bucket corpus buckets, including: calculating the attribute vector of each text corpus in the first corpus bucket; calculating the perplexity of each text corpus in the second corpus bucket, as well as the distribution parameters of the attribute vectors of all text corpora; if the attribute vector of the text corpus in the first corpus bucket meets the set distribution parameter range of the attribute vector in the second corpus bucket, deduplication is performed based on the similarity between the semantic vector of the text corpus in the first corpus bucket and the semantic vector of the text corpus in the second corpus bucket. In this way, the accuracy and efficiency of deduplication are improved through the screening mechanism of distribution parameters, and the accidental deletion of important information is effectively avoided. And through the refined comparison of semantic similarity, duplicate or highly similar texts are effectively identified and removed, significantly improving the accuracy and efficiency of text deduplication.

[0089] Specifically, the text corpus in the first corpus bucket is labeled Ta, and the text corpus in the second corpus bucket is labeled Tb. Calculate whether Ta's attribute vector Va conforms to the set distribution parameter range of the attribute vector in the second corpus bucket. If it is not within the preset threshold range, Ta does not need to be subsequently compared with the text corpus in the second corpus bucket. If Ta's attribute vector Va conforms to the set distribution parameter range of the attribute vector in the second corpus bucket, that is, within the preset threshold range, Ta is compared with the text corpus in the second corpus bucket for deduplication.

[0090] Specifically, deduplication is performed based on the similarity between the semantic vectors of the text corpus and the semantic vectors of the text corpus in the second corpus bucket, including: determining the text corpus in the second corpus bucket whose semantic vector similarity with the text corpus in the first corpus bucket is greater than a set similarity threshold; retaining the text corpus with the lowest perplexity among the text corpus whose semantic vector similarity with the text corpus in the first corpus bucket is greater than the set similarity threshold, and deleting the text corpus in the first corpus bucket. Through the deduplication strategy based on semantic vector similarity, duplicate or highly similar text is effectively removed. On this basis, through perplexity comparison, the text corpus with the clearest and most accurate semantic expression is retained, which is conducive to improving the quality and usability of the deduplicated text corpus and providing a higher-quality data foundation for subsequent text processing and analysis. More specifically, Ta is compared with the second corpus bucket using a locality-sensitive hashing algorithm for deduplication. If Ta has a similarity with a text corpus in the second corpus bucket that is greater than the set similarity threshold, the text corpus with the lowest perplexity is retained in the second corpus bucket. After all corpus buckets are traversed and calculated, all remaining corpus buckets are the deduplicated corpus.

[0091] Combine Figure 9 As shown, the deduplication method further includes steps S601 to S603.

[0092] Step S601: When text corpus is increased, the newly added incremental text corpus is divided into corpus buckets according to the hierarchical clustering structure.

[0093] Step S602: If the distribution of the attribute vector of the incremental text corpus relative to the attribute vector of the corpus bucket is an outlier, the incremental text corpus is retained.

[0094] Step S603: If the distribution of the attribute vector of the incremental text corpus relative to the attribute vector of the corpus bucket is not an outlier, deduplication is performed based on the similarity between the incremental text corpus and the text corpus in the corpus bucket.

[0095] In this way, efficient and accurate deduplication operations are achieved by dynamically processing the newly added incremental text corpus.

[0096] Specifically, the newly added incremental text corpus is divided into corpus buckets according to the hierarchical clustering structure, including: determining the attribute vector of the incremental text corpus according to the preset text classification model. According to its attribute vector, the classification set to which the incremental text corpus belongs is determined, for example, by marking it with a classification number. According to the classification set to which the incremental text corpus belongs and the hierarchical clustering structure, the cluster cluster to which it belongs is determined, thereby achieving the division of the incremental text corpus into corpus buckets. Among them, the branch nodes of the cluster clusters are obtained according to the dependency relationship between the cluster clusters in the hierarchical clustering structure, and the specific cluster cluster is further determined to find the corpus bucket. For example, combined with Figure 10 As shown, for the incremental text corpus 401, the classification number of the classification set to which the incremental text corpus 401 belongs is determined to be C1 according to its attribute vector. On this basis, the branch node C of the cluster is obtained according to the dependency relationship between clusters in the hierarchical clustering structure. 21 , and further determine the specific cluster K g .

[0097] If the attribute vector of the incremental text corpus is an outlier relative to the attribute vector distribution of the corpus bucket to which it belongs, the incremental text corpus is determined to be non-duplicate text and is retained. If the attribute vector of the incremental text corpus is not an outlier relative to the attribute vector distribution of the corpus bucket to which it belongs, the incremental text corpus is considered to be a duplicate or highly similar text corpus and requires further deduplication. The attribute vector distribution can be obtained from the global cluster attribute vector distribution table based on the cluster number.

[0098] Deduplication is performed based on the similarity between the incremental text corpus and the text corpus in the corpus bucket to which it belongs, including: calculating the perplexity of the incremental text corpus. Based on the deduplication algorithm, a text corpus group whose similarity to the incremental text corpus is higher than a similarity threshold is determined, and the incremental text corpus and the text corpus with the lowest perplexity in the text corpus group are retained, and the other text corpuses are deleted. In this way, not only are duplicate or highly similar texts effectively removed, but the perplexity is also optimized to ensure that the text corpus retained in the corpus is the clearest and most accurate in semantic expression. In some embodiments, based on the cluster number of the incremental text corpus, a query is made to see whether the corresponding local sensitive hash pool vector table has been loaded into memory. If not, the hash pool vector table corresponding to the cluster number is loaded from persistent storage into memory to construct the corresponding local sensitive hash pool vector table. If the number of local sensitive hash pool vector tables loaded into memory exceeds a preset threshold, the hash pool vector table that has been used the least in the most recent set time is recycled. After the text corpus with the lowest perplexity is retained in the corpus bucket, the corresponding local sensitive hash pool vector table is updated.

[0099] The present application provides a text corpus deduplication processing system, comprising one or more processors for implementing the aforementioned text corpus deduplication method.

[0100] The present application also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for deduplicating text corpora. Here, the computer-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, and the like. For example, the computer-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as a hard disk drive), solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0101] In the description of this application, it should be understood that the terms "first," "second," etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first," "second," etc. may explicitly or implicitly include at least one of such features. In the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

Claims

1. A method for removing duplicate text corpus, characterized in that: include: Randomly sample the global text corpus to be deduplicated to obtain a subset of the text corpus to be deduplicated; Determining, according to a preset text classification model, an attribute vector of each text corpus in the subset of text corpora to be deduplicated; the attribute vector includes a confidence level of a classification result on each of a plurality of classification dimensions; Dividing the to-be-deduplicated text corpus subset into a plurality of classification sets according to the attribute vector and a preset rule; The method includes: if the confidence level of the classification result of the text corpus represented by the attribute vector of the text corpus on a classification dimension is greater than or equal to a confidence threshold, it indicates that the attribute of the text corpus on the classification dimension is significant; if the attribute vector of the text corpus represents that the attribute of the text corpus on at least one classification dimension is significant, it is determined that the text corpus belongs to a significant classification set; wherein there are multiple significant classification sets, and the text corpus with significant attributes is divided into and only into one significant classification set; if the confidence level of the classification result of the text corpus represented by the attribute vector of the text corpus on a classification dimension is less than a confidence threshold, it indicates that the attribute of the text corpus on the classification dimension is not significant; if the attribute vector of the text corpus represents that the attributes of the text corpus on all classification dimensions are not significant, it is determined that the text corpus belongs to a fuzzy classification set; Perform hierarchical clustering on the text corpus in each classification set to obtain a hierarchical clustering structure; Dividing the global text corpus to be deduplicated into multiple corpus buckets according to the hierarchical clustering structure; After deduplication is performed on all corpus buckets, global deduplication is performed to obtain the deduplicated text.

2. The deduplication method according to claim 1, characterized in that: The step of determining the attribute vector of each text corpus in the subset of text corpora to be deduplicated based on a preset text classification model includes: Classifying the text corpora in the to-be-deduplicated text corpus subset according to a plurality of preset text classifiers, obtaining the classification result confidence of each text corpus on each classification dimension, and forming an attribute vector of each text corpus; The text classifier corresponds to the classification dimension one by one; the text classifier includes at least one of a language classifier, a field classifier, a subject classifier, a sentiment classifier, a writing style classifier, and a text carrier classifier.

3. The deduplication method according to claim 1, wherein: The multiple classification sets include multiple specific significant classification sets, at least one comprehensive significant classification set, and at least one non-specific significant classification set; wherein each specific significant classification set corresponds to at least one classification dimension; If the attribute vector of the text corpus represents that the confidence of the classification result of the text corpus in at least one classification dimension is greater than or equal to a confidence threshold, then determining that the text corpus belongs to a significant classification set includes: If the attribute vector of the text corpus represents that the confidence of the classification result of the text corpus on the classification dimension corresponding to a specific significant classification set is greater than or equal to the confidence threshold, then it is determined that the text corpus belongs to the specific significant classification set that is met; If the attribute vector of the text corpus represents that the confidence of the classification result of the text corpus on the classification dimensions corresponding to multiple specific significant classification sets is greater than or equal to the confidence threshold, then it is determined that the text corpus belongs to the comprehensive significant classification set; If the attribute vector of the text corpus represents that the classification result confidence of the text corpus on the classification dimension corresponding to any specific significant classification set cannot be greater than or equal to the confidence threshold at the same time, then it is determined that the text corpus belongs to a non-specific significant classification set.

4. The deduplication method according to claim 1, wherein: The hierarchical clustering of the text corpora in each classification set to obtain a hierarchical clustering structure includes: Determine the semantic vectors of the text corpora in each classification set according to the semantic vector model; According to the semantic vectors of each text corpus, each classification set is hierarchically clustered according to the clustering model to form multiple clusters, thereby obtaining a hierarchical clustering structure; wherein the hierarchical clustering structure includes dependency relationships between the multiple clusters.

5. The deduplication method according to claim 4, characterized in that: The step of dividing the global text corpus to be deduplicated into multiple corpus buckets according to the hierarchical clustering structure includes: Marking the global text corpus to be deduplicated with cluster numbers according to the hierarchical clustering structure; The global text corpus to be deduplicated is divided into different corpus buckets according to the cluster numbers marked on the text corpora.

6. The deduplication method according to claim 4, characterized in that: Perform deduplication on all corpus buckets, including: Calculate the perplexity of each text corpus in the cluster; Use deduplication algorithms to determine the similarity between text corpora in each corpus bucket; For the text corpus group whose similarity exceeds the similarity threshold, duplicate removal is performed to retain the text corpus with the lowest perplexity in the text corpus group, and delete the other text corpora in the text corpus group.

7. The deduplication method according to claim 4, characterized in that: The deduplication method further includes: Calculate the centroid of the attribute vectors of all text corpora in each cluster to obtain the attribute centroid vector of each cluster; The global deduplication process includes: If the attribute centroid vector of the cluster in the corpus bucket does not intersect with the attribute centroid vectors of other clusters globally, it is determined that the global deduplication of the corpus bucket is completed; If the attribute centroid vector of a cluster in a corpus bucket intersects with the attribute centroid vectors of other clusters, the corpus buckets with the intersection are deduplicated between buckets.

8. The deduplication method according to claim 7, characterized in that: The corpus buckets with intersection include a first corpus bucket and a second corpus bucket; Deduplication between corpus buckets with intersections includes: Calculate the attribute vector of each text corpus in the first corpus bucket; Calculate the perplexity of each text corpus in the second corpus bucket and the distribution parameters of the attribute vectors of all text corpora; If the attribute vector of the text corpus in the first corpus bucket meets the set distribution parameter range of the attribute vector in the second corpus bucket, deduplication is performed based on the similarity between the semantic vector of the text corpus in the first corpus bucket and the semantic vector of the text corpus in the second corpus bucket.

9. The deduplication method according to claim 8, characterized in that: The performing duplicate removal based on the similarity between the semantic vectors of the text corpus in the first corpus bucket and the semantic vectors of the text corpus in the second corpus bucket includes: Determining text corpora in the second corpus bucket whose semantic vector similarity with the text corpora in the first corpus bucket is greater than a set similarity threshold; The text corpus with the lowest perplexity among the text corpora whose semantic vector similarity with the text corpus in the first corpus bucket is greater than the set similarity threshold is retained in the second corpus bucket, and the text corpus in the first corpus bucket is deleted.

10. The deduplication method according to claim 1, characterized in that: The deduplication method further includes: When the text corpus increases, the newly added incremental text corpus is divided into corpus buckets according to the hierarchical clustering structure; If the attribute vector of the incremental text corpus is an outlier relative to the attribute vector distribution of the corpus bucket, retaining the incremental text corpus; If the attribute vector of the incremental text corpus is not an outlier relative to the attribute vector distribution of the corpus bucket, deduplication is performed based on the similarity between the incremental text corpus and the text corpus in the corpus bucket to which it belongs.

11. A text corpus deduplication processing system, characterized in that: The method comprises one or more processors for implementing the method for deduplicating text corpus according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the method for removing duplicate text corpus according to any one of claims 1 to 10 is implemented.

Images