Text deduplication method, apparatus, device, storage medium, and program product

Abstract

Embodiments of the present application provide a text deduplication method, device, equipment, storage medium and program product, relating to the field of financial technology or other related fields. The method comprises: performing sentence segmentation processing on first text data to obtain a plurality of corresponding sentences; determining a fuzzy hash value corresponding to each sentence in the plurality of sentences by using a fuzzy hash algorithm; in the case that the similarity between the fuzzy hash value corresponding to a first sentence and each fuzzy hash value in a global fuzzy hash set is greater than a preset first threshold, determining that the first sentence is a duplicate sentence, the first sentence being any one of the plurality of sentences; removing the duplicate sentence in the first text data to obtain deduplicated first text data. The method provided by the present application improves the accuracy and efficiency of text deduplication.

Description

Technical Field

[0001] This application relates to the field of financial technology or other related fields, and in particular to a text deduplication method, apparatus, device, storage medium and program product. Background Technology

[0002] In the training of large-scale language models, data quality directly determines the model's performance and generalization ability. With the explosive growth of internet data, the sources of training corpora are becoming increasingly diverse, including public datasets, web page content crawled by web crawlers, and internal enterprise document repositories.

[0003] However, these data sources commonly suffer from text duplication issues. For example, the same news content may be reprinted by multiple websites, code snippets may appear repeatedly in different projects, or semantically similar but differently expressed text may be generated through synonym replacement, punctuation modification, etc. This duplicate data not only wastes storage resources but also leads to overfitting or noise interference during model training, severely affecting model performance. Training data typically reaches TB / PB levels, and traditional deduplication methods struggle to meet practical needs in terms of processing efficiency, accuracy, and resource consumption.

[0004] Therefore, a precise and efficient method for text deduplication is needed. Summary of the Invention

[0005] This application provides a text deduplication method, apparatus, device, storage medium, and program product to solve the technical problems of low accuracy and efficiency in text deduplication.

[0006] In a first aspect, this application provides a text deduplication method, comprising: performing sentence segmentation processing on first text data to obtain multiple corresponding sentences; determining the fuzzy hash value corresponding to each sentence in the multiple sentences using a fuzzy hash algorithm; determining that the first sentence is a duplicate sentence and that the first sentence is any one of the multiple sentences if the similarity between the fuzzy hash value corresponding to the first sentence and each fuzzy hash value in the global fuzzy hash set is greater than a preset first threshold; and removing the duplicate sentences from the first text data to obtain the deduplicated first text data.

[0007] In one possible implementation of the first aspect, the method further includes: determining the number of repeated statements among the multiple statements included in the target line text, wherein the target line text is any line text among the multiple lines of text included in the first text data; determining the target line text as a repeated line text if the ratio of the number of repeated statements in the target line text to the total number of statements in the target line text is greater than a second threshold; correspondingly, removing repeated statements from the first text data to obtain deduplicated first text data, including: removing repeated statements and repeated line texts from the first text data to obtain deduplicated first text data.

[0008] In one possible implementation of the first aspect, determining the fuzzy hash value corresponding to each statement among multiple statements using a fuzzy hash algorithm includes: dividing the first statement into multiple word groups, where the first statement is any statement among the multiple statements; calculating multiple local hash values ​​for the multiple word groups using a locality-sensitive hash algorithm; and determining the fuzzy hash value of the first statement based on the multiple local hash values.

[0009] In one possible implementation of the first aspect, multiple local hash values ​​of multiple words are calculated using a locality-sensitive hashing algorithm, including: calculating the feature vector hash value of multiple words using the SimHash algorithm; or, calculating the random permutation hash value of multiple words using the MinHash algorithm; or, calculating the hash value of the triplet difference histogram of multiple words using the trend micro locality sensitive hashing (TLSH) algorithm.

[0010] In one possible implementation of the first aspect, the hash value of the triple difference histogram of multiple words is calculated using the TLSH algorithm, including: constructing local sensitivity histograms corresponding to multiple words based on the triple difference values ​​of multiple words; and performing quantization encoding on the local sensitivity histograms corresponding to multiple words to obtain the hash value of the triple difference histogram.

[0011] In one possible implementation of the first aspect, the length of the fuzzy hash value corresponding to each statement is a preset length.

[0012] In one possible implementation of the first aspect, the method further includes: dividing the original text data into multiple text data blocks, wherein the first text data is any one of the multiple text data blocks.

[0013] In one possible implementation of the first aspect, the first threshold and / or the second threshold are determined based on text information of the first text data, the text information including text type information and / or text content feature information.

[0014] Secondly, this application provides a text deduplication device, comprising:

[0015] The first segmentation module is used to perform sentence segmentation processing on the first text data to obtain multiple corresponding sentences;

[0016] The first determining module is used to determine the fuzzy hash value corresponding to each statement among multiple statements using a fuzzy hash algorithm;

[0017] The second determining module is used to determine that the first statement is a repeated statement and that the first statement is any statement among multiple statements when the similarity between the fuzzy hash value corresponding to the first statement and each fuzzy hash value in the global fuzzy hash set is greater than a preset first threshold.

[0018] The removal module is used to remove duplicate sentences from the first text data to obtain the deduplicated first text data.

[0019] Thirdly, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0020] The memory stores the instructions that the computer executes;

[0021] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0022] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0023] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects.

[0024] The text deduplication method provided in this application improves the accuracy, efficiency, and scalability of text deduplication by combining a fuzzy hash algorithm with a threshold mechanism. The locality-sensitive property of the fuzzy hash algorithm maps semantically similar text to similar hash values, significantly outperforming traditional exact matching and substring matching methods. Furthermore, the fuzzy hash algorithm has low time complexity and only requires storing fixed-length hash values, resulting in low memory consumption. While ensuring semantic recognition accuracy, it significantly reduces computational costs and resource consumption, resolving the core contradiction of existing technologies in large-scale text deduplication scenarios. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] Figure 1 A schematic diagram illustrating a scenario for a text deduplication method provided in an embodiment of this application;

[0027] Figure 2 A flowchart illustrating a text deduplication method provided in an embodiment of this application;

[0028] Figure 3 This is a schematic diagram of the structure of a text deduplication device provided in an embodiment of this application;

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0030] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0033] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0034] It should be noted that the text deduplication method, apparatus, device, storage medium and product provided in this application can be used in financial technology and related fields, as well as in any field other than financial technology and related fields. The application fields of the text deduplication method, apparatus, device, storage medium and product in this application are not limited.

[0035] Figure 1 This is a schematic diagram illustrating a scenario where the text deduplication method of this application is applied. For example... Figure 1As shown, the customer can interact with the electronic device, which can execute the text deduplication method of this embodiment to provide interactive services to the user. The electronic device can also interact with the server to obtain the relevant data required to execute the text deduplication method of this embodiment.

[0036] In the training of large-scale language models, data quality directly determines the model's performance and generalization ability. With the explosive growth of internet data, the sources of training corpora are becoming increasingly diverse, including public datasets, web page content crawled by web crawlers, and internal enterprise document repositories.

[0037] However, these data sources commonly suffer from text duplication issues. For example, the same news content may be reprinted by multiple websites, code snippets may appear repeatedly in different projects, or semantically similar but differently expressed text may be generated through synonym replacement, punctuation modification, etc. This duplicate data not only wastes storage resources but also leads to overfitting or noise interference during model training, severely affecting model performance. Training data typically reaches TB / PB levels, and traditional deduplication methods struggle to meet practical needs in terms of processing efficiency, accuracy, and resource consumption.

[0038] Therefore, a precise and efficient method for text deduplication is needed.

[0039] The text deduplication method, apparatus, device, storage medium, and product provided in this application are intended to solve the above-mentioned technical problems of the prior art.

[0040] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0041] Figure 2 This is a flowchart illustrating a text deduplication method provided in an embodiment of this application, as shown below. Figure 2 As shown, the method includes:

[0042] S201. The electronic device performs sentence segmentation processing on the first text data to obtain multiple corresponding sentences.

[0043] For example, electronic devices can perform structured segmentation processing according to statement units to obtain multiple corresponding statements.

[0044] The first piece of text data is unstructured text to be processed, such as webpage content crawled by web crawlers or technical reports from a company's document library. For example, the input text could be: "Deep learning is the core technology of AI. Neural network models achieve feature extraction through multi-layer structures."

[0045] A statement unit refers to the smallest semantic unit defined based on natural language processing rules, such as a text segment separated by a period, question mark, or newline character. For example, a set of statements could be {"Deep learning is the core technology of AI.", "Neural network models achieve feature extraction through multi-layered structures."}.

[0046] S202. The electronic device uses a fuzzy hash algorithm to determine the fuzzy hash value corresponding to each statement among multiple statements.

[0047] This fuzzy hashing algorithm refers to an algorithm that can map semantically similar but expressively different texts to similar hash values, such as the Triple Locality Sensitive Hash (TLSH) algorithm. For example, the TLSH algorithm generates hash values ​​by constructing a feature histogram using the difference values ​​of triples.

[0048] S203. If the similarity between the fuzzy hash value corresponding to the first statement and each fuzzy hash value in the global fuzzy hash set is greater than a preset first threshold, the electronic device determines that the first statement is a duplicate statement, and the first statement is any statement among multiple statements.

[0049] Here, the similarity refers to hash similarity, that is, the hash value similarity between the fuzzy hash value corresponding to the first statement and each fuzzy hash value in the global fuzzy hash set.

[0050] For example, if the similarity between the fuzzy hash value corresponding to the first statement and a certain fuzzy hash value in the global fuzzy hash set is greater than the preset first threshold of 90%, then the first statement can be determined to be a duplicate statement.

[0051] This global fuzzy hash set is a pre-stored collection of hash values ​​of processed statements, which can be used to quickly determine duplicates. For example, if the global fuzzy hash set already contains the hash value "TLSH_12345...", and the current statement's hash value is "TLSH_12346...", their difference score is 85, which is less than the difference threshold of 90, so it is determined to be a duplicate.

[0052] For example, the first text data is first structured according to sentence units to ensure granular control in subsequent processing. The structured sentence set is used as input, and a hash value for each sentence is calculated using a fuzzy hashing algorithm (such as TLSH), generating a hash value set. Each hash value in the hash value set represents the semantic features of the corresponding sentence, and sentences with similar semantics but different expressions will generate similar hash values. In the subsequent judgment stage, by comparing the similarity with each fuzzy hash value in the global fuzzy hash set, it can be determined whether the sentence is duplicate content.

[0053] S204. The electronic device removes duplicate sentences from the first text data to obtain the deduplicated first text data.

[0054] After identifying duplicate statements in the first text data, the electronic device can remove the duplicate statements from the first text data to obtain the deduplicated first text data.

[0055] It is understood that the repeated statements in the first text data in this application embodiment are not only repeated with a statement in the first text data, but also repeated with a statement in the text data corresponding to the already established global fuzzy hash set. That is, it not only removes statements that are repeated with the content in the first text data, but also removes statements that are repeated with the previously input text data.

[0056] For example, the first text data (one line) is: "Artificial intelligence is the technology of the future. AI will shape the future."

[0057] Electronic devices can be divided into three sentences: s1: "Artificial intelligence is the technology of the future."; s2: "AI will shape the future."; s3: "Artificial intelligence will shape the future."

[0058] Then calculate their fuzzy hash values: the fuzzy hash value of s1 is h1, the fuzzy hash value of s2 is h2, and the fuzzy hash value of s3 is h3.

[0059] Assuming we have previously seen "Artificial intelligence will shape the future" (i.e., the fuzzy hash value h3 is in the library), and s1 and s3 are semantically highly similar (TLSH.diff < 90), then both s1 and s3 are considered duplicate statements. Statements s1 and s3 will be removed and will not be included in the final training data.

[0060] In this possible implementation, the locality-sensitive property of fuzzy hashing algorithms maps semantically similar but expressively different texts to similar hash values, thereby accurately identifying sentence-level semantic repetitions. For example, although "deep learning model" and "deep neural network model" are different expressions, their hash values ​​have a high similarity due to semantic similarity, and can be judged as repetitions. Compared to traditional exact hashing (such as MD5), which can only identify texts that are literally identical, this method solves the problem of insufficient semantic repetition recognition capability of existing technologies through semantically aware hash calculation. In addition, this method does not rely on high-dimensional semantic models (such as BERT embeddings), avoiding the high computational overhead of deep learning solutions, and achieving high-precision text deduplication with low resource consumption.

[0061] In some embodiments, the method further includes:

[0062] S205. The electronic device determines the number of repeated statements among the multiple statements included in the target line text, wherein the target line text is any line of text among the multiple lines of text included in the first text data.

[0063] That is, the electronic device can divide the first text data into multiple lines of text, and for the multiple lines of text included in the first text data, the electronic device can determine the number of repeated statements in the multiple statements included in each line of text.

[0064] The number of duplicate statements refers to the number of statements marked as duplicates in the hash similarity determination. For example, if 3 statements in a row are determined to be duplicates, then the number of duplicate statements is 3.

[0065] S206. If the ratio of the number of repeated statements in the target line text to the total number of statements in the target line text is greater than a second threshold, the electronic device determines that the target line text is a repeated line text.

[0066] In other words, electronic devices can determine whether a line of text is repeated by comparing the number of repeated sentences to the total number of sentences. If the ratio is greater than a second threshold, the line of text is considered repeated; if it is less than or equal to the second threshold, the line of text is not repeated.

[0067] Correspondingly, the electronic device removes duplicate sentences from the first text data to obtain deduplicated first text data, including:

[0068] The electronic device removes duplicate statements and duplicate lines from the first text data to obtain the deduplicated first text data. In other words, the electronic device can remove both duplicate statements and duplicate lines.

[0069] Understandably, if the electronic device determines that a line of text is a duplicate, it can remove all text from that line.

[0070] For example, after determining the hash similarity of statements, the electronic device can further count the number of repeated statements in each line of text and calculate its ratio to the total number of statements. For instance, if a line of text contains 5 statements, and 3 of them are determined to be repeated, then the ratio of repeated statements to the total number of statements in the line is R = 0.6. If R is greater than a preset second threshold, such as 0.5, then the line is marked as repeated. By using a line-level aggregation determination strategy, misjudgments caused by local repetitions are avoided, improving the overall rationality of the determination.

[0071] This possible implementation effectively addresses the fragmentation-based misjudgment problem caused by sentence-by-sentence matching by statistically analyzing the proportion of repeated sentences at the line level and combining this with a second threshold. For example, in news aggregation scenarios, the same news headline may be reprinted by multiple websites, but the body text may not be judged as duplicate due to differences in expression. In this case, determining the line-level ratio can prevent the complete news from being misjudged as non-duplicate content. This significantly improves the practicality of line-level judgment while maintaining low computational complexity.

[0072] In some embodiments, the electronic device determines the fuzzy hash value corresponding to each statement among multiple statements using a fuzzy hash algorithm, which may specifically include:

[0073] The electronic device can divide the first statement into multiple word groups, with the first statement being any one of the multiple statements; then, it can use a locality-sensitive hashing algorithm to calculate multiple local hash values ​​for the multiple word groups; and finally, it can determine the fuzzy hash value of the first statement based on the multiple local hash values.

[0074] For example, an electronic device can segment a first statement into multiple word groups, generate a local hash value for each word group using a locality-sensitive hashing algorithm, and combine the local hash values ​​into the statement's hash value through concatenation or XOR operations. For instance, after the statement "Deep learning models achieve feature extraction through neural networks" is split into a set of word groups, local hash values ​​"TLSH_12345..." and "TLSH_67890..." are generated respectively, and finally, the statement hash value "TLSH_12345...67890..." is generated by concatenation.

[0075] In this context, dividing the first statement into multiple phrases involves splitting the statement into phrases based on semantic or syntactic units. For example, the statement "Deep learning models achieve feature extraction through neural networks" can be broken down into a set of phrases: {"deep learning model", "neural network", "feature extraction"}.

[0076] Locality-Sensitive Hash (TLSH) algorithms generate hash values ​​by statistically analyzing the distribution of local features. For example, the TLSH algorithm generates the hash value "TLSH_12345..." for the phrase "deep learning model".

[0077] In this possible implementation, the semantic sensitivity of sentence hash values ​​is improved through phrase-level local hashing combinations. For example, after phrase-level splitting, the differences in the combinations of local hash values ​​between the sentences "deep learning model" and "deep neural network model" are more obvious, thereby reducing the problem of missed detections caused by local substitution. By refining the granularity of hash generation, the ability to recognize texts with similar semantics but different expressions is enhanced, while avoiding the high resource consumption of sentence-by-sentence full feature hashing calculations.

[0078] In some embodiments, the electronic device calculates multiple local hash values ​​for multiple phrases using a locality-sensitive hashing algorithm, which may specifically include:

[0079] Electronic devices calculate the feature vector hash values ​​of multiple word groups using the SimHash algorithm; or,

[0080] Calculate the hash value of a random permutation of multiple words using the MinHash algorithm; or,

[0081] The hash value of the triplet difference histogram of multiple words is calculated using the triplet locality-sensitive hashing (TLSH) algorithm.

[0082] Electronic devices can generate local hash values ​​for phrases using SimHash, MinHash, or TLSH algorithms. For example, the SimHash algorithm calculates the hash value of a feature vector, the MinHash algorithm calculates the hash value of a random permutation, and the TLSH algorithm calculates the hash value of a triplet difference histogram.

[0083] The SimHash algorithm calculates text similarity by hashing feature vectors. For example, the phrase "deep learning model" generates the SimHash value "SIM_12345...".

[0084] The MinHash algorithm calculates local similarity by randomly permuting hash values. For example, the phrase "neural network" generates the MinHash value "MIN_67890...".

[0085] This possible implementation allows for the calculation of local hash values ​​using various algorithms, thus increasing the flexibility of the implementation.

[0086] In some embodiments, the electronic device calculates the hash value of a triplet difference histogram of multiple phrases using the TLSH algorithm, which may specifically include:

[0087] The electronic device constructs local sensitivity histograms corresponding to multiple word groups based on the triple difference values ​​of multiple word groups; then, it performs quantization encoding on the local sensitivity histograms corresponding to multiple word groups to obtain the hash value of the triple difference histogram.

[0088] For example, an electronic device constructs a Locality Sensitive Histogram (LSH) by extracting the difference values ​​of triples in text. For instance, for the statement "Deep learning is the core technology of AI", it extracts triples such as "deep-degree-learning" and "degree-learning-is", calculates the difference values, and statistically analyzes their distribution. Subsequently, the histogram is quantized and encoded into a hash value (such as 128 bytes of TLSH).

[0089] The triplet difference value refers to the difference value of three consecutive characters or bytes extracted from the text, which is used to construct local features. For example, in the string "abc", the triplet is "abc", and the difference value is "ba,cb".

[0090] Locality-sensitive histograms (LSHs) are feature histograms generated by statistically analyzing the distribution of triplet difference values, used to characterize local features of text. For example, the TLSH algorithm quantifies the distribution of triplet difference values ​​using a 128-dimensional histogram.

[0091] In this possible implementation, constructing a locally sensitive histogram using triplet difference values ​​can more accurately capture the local semantic features of the text. For example, although "deep learning model" and "deep neural network model" are different expressions, they share triples such as "deep-learning-model," resulting in similar histogram distributions and high similarity in the generated hash values. This improves the ability of fuzzy hashing to identify semantic repetition.

[0092] In some embodiments, the length of the fuzzy hash value corresponding to each statement is a preset length. That is, when calculating the fuzzy hash value corresponding to each statement, the output of the fuzzy hash algorithm is a preset fixed length value, such as 70 bytes for TLSH.

[0093] For example, a fixed-length hash string generated through quantization encoding (such as 70 bytes for TLSH) is a hash value of fixed length. For example, the hash value generated by TLSH is "6F02A...C31D".

[0094] In this possible implementation, the length of the fuzzy hash value corresponding to each statement is a preset length. The fixed-length hash value of the fuzzy hash algorithm (such as 70 bytes of TLSH) significantly reduces memory usage, avoids the complexity of high-dimensional feature storage, achieves high-precision deduplication with low resource consumption, and supports real-time processing of ultra-large-scale corpora.

[0095] In some embodiments, the method further includes:

[0096] S207. The electronic device divides the original text data into multiple text data blocks, and the first text data is any one of the multiple text data blocks.

[0097] That is, electronic devices can divide the original text data into multiple text data blocks and allocate each text data block to distributed computing nodes for parallel processing. The first text data is any one of the multiple text data blocks.

[0098] For example, before sentence segmentation processing, electronic devices can divide the original text data into text data blocks and distribute them to distributed computing nodes. For instance, 10GB of text can be divided into 100 fragments, with each node processing one fragment independently.

[0099] Splitting text data into multiple blocks involves dividing the original text data into multiple subsets based on lines or paragraphs, which facilitates distributed processing. For example, 10GB of text can be divided into 100 100MB chunks.

[0100] Distributed computing nodes refer to computing resources that support parallel processing (such as worker nodes in MapReduce). For example, in a Hadoop cluster, each worker node processes a block of text data.

[0101] This possible implementation addresses the efficiency bottleneck of traditional methods in processing ultra-large-scale data through data partitioning and distributed computing. For example, in petabyte-scale corpus preprocessing scenarios, parallel processing through data partitioning keeps single-machine memory usage within a reasonable range, and processing time is significantly lower than centralized processing.

[0102] In some embodiments, the first threshold and / or the second threshold are determined based on text information of the first text data, the text information including text type information and / or text content feature information.

[0103] For example, an electronic device can calculate a context sensitivity index based on text type (such as news, code, technical documents) and content features (such as sentence length, punctuation density), and determine a first threshold and / or a second threshold based on the context sensitivity index.

[0104] For example, after splitting the statement (by period), we get: Statement 1: "Initialize database connection." and Statement 2: "Parameter host specifies the server address, port specifies the port number, default is 3306." The number of statements is 2, and the average statement length (number of characters) is approximately 14.5 characters. Regarding punctuation density (number of punctuation marks / total number of characters), punctuation: period × 2, comma × 2 → a total of 4; total number of characters (excluding spaces): 7 + 22 = 29 characters, so the punctuation density = 4 / 29 ≈ 0.138 (13.8%).

[0105] The weights for each feature are as follows: text type="code" = 0.5, text type="technical document" = 0.4, text type="news" = 0.2, text type="social media" = 0.1, average sentence length ≤ 10 characters = 0.2, punctuation density ≥ 0.2 (i.e., 20%) = 0.2, and inclusion of keywords (such as "parameters", "error codes", "interfaces") = 0.1. The calculated total score for the context sensitivity index is: 0.4 (type) + 0 (sentence length) + 0 (punctuation) + 0.1 (keywords) = 0.5.

[0106] Mapping Rule 1: First Threshold (Statement-level TLSH Similarity Threshold), Range: 70~110 (the larger the value, the easier it is to detect duplicates), Formula: First Threshold = 70 + (Sensitivity Index × 40) First Threshold = 70 + (Sensitivity Index × 40), Substituting 0.5: 70 + (0.5 × 40) = 70 + 20 = 90 70 + (0.5 × 40) = 70 + 20 = 90.

[0107] Mapping Rule 2: Second Threshold (Row-level Repetition Rate Threshold), Range: 0.3~0.7 (the smaller the value, the easier it is to remove the entire row), Formula: Second Threshold = 0.7 − (Sensitivity Index × 0.4). Substituting 0.5: 0.7 − (0.5 × 0.4) = 0.7 − 0.2 = 0.5.

[0108] Thus, the corresponding first threshold is determined to be 90, and the second threshold is determined to be 0.5.

[0109] This possible implementation dynamically adjusts the threshold using a context-sensitive indicator, solving the adaptability problem of traditional uniform thresholds in cross-domain scenarios. For example, it avoids false positives caused by punctuation changes in code text and false negatives caused by multiple similar paragraphs in news text. This improves the flexibility and rationality of the judgment strategy.

[0110] In some embodiments, the fuzzy hashing algorithm is the Triple Locality Sensitive Hash (TLSH) algorithm. The electronic device determines the similarity between the fuzzy hash value corresponding to the first statement and each fuzzy hash value in the global fuzzy hash set. This can be achieved by calculating the difference score between the current hash value and the hash values ​​in the pre-stored hash set using the TLSH.diff() function. This difference score can be understood as the inverse parameter of similarity; a larger difference score indicates lower similarity. If the difference score of two hash values ​​is less than or equal to a preset threshold, they are determined to be similar hash values.

[0111] For example, the electronic device calculates the difference score between the current hash value and the hash values ​​in the pre-stored hash set using the TLSH.diff() function. If the score is less than a preset threshold, the hash value is considered similar. For instance, the difference score between the current hash value "TLSH_12345..." and the pre-stored hash value "TLSH_12346..." is 85 (less than the threshold of 90), and the hash value is considered similar.

[0112] The TLSH.diff() function calculates the difference score between two TLSH hash values. For example, the TLSH.diff() difference score between the hash values ​​“TLSH_12345...” and “TLSH_12346...” is 85.

[0113] The preset threshold is a user-configured boundary for determining the difference score. For example, if the preset threshold is 90, hash values ​​with a difference score less than 90 are considered similar.

[0114] In this possible implementation, the quantification mechanism of the TLSH.diff() function enhances the flexibility of similarity judgment. For example, in news media reprint scenarios, the hash value difference score of some sentences in the same news article may be slightly higher than the threshold due to title adjustments. However, by dynamically adjusting the threshold (e.g., setting it to 85), false positives can be avoided. By quantifying the difference score, accurate recognition of semantically similar text is ensured, while also allowing users to adjust the judgment strictness according to scenario requirements.

[0115] In some embodiments, after receiving the first deduplicated text data, the electronic device can output the generated deduplicated text and duplicate location report to the storage system and integrate it into a real-time data pipeline through a streaming processing framework. For example, Spark Streaming reads the deduplicated text and writes it to HDFS in real time.

[0116] This possible implementation integrates real-time data pipelines through a streaming processing framework, solving the latency problem of traditional batch processing. For example, in real-time news feed scenarios, this technique supports second-level deduplication response, providing a cost-effective data cleaning solution for large-scale AI training.

[0117] In this embodiment, the combination of fuzzy hashing algorithm and threshold mechanism improves the accuracy, efficiency, and scalability of text deduplication, specifically including:

[0118] 1. Improved semantic-level deduplication accuracy: The locality sensitivity of the fuzzy hash algorithm can map semantically similar text to similar hash values, which is significantly better than traditional exact matching and substring matching methods;

[0119] 2. Optimized computational efficiency: The fuzzy hash algorithm has low time complexity and only needs to store a fixed-length hash value, resulting in low memory consumption and support for TB / PB level data processing;

[0120] 3. Dynamic judgment flexibility: The strictness of row-level repetition judgment is controlled by adjustable thresholds to avoid fragmented misjudgments and adapt to different scenario requirements;

[0121] 4. Engineering adaptability extension: Supports integration with streaming processing and distributed computing frameworks, enabling real-time data pipelines and providing efficient data cleaning solutions for large-scale AI training.

[0122] While ensuring semantic recognition accuracy, it significantly reduces computational costs and resource consumption, resolving the core contradiction of existing technologies in large-scale text deduplication scenarios.

[0123] Figure 3 This is a schematic diagram of the structure of a text deduplication device provided in an embodiment of this application, as shown below. Figure 3 As shown, the text deduplication device 300 provided in this embodiment includes:

[0124] The first segmentation module 301 is used to perform sentence segmentation processing on the first text data to obtain multiple corresponding sentences;

[0125] The first determining module 302 is used to determine the fuzzy hash value corresponding to each statement among multiple statements using a fuzzy hash algorithm;

[0126] The second determining module 303 is used to determine that the first statement is a repeated statement and that the first statement is any statement among multiple statements when the similarity between the fuzzy hash value corresponding to the first statement and each fuzzy hash value in the global fuzzy hash set is greater than a preset first threshold.

[0127] The removal module 304 is used to remove duplicate sentences from the first text data to obtain the deduplicated first text data.

[0128] In one possible implementation, the text deduplication device further includes:

[0129] The third determining module 305 is used to determine the number of repeated statements in the multiple statements included in the target line text, where the target line text is any line of text in the multiple lines of text included in the first text data;

[0130] The fourth determining module 306 is used to determine that the target line text is a repeating line text when the ratio of the number of repeated statements in the target line text to the total number of statements in the target line text is greater than a second threshold.

[0131] Correspondingly, the removal module 304 is also used to: remove duplicate statements and duplicate lines of text from the first text data to obtain the deduplicated first text data.

[0132] In one possible implementation, the first determining module 302 is specifically used to: divide the first statement into multiple word groups, wherein the first statement is any one of the multiple statements; calculate multiple local hash values ​​of the multiple word groups using a locality-sensitive hashing algorithm; and determine the fuzzy hash value of the first statement based on the multiple local hash values.

[0133] In one possible implementation, the first determining module 302 is specifically used to: calculate the feature vector hash value of multiple words using the SimHash algorithm; or, calculate the hash value of the random permutation of multiple words using the MinHash algorithm; or, calculate the hash value of the triplet difference histogram of multiple words using the Triple Local Sensitive Hash (TLSH) algorithm.

[0134] In one possible implementation, the first determining module 302 is specifically used to: construct local sensitivity histograms corresponding to multiple word groups based on the triplet difference values ​​of multiple word groups; and perform quantization encoding processing on the local sensitivity histograms corresponding to multiple word groups to obtain the hash value of the triplet difference histogram.

[0135] In one possible implementation, the length of the fuzzy hash value corresponding to each statement is a preset length.

[0136] In one possible implementation, the text deduplication device further includes:

[0137] The second segmentation module 307 is used to segment the original text data into multiple text data blocks, wherein the first text data is any one of the multiple text data blocks.

[0138] In one possible implementation, the first threshold and / or the second threshold are determined based on text information of the first text data, which includes text type information and / or text content feature information.

[0139] The text deduplication device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0140] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 may include a memory 401 and a processor 402. Optionally, the electronic device may also include a transceiver 403, wherein the memory 401 and the processor 402 communicate with each other; for example, the memory 401, the processor 402 and the transceiver 403 may communicate via a communication bus 404, the memory 401 is used to store a computer program, and the processor 402 executes the computer program to implement the method of the above embodiments.

[0141] Optionally, the processor mentioned above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps in the method embodiments disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0142] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0143] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0144] All or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0145] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0148] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0149] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0150] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0151] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0152] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0153] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or in the form of software program modules.

[0154] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0155] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0156] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0157] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0158] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A text deduplication method, characterized in that, The method includes: The first text data is processed into sentence segmentation to obtain multiple corresponding sentences; The fuzzy hash value corresponding to each of the multiple statements is determined by using a fuzzy hash algorithm; If the similarity between the fuzzy hash value corresponding to the first statement and each fuzzy hash value in the global fuzzy hash set is greater than a preset first threshold, the first statement is determined to be a duplicate statement, and the first statement is any one of the multiple statements. Remove duplicate sentences from the first text data to obtain the deduplicated first text data.

2. The method according to claim 1, characterized in that, The method further includes: Determine the number of repeated statements among the multiple statements included in the target line text, wherein the target line text is any line of text among the multiple lines of text included in the first text data; If the ratio of the number of repeated statements in the target line text to the total number of statements in the target line text is greater than a second threshold, the target line text is determined to be a repeated line text. Correspondingly, the step of removing duplicate statements from the first text data to obtain deduplicated first text data includes: Remove duplicate statements and duplicate lines from the first text data to obtain the deduplicated first text data.

3. The method according to claim 1 or 2, characterized in that, The fuzzy hash value for each of the plurality of statements is determined using a fuzzy hash algorithm, including: The first statement is divided into multiple word groups, and the first statement is any one of the multiple statements; Calculate multiple local hash values ​​for multiple word groups using the locality-sensitive hashing algorithm; The fuzzy hash value of the first statement is determined based on the plurality of local hash values.

4. The method according to claim 3, characterized in that, The calculation of multiple local hash values ​​for multiple word groups using the locality-sensitive hashing algorithm includes: The feature vector hash values ​​of the multiple word groups are calculated using the SimHash algorithm; or... The hash value of the random permutation of the multiple word groups is calculated using the MinHash algorithm; or, The hash value of the triplet difference histogram of the multiple phrases is calculated using the triplet locality-sensitive hashing (TLSH) algorithm.

5. The method according to claim 4, characterized in that, The step of calculating the hash value of the triplet difference histogram of the multiple phrases using the TLSH algorithm includes: Construct local sensitivity histograms corresponding to the multiple word groups based on the triplet difference values ​​of the multiple word groups; The local sensitivity histograms corresponding to the multiple word groups are quantized and encoded to obtain the hash value of the triplet difference histogram.

6. The method according to any one of claims 1-5, characterized in that, The length of the fuzzy hash value corresponding to each statement is a preset length.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: The original text data is divided into multiple text data blocks, and the first text data is any one of the multiple text data blocks.

8. The method according to any one of claims 1-5, characterized in that, The first threshold and / or the second threshold are determined based on the text information of the first text data, wherein the text information includes text type information and / or text content feature information.

9. A text deduplication device, characterized in that, The device includes: The first segmentation module is used to perform sentence segmentation processing on the first text data to obtain multiple corresponding sentences; The first determining module is used to determine the fuzzy hash value corresponding to each of the plurality of statements using a fuzzy hash algorithm; The second determining module is used to determine that the first statement is a repeated statement when the similarity between the fuzzy hash value corresponding to the first statement and each fuzzy hash value in the global fuzzy hash set is greater than a preset first threshold, and the first statement is any one of the plurality of statements. The removal module is used to remove duplicate sentences from the first text data to obtain the deduplicated first text data.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images