Density clustering and semantic verification-based Chinese and English mixed text deduplication method

Abstract

The invention discloses a Chinese and English mixed text deduplication method based on density clustering and semantic verification. The method comprises the following steps: converting each text into each semantic vector, and obtaining a first cluster based on a density clustering algorithm; performing iterative optimization on the first cluster based on a binary search framework in combination with semantic verification of the large model to obtain a cluster threshold value; updating the first cluster based on a cluster threshold to obtain a plurality of second clusters; and based on a preset semantic screening algorithm and a preset expression screening algorithm, de-duplicating each second cluster to obtain a de-duplicated text list. By means of the mode, the optimal clustering threshold value can be obtained through automatic iterative optimization, the clustering threshold value does not need to be set manually, Chinese and English mixed texts with different semantic densities can be adapted, and semantic consistency in the same cluster is achieved; through the preset semantic screening algorithm and the preset expression screening algorithm, semantic uniqueness and expression diversity can be reserved, and expression similarity misjudgment caused by language structure difference is avoided.

Description

Technical Field

[0001] This application relates to the field of natural language processing, and in particular to a method for deduplicating mixed Chinese and English text based on density clustering and semantic verification. Background Technology

[0002] With the explosive growth of internet text data, especially texts containing mixed Chinese and English expressions, such as technical Q&A, product descriptions, and user reviews, repetitive or highly similar texts not only consume redundant storage resources but also reduce the efficiency and accuracy of subsequent text analysis (such as intent recognition or knowledge graph construction).

[0003] However, existing text deduplication techniques for mixed Chinese and English text have the following shortcomings: Limitations of pure algorithm clustering: Traditional pure density clustering algorithms rely on manually setting distance thresholds, which are difficult to adapt to the semantic differences of mixed Chinese and English texts. Chinese is based on single characters, while English is based on letters or words. The expression of the same semantics is quite different. Fixed thresholds can easily lead to "over-clustering" (i.e. the same semantic is split into multiple clusters) or "under-clustering" (i.e. different semantics are merged into one cluster).

[0004] Limitations of pure large model methods: Although existing large models have relatively long context windows, in corpus deduplication tasks such as dataset construction, there is a demand for hundreds of millions of tokens, which makes large model methods unable to meet the deduplication work of the entire corpus. Even for deduplication tasks of very small corpora with context requirements controlled within the window range, there are still problems such as the unusability of deduplication results caused by context corruption of large models.

[0005] Imbalance between representational diversity and semantic consistency: Existing methods either retain only a single sample and lose representational diversity, or they cannot effectively filter texts that are semantically consistent but have repetitive representations (such as "How to use the API?" and "How to use the API?"), making it difficult to meet the business requirement of "retaining multiple representations and removing semantic repetitions".

[0006] Poor adaptability to mixed Chinese and English scenarios: Most deduplication methods are designed for pure Chinese or pure English text. For Chinese sentences containing English words (such as "API" and "user_id"), clustering bias is easily caused by differences in character structure, making it impossible to accurately judge the similarity of expressions. Summary of the Invention

[0007] This application provides a method for deduplication of mixed Chinese and English text based on density clustering and semantic verification, in order to solve the problems of difficulty in setting thresholds and imbalance between expression diversity and semantic consistency in existing deduplication technologies in mixed Chinese and English scenarios.

[0008] To address the aforementioned technical problems, this application adopts the following technical solution: a method for deduplicating mixed Chinese and English text based on density clustering and semantic verification. This method includes: Each text is converted into a semantic vector, and the first cluster is obtained based on the density clustering algorithm; Based on the binary search framework combined with the semantic verification of the large model, the first clustering is iteratively optimized to obtain the clustering threshold; The first cluster is updated based on the clustering threshold to obtain multiple second clusters; Based on the preset semantic filtering algorithm and the preset expression filtering algorithm, duplicates are removed from each of the second clusters to obtain a list of deduplicated texts.

[0009] In one optional embodiment of this application, the step of converting each text into a semantic vector and obtaining a first cluster based on a density clustering algorithm includes: Based on the text embedding model, each of the texts is converted into a corresponding semantic vector. Normalize each of the semantic vectors and calculate the cosine distance between the normalized semantic vectors; Based on the cosine distance and density clustering algorithms described above, the sample access order and multiple first clusters and their corresponding reachability distances are obtained.

[0010] In an optional implementation of this application, the step of iteratively optimizing the first clustering based on a binary search framework combined with semantic verification of a large model to obtain a clustering threshold includes: Based on the binary search framework, iterative search is performed within a preset neighborhood radius, and the intermediate threshold is calculated in each iteration; Update each of the first clusters based on the intermediate threshold, the sample access order, and the reachability distance; The median value of the cluster diameter corresponding to the updated first cluster is used as the median diameter, and the two samples corresponding to the median diameter are input into the large model to obtain the semantic verification result. The neighborhood radius range is optimized based on the semantic verification results until the iteration is completed, and the clustering threshold is obtained.

[0011] In an optional embodiment of this application, optimizing the neighborhood radius range based on the semantic verification result until the iteration is complete to obtain the clustering threshold includes: In response to the semantic verification result being semantically consistent, the minimum neighborhood radius in the neighborhood radius range is updated to the intermediate threshold; In response to the semantic verification result being semantically inconsistent, the maximum neighborhood radius within the neighborhood radius range is updated to the intermediate threshold; The iteration continues until the neighborhood radius does not exceed a preset precision threshold or reaches a preset number of iterations. Once the iteration is complete, the minimum neighborhood radius is used as the clustering threshold.

[0012] In an optional embodiment of this application, the step of deduplicating each of the second clusters based on a preset semantic filtering algorithm and a preset representation filtering algorithm to obtain a deduplicated text list includes: For each of the second clusters, the average of all the samples is calculated and normalized to obtain the cluster center; Based on the preset semantic filtering algorithm and the cluster centers, several candidate samples are selected from all the second clusters; The expression similarity between the candidate samples is calculated based on the preset expression filtering algorithm, and several final samples are selected. The original text corresponding to several final samples is converted into a structured list to obtain the deduplicated text list.

[0013] In an optional embodiment of this application, the step of selecting several candidate samples from all the second clusters based on the preset semantic filtering algorithm and the cluster centers includes: For each sample in each of the second clusters, calculate the cosine distance between the sample and the cluster centers of the remaining second clusters, and calculate the average cosine distance; In each of the second clusters, the top few samples with the largest average cosine distance are selected as candidate samples.

[0014] In an optional embodiment of this application, the step of calculating the expression similarity between the candidate samples based on the preset expression filtering algorithm to filter out a number of final samples includes: Based on a character-level n-gram model, the similarity of the expressions among the candidate samples is calculated. In response to the existence of candidate samples whose expression similarity does not exceed a preset similarity threshold, several candidate samples whose expression similarity does not exceed the preset similarity threshold are used as the final samples; In response to the absence of a sample whose representation similarity does not exceed a preset similarity threshold, the candidate sample with the largest average cosine distance is selected as the final sample.

[0015] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a deduplication device for mixed Chinese and English text based on density clustering and semantic verification, comprising: The first clustering module is used to convert each text into a semantic vector and obtain the first cluster based on the density clustering algorithm; The threshold optimization module is used to iteratively optimize the first clustering based on the binary search framework combined with the semantic verification of the large model to obtain the clustering threshold. The second clustering module is used to update the first clustering based on the clustering threshold to obtain multiple second clusterings; The filtering and deduplication module is used to deduplicat each of the second clusters based on a preset semantic filtering algorithm and a preset expression filtering algorithm to obtain a deduplicated list of texts.

[0016] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a computer device, including a memory, a processor and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the above-mentioned method for deduplication of mixed Chinese and English text based on density clustering and semantic verification.

[0017] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a storage medium on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the above-mentioned method for deduplication of mixed Chinese and English text based on density clustering and semantic verification.

[0018] The beneficial effects of this application are as follows: Unlike existing technologies, this application discloses a method for deduplicating mixed Chinese and English text based on density clustering and semantic verification. This method uses a binary search framework combined with semantic verification of a large model to automatically iteratively optimize the first cluster to obtain the optimal clustering threshold. It eliminates the need for manual setting of the clustering threshold, adapts to mixed Chinese and English text with different semantic densities, avoids "over-clustering" or "under-clustering," and achieves semantic consistency within the same cluster. Furthermore, through preset semantic filtering and expression filtering algorithms, samples in the second cluster are further filtered for deduplication, preserving semantic uniqueness and expression diversity while avoiding misjudgments of expression similarity due to differences in language structure. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a flowchart illustrating an embodiment of the Chinese-English mixed text deduplication method based on density clustering and semantic verification provided in this application; Figure 2 This is a schematic diagram of the automatic deduplication process of an embodiment of the Chinese-English mixed text deduplication method based on density clustering and semantic verification provided in this application; Figure 3 This is a schematic diagram of an embodiment of the Chinese-English mixed text deduplication device based on density clustering and semantic verification provided in this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the storage medium provided in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the computer device provided in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] The terms "first," "second," and "third" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This application provides a method for deduplication of mixed Chinese and English text based on density clustering and semantic verification. (See reference...) Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the Chinese-English mixed text deduplication method based on density clustering and semantic verification provided in this application. The method includes: S10: Convert each text into a semantic vector, and obtain the first cluster based on the density clustering algorithm.

[0024] In this application, the mixed Chinese and English text to be deduplicated (hereinafter referred to as "text") includes pure Chinese text, pure English text, and mixed Chinese and English text, which may take various forms such as a list of question and answer sentences, a collection of news reports, social media posts, or excerpts from academic papers.

[0025] Text deduplication involves identifying and removing duplicate or highly similar text from a given text set to reduce data redundancy and improve the efficiency of information retrieval and processing. In practical applications, text deduplication not only helps improve user experience but also plays an important role in fields such as data mining and natural language processing.

[0026] In this application, each text is converted into a semantic vector, and based on a density clustering algorithm, the sample access order and reachability distance are obtained, including: S11: Based on the text embedding model, each text is converted into its corresponding semantic vector.

[0027] A pre-trained text embedding model is used to convert the text to be deduplicated into high-dimensional semantic vectors to ensure that text vectors with similar semantics are closer together.

[0028] Specifically, leveraging the powerful language representation capabilities of text embedding models, the semantic features of each input text are accurately captured, and the text is mapped into a high-dimensional space to form corresponding semantic vectors. In this process, the text embedding model fully considers various factors such as the semantics of words, grammatical structure, and contextual information in the text, ensuring that the generated semantic vectors comprehensively and accurately reflect the semantic content of the text. In this way, the distance between the vectors corresponding to semantically similar texts in the high-dimensional space becomes closer, providing a solid foundation for subsequent clustering operations.

[0029] In this application, the text embedding model can be a model with excellent semantic representation capabilities, such as the Qwen3-Embedding model, the Bidirectional Encoder Representations from Transformers (BERT) model, or the Enhanced Representation through Knowledge Integration (ERNIE) model. These models are pre-trained on large-scale corpora and can deeply understand the semantic information of the text, converting the text into a vector representation rich in semantic features. In practical applications, an appropriate text embedding model can be selected according to specific needs and scenarios to ensure the accuracy and effectiveness of semantic vector conversion.

[0030] The high-dimensional semantic vectors obtained from text conversion can be 768-dimensional, 1024-dimensional, or higher. These high-dimensional vectors can capture the semantic features of the text more comprehensively and meticulously, providing a rich and accurate data foundation for subsequent density clustering and semantic verification. Different dimension choices can be flexibly adjusted according to factors such as the actual data scale, accuracy requirements, and computing resources.

[0031] S12: Normalize each semantic vector and calculate the cosine distance between the normalized semantic vectors.

[0032] Each high-dimensional semantic vector generated in step S11 is normalized. The normalization method can be L2 normalization, which divides the semantic vector by its L2 norm (i.e., magnitude). Alternatively, L1 normalization, maximum normalization, or minimum-maximum normalization can be used, and the appropriate method can be chosen flexibly based on actual needs and vector characteristics. This application uses L2 normalization as an example for explanation.

[0033] First, using the vector magnitude formula, the magnitude of each semantic vector in its corresponding dimensional space is calculated. Then, by dividing the semantic vector by its magnitude, the L2 norm of the normalized vector becomes 1, resulting in a unit vector. This eliminates the influence of vector scale differences on subsequent distance calculations. The L2 normalization formula is as follows: Formula 1; in, Represents the original semantic vector. This represents the normalized semantic vector. Semantic vectors The length of the module.

[0034] The above normalization process can eliminate the differences in length between different semantic vectors, making the subsequent cosine distance calculation more accurate and stable, reflecting only the directional differences between vectors, and not affected by the vector length.

[0035] For each normalized semantic vector, the cosine distance between every two semantic vectors is calculated. The cosine distance measures the semantic similarity between semantic vectors; the smaller the cosine distance, the more semantically similar the two vectors are. The formula for calculating the cosine distance is as follows: Formula 2; in, for and The cosine distance; and For two normalized semantic vectors, i.e. Then formula 2 can be simplified to .

[0036] By calculating the cosine distance, the similarity between each pair of semantic vectors can be accurately quantified, providing a reliable basis for subsequent density clustering operations. This allows for more precise classification of semantically similar vectors into the same category during the clustering process, thereby effectively distinguishing text content with different semantics and laying a solid foundation for deduplication of mixed Chinese and English texts.

[0037] S13: Based on cosine distance and density clustering algorithms, obtain the sample access order and multiple first clusters and their corresponding reachability distances.

[0038] Density clustering is an algorithm that clusters data based on density reachability between samples. It divides the data space into multiple clusters by finding the largest set of density-connected data points. This algorithm does not require pre-specifying the number of clusters and can automatically identify natural groupings in the data, making it particularly suitable for processing datasets with irregular shapes or uneven density. In text deduplication scenarios, density clustering can effectively identify semantically similar text vectors.

[0039] Density clustering algorithms include Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and Ordering Points To Identify the Clustering Structure (OPTICS). OPTICS is an improvement on DBSCAN; instead of directly generating clusters, it generates a reachability order for each data point. Subsequent processing can then yield clusters of varying densities, better handling datasets with uneven density. This application employs the OPTICS clustering algorithm for clustering.

[0040] For all normalized semantic vectors, perform OPTICS clustering. The three core inputs of the OPTICS clustering algorithm include the sample dataset, the upper bound of the neighborhood radius, and the minimum number of core points. The sample dataset consists of all normalized semantic vectors, with each semantic vector representing a sample; the upper bound of the neighborhood radius... This refers to the maximum radius used to define the "neighborhood." The OPTICS clustering algorithm analyzes the distribution of data points within this neighborhood radius. The value can range from 0.5 to 0.8, and can be set flexibly according to actual needs; minimum number of core points The minimum number of samples in the neighborhood required to determine whether a point is a "core point" is set based on the sample size, such as in a medium-sample scenario. The value can be between 10 and 20.

[0041] The two core outputs of the OPTICS clustering algorithm include the sample access order ( ) and reachable distance ( Here, sample access order refers to the sample index sorted from high to low text density. Sample access order ensures that samples with the same semantics (i.e., density contiguous) are arranged continuously. Reachability distance refers to the minimum distance from each sample to its corresponding core point. The core point refers to the sample with the smallest total distance from other samples in the same cluster. Reachability distance reflects the degree of association between the sample and the core semantics.

[0042] It is important to note that although the OPTICS clustering algorithm does not directly return cluster labels, it can be understood that the result of the OPTICS clustering algorithm is a number of clusters arranged in order of semantic density. Therefore, a core point can always be found in each cluster.

[0043] For ease of understanding, in this application, these clusters arranged in semantic density order are referred to as the first cluster, the reachability distance is the reachability distance of each sample in the first cluster, and the sample access order is the semantic density sorting of all samples in all first clusters.

[0044] S20: Based on the binary search framework combined with the semantic verification of the large model, the first cluster is iteratively optimized to obtain the clustering threshold.

[0045] In this application, based on a binary search framework combined with semantic verification of a large model, the first clustering is iteratively optimized to obtain the clustering threshold, including: S21: Based on the binary search framework, perform iterative search within a preset neighborhood radius, and calculate the intermediate threshold in each iteration.

[0046] Binary search is an algorithm for finding a specific element in a sorted array. It quickly locates the target by repeatedly halving the search range. Initializing the binary search boundaries involves setting an initial neighborhood radius as the upper and lower bounds of the search; the upper bound is the maximum neighborhood radius. The lower bound is the minimum neighborhood radius. Preset neighborhood radius range The value is typically [0.1, 0.8], but can be set according to actual clustering needs, such as [0.2, 0.8] or [0.2, 0.9]. A precision threshold is also preset. As the termination condition for the iteration, the precision threshold can be set according to the actual requirements for the clustering effect, such as 0.01 or 0.005. A preset upper limit for the number of iterations is set to control the maximum number of times the binary search is executed, preventing infinite loops caused by setting the precision threshold too low or special data distribution. It can usually be set according to the data scale and computing resources, such as 20 or 30 times. When the number of iterations reaches the upper limit, the search is forcibly terminated even if the precision threshold condition is not met, and the current best result is returned.

[0047] In each iteration, the median value of the current neighborhood radius is calculated as an intermediate threshold. This intermediate threshold serves as a crucial criterion for determining whether samples belong to the same cluster. Specifically, the formula for calculating the intermediate threshold is as follows: Formula 3; in, The intermediate threshold; The minimum neighborhood radius; The maximum neighborhood radius.

[0048] S22: Update each first cluster based on the intermediate threshold, sample access order, and reachability distance.

[0049] In each iteration, after calculating the current intermediate threshold, based on the sample access order and reachability distance, consecutive samples whose reachability distance does not exceed the intermediate threshold are selected. These consecutive samples are labeled into the same cluster, and cluster labels are generated, resulting in the updated first cluster. In this way, similar samples can be effectively grouped together to form clusters with clear semantic and spatial features, providing an accurate foundation for subsequent text deduplication operations.

[0050] S23: Take the median value of the cluster diameter corresponding to the updated first cluster as the median diameter, and input the two samples corresponding to the median diameter into the large model to obtain the semantic verification result.

[0051] For each updated first cluster obtained in step S22, calculate its cluster diameter and pivot diameter. Specifically, calculate the cosine distance of each sample pair within the cluster, and select the largest cosine distance as the cluster diameter corresponding to that cluster. Sort all the calculated cluster diameters by size, and select the median value as the pivot diameter, which represents the average semantic dispersion in the scenario where the current pivot threshold is used as the cluster threshold.

[0052] Extract the two samples corresponding to the middle diameter (i.e., the two samples with the largest cosine distance in the first cluster to which the middle diameter belongs, indicating the greatest semantic difference between them). Input these two samples into the Large Language Model (LLM) for semantic verification. Based on the semantic verification result of "true" or "false" returned by the LLM, determine whether the core semantics of the two samples are consistent. The choice of the LLM can be flexibly adjusted according to actual needs and application scenarios, such as GLM-4.6, GPT-4, or ERNIE Bot.

[0053] By leveraging the deep semantic understanding capabilities of large models, it is possible to accurately determine whether two input samples are consistent at the core semantic level, thus providing a reliable basis for subsequent text deduplication decisions. If the semantic verification result is "true", it indicates that the two samples are consistent in core semantics, although they may differ in expression or specific word choice; if the result is "false", it means that the two samples have significant differences in core semantics and should not be considered duplicate text.

[0054] S24: Optimize the neighborhood radius range based on the semantic verification results until the iteration is complete, and obtain the clustering threshold.

[0055] In this application, the neighborhood radius range is optimized based on the semantic verification results until the iteration is completed, and the clustering threshold is obtained, including: In response to the semantic verification result being semantically consistent, the minimum neighborhood radius within the neighborhood radius range is updated to the intermediate threshold; In response to a semantic inconsistency result, the maximum neighborhood radius within the neighborhood radius range is updated to the intermediate threshold. The iteration continues until the neighborhood radius does not exceed the preset precision threshold or the preset number of iterations is reached. Once the iteration is complete, the minimum neighborhood radius is used as the clustering threshold.

[0056] In this application, when the semantic verification result returned by the large model is "true", meaning that the two samples are semantically consistent, it indicates that the current intermediate threshold is too fine. The intermediate threshold can be expanded to merge more synonymous text. The way to expand the intermediate threshold is to update the smallest neighborhood radius in the current neighborhood radius range to the current intermediate threshold. For example, if the current neighborhood radius range is [0.1, 0.8], and the current intermediate threshold is (0.1+0.8) / 2=0.45, updating the smallest neighborhood radius in the neighborhood radius range to the intermediate threshold will result in an updated neighborhood radius range of [0.45, 0.8], and an updated intermediate threshold of (0.45+0.8) / 2=0.625.

[0057] When the semantic verification result returned by the large model is "false," meaning the two samples are semantically inconsistent, it indicates that the current intermediate threshold is too coarse. The intermediate threshold can be narrowed to separate text with different semantic meanings. Narrowing the intermediate threshold involves updating the maximum neighborhood radius within the current neighborhood radius range to the current intermediate threshold. For example, if the current neighborhood radius range is [0.1, 0.8], and the current intermediate threshold is (0.1 + 0.8) / 2 = 0.45, updating the maximum neighborhood radius within the current neighborhood radius range to the intermediate threshold will result in an updated neighborhood radius range of [0.1, 0.45] and an updated intermediate threshold of (0.1 + 0.45) / 2 = 0.275.

[0058] After determining the neighborhood radius, calculate the intermediate threshold and repeat steps S22-S24 until the iteration is complete. The minimum neighborhood radius at the end of the iteration is then taken as the optimal clustering threshold. The criteria for determining the completion of the iteration include that the neighborhood radius does not exceed a preset precision threshold, i.e. The iteration is considered complete when either condition is met, or the number of iterations reaches the preset number (e.g., 20 times). The minimum neighborhood radius obtained at this time can more accurately reflect the semantic similarity between texts, and thus serve as the optimal clustering threshold for deduplication of mixed Chinese and English texts.

[0059] Unlike existing density-based clustering algorithms that rely on manually set clustering thresholds and struggle to adapt to the semantic differences in mixed Chinese and English text, this application provides a density-based clustering and semantic verification method for deduplicating mixed Chinese and English text. This method uses density-based clustering to initially cluster the text, and then further optimizes the clustering results through a semantic verification mechanism. During the iterative clustering process, the neighborhood radius is automatically adjusted based on the semantic similarity between texts, resulting in a more accurate clustering threshold. This method better adapts to mixed Chinese and English text with varying semantic densities, avoiding over-clustering or under-clustering, and improving the accuracy and efficiency of deduplication.

[0060] S30: Update the first cluster based on the clustering threshold to obtain multiple second clusters.

[0061] In this application, the optimal clustering threshold is obtained through iteration in step S20 above. The cluster labels (IDs) are re-extracted based on the sample access order and reachability distance to obtain the final clustering results (Clusters, in the format of "cluster ID → sample index list", that is, each cluster uses its cluster ID as the key and the index list of its samples as the value). These clusters are recorded as the second cluster.

[0062] These secondary clusters, generated based on the optimal clustering threshold, can more accurately segment mixed Chinese and English text. The text within each secondary cluster exhibits high semantic similarity, while the semantic differences between different secondary clusters are relatively large. This method effectively distinguishes duplicate text content, thereby achieving deduplication of mixed Chinese and English text, improving the accuracy and efficiency of text processing, and providing a higher-quality data foundation for subsequent text analysis and other tasks.

[0063] S40: Based on the preset semantic filtering algorithm and the preset expression filtering algorithm, deduplication is performed on each second cluster to obtain a deduplicated text list.

[0064] In this application, based on a preset semantic filtering algorithm and a preset representation filtering algorithm, duplicates are removed from each second cluster to obtain a list of deduplicated texts, including: S41: For each second cluster, calculate the average of all samples and normalize it to obtain the cluster center.

[0065] Specifically, for each second cluster, the arithmetic mean of all normalized semantic vectors within the cluster is calculated, and then this arithmetic mean is L2 normalized to transform it back into a unit vector. Assume there are m samples in a cluster, and the normalized semantic vectors are v1, v2, …, v m (each v) i ∈R d And ||v i ||2 = 1, R d (i.e., d-dimensional real space), the average value is μ = 1 / m · Σ i=1 m v i μ is the average value of all samples within the cluster. Then, μ is L2 normalized, i.e., μ no m = μ / ||μ||2, μ no m This is the normalized mean. Using this normalized mean as the core semantic representation of the cluster, denoted as the cluster center, ensures that the cluster center falls within the "geometric center" of the semantic space while maintaining the comparability of cosine distance.

[0066] S42: Based on the preset semantic filtering algorithm and cluster centers, select several candidate samples from all the second clusters.

[0067] In this application, based on a preset semantic filtering algorithm and cluster centers, several candidate samples are selected from all second clusters, including: For each sample in each second cluster, calculate the cosine distance between the sample and the cluster centers of the other second clusters, and calculate the average cosine distance. In each second cluster, the top few samples with the largest average cosine distance are selected as candidate samples.

[0068] Specifically, for each sample in each second cluster, the cosine distance between that sample and the cluster centers of all other second clusters is calculated to accurately quantify the semantic similarity between the sample and different cluster centers. All calculated cosine distances are averaged to obtain the mean cosine distance for that sample, comprehensively reflecting the degree of deviation of the sample from the overall cluster centers. For each second cluster, samples are sorted according to the magnitude of the mean cosine distance across all samples in that second cluster, and the top few samples with the largest distances are selected as candidate samples due to their significant differences from the cluster centers, thereby improving the semantic uniqueness of deduplication.

[0069] In one embodiment, assuming there are 50 second clusters, each with 20 samples, then there are a total of 1000 samples in all the second clusters. For any sample x among these 1000 samples, calculate the cosine distance between x and the cluster centers of the other 49 second clusters, thus obtaining 49 cosine distances. Then, take the average of these 49 cosine distances. Where c(x) is the cluster ID to which x belongs. For the k-th cluster center, Let x be the mean cosine distance. For each of the 20 samples in the second cluster, calculate a corresponding mean cosine distance, resulting in 20 mean cosine distances {D(xi)}. Sort the samples according to their mean cosine distances and select the samples with the largest distances. Sample of names ( The maximum number of samples to be retained for each second cluster is usually 2-3, depending on actual needs. If the value is 3, then 2×3=6 candidate samples need to be selected as the most representative or the most dispersed. These can be obtained by calculating the maximum edge or the average edge.

[0070] The formula for calculating the maximum edge of 6 candidate samples is as follows: Formula 4; in, The cosine similarity between sample x and sample y. Let S be the cosine distance between two samples; S is a subset of 6 candidate samples; X is a set of 20 samples. A subset of 6 candidate samples is selected to maximize the difference between the two most similar samples within each sample.

[0071] The formula for calculating the 6 candidate samples using the average edge method is as follows: Formula 5; in, The cosine similarity between sample x and sample y. Let S be the cosine distance between two samples; S is a subset of 6 candidate samples; X is a set of 20 samples. This represents the average difference between each pair of samples, i.e., the average cosine distance between all sample pairs within subset S. To maximize the average difference between all sample pairs within it, a subset consisting of 6 candidate samples is selected.

[0072] The two screening methods mentioned above are essentially about selecting 6 samples from 20 samples, so that the pair of samples that are semantically "most similar" among these 6 samples is also as "least similar" as possible. In other words, they aim to maximize the cosine distance between each pair of samples. That is, in all the second clusters, the top few samples with the largest average cosine distance are selected as candidate samples.

[0073] By employing the semantic uniqueness screening method described above, samples with significant semantic differences can be effectively selected from a large number of samples. This screening method not only considers the direct similarity between samples but also ensures the semantic diversity and uniqueness of the selected samples by maximizing the mean cosine distance, thereby improving the accuracy and efficiency of deduplication.

[0074] S43: Calculate the similarity of expressions between candidate samples based on the preset expression filtering algorithm, and select a number of final samples.

[0075] In this application, a preset expression screening algorithm is used to calculate the expression similarity between candidate samples, and a number of final samples are selected, including: Based on a character-level n-gram model, the representation similarity between candidate samples is calculated. In response to the existence of candidate samples whose expression similarity does not exceed a preset similarity threshold, several candidate samples whose expression similarity does not exceed the preset similarity threshold are used as the final samples. In response to the absence of samples whose representation similarity does not exceed a preset similarity threshold, the candidate sample with the largest average cosine distance is selected as the final sample.

[0076] Specifically, the candidate samples obtained in step S42 are screened for representation diversity. A character-level n-gram model is used to calculate the representation similarity between candidate samples, and candidate samples with similarity not exceeding a preset similarity threshold are selected as the final samples to ensure that the representations are not duplicated. Here, the value of n can be 2 or 3, and the preset similarity threshold can be 0.3, 0.35 or 0.4, etc., which can be flexibly set according to actual needs.

[0077] In some embodiments, there may be no candidate samples that can satisfy the diversity of expression, that is, there are no samples whose expression similarity does not exceed a preset similarity threshold. In this case, in order to avoid clustering without representative samples, at least one candidate sample with the most unique semantics is retained as the final sample. This candidate sample with the most unique semantics is the candidate sample with the largest average cosine distance.

[0078] Unlike existing technologies that are insufficiently adapted to mixed Chinese and English text and lack dual constraints on semantics and expression, this application provides a method for deduplicating mixed Chinese and English text based on density clustering and semantic verification. This method ensures semantic consistency within clusters through cosine distance filtering and preserves expression diversity through n-gram filtering, solving the problems of "loss of expression during deduplication" or "retention of duplicate expressions." Furthermore, by employing character-level n-gram processing of mixed Chinese and English text, it can preserve special English symbols (such as "-" and "_"), thereby avoiding misjudgments of expression similarity due to differences in language structure.

[0079] S44: Convert the original text corresponding to several final samples into a structured list to obtain a deduplicated text list.

[0080] For the several final samples obtained in step S43, organize the original text corresponding to these final samples into a structured list for storage, such as storing it in JSON format. This structured list is the deduplicated text list.

[0081] Structured lists stored in JSON format offer excellent readability and scalability, facilitating further processing and analysis of the deduplicated text, such as quickly retrieving specific content or performing text classification and statistical operations. Furthermore, JSON format enjoys broad support across different programming languages ​​and platforms, promoting data sharing and interaction.

[0082] In some embodiments, the process of performing the above steps to remove duplicates includes a boundary check mechanism and an anomaly capture mechanism, which can handle abnormal text data (such as short text or text with special symbols).

[0083] Specifically, the boundary checking mechanism ensures that the index does not exceed the valid range when processing text data, and properly handles null values ​​or invalid inputs to prevent the program from crashing or producing incorrect results due to invalid data. The exception handling mechanism, on the other hand, can promptly capture and handle potential anomalies, such as network problems or model service unavailability, during critical steps like calling large models for semantic verification, ensuring the stability and reliability of the deduplication process. This comprehensive error handling and exception handling mechanism enables the method to cope with various abnormal text data, including short texts and text containing special symbols, thereby improving the robustness and applicability of the deduplication method.

[0084] See Figure 2 , Figure 2 This is a schematic diagram of the automatic deduplication process of an embodiment of the Chinese-English mixed text deduplication method based on density clustering and semantic verification provided in this application. First, the mixed Chinese and English text is input into the text embedding model, converted into semantic vectors, and normalized. The cosine distance between each pair of normalized semantic vectors is calculated to obtain a cosine distance matrix. Next, OPTICS clustering is performed to obtain the sample access order, multiple first clusters, and their corresponding reachability distances. Then, a binary search neighborhood radius is initialized for searching, and an intermediate threshold is calculated. Based on the intermediate threshold, sample access order, and reachability distance, the first clusters are updated, and the intermediate diameter of multiple updated first clusters is determined. The two samples corresponding to the intermediate diameter are input into the larger model for semantic verification, and the neighborhood radius is updated according to the semantic verification results until the iteration terminates, obtaining the optimal clustering threshold. The second cluster is then updated based on the clustering threshold. Finally, multiple second clusters are further deduplicated. Semantic uniqueness deduplication is performed based on a preset semantic filtering algorithm to obtain several candidate samples. Representation diversity deduplication is performed on several candidate samples based on a preset representation filtering algorithm to obtain several final samples. The original text corresponding to the final samples is output as a structured list to obtain a deduplicated text list.

[0085] In one embodiment, a technical Q&A list (containing several mixed Chinese and English sentences) is deduplicated. The list contains 500 sentences in total. First, these 500 sentences are converted into 768-dimensional semantic vectors using a text embedding model, and L2 normalization is performed to obtain 500 normalized semantic vectors. Then, the cosine distance matrix is ​​calculated, OPTICS clustering is performed, and an upper limit on the neighborhood radius is set. =0.5, minimum number of core points =10, based on the cosine distance matrix, the sample access order and the reachability distances of multiple first clusters are obtained; then, based on the binary search framework combined with semantic verification of the large model, the first clusters are iteratively optimized, and the neighborhood radius range is set. =[0.1, 0.8], sets the precision threshold. =0.01, the clustering threshold is obtained after 12 iterations. The clustering process yields multiple second clusters; further, based on preset semantic filtering algorithms and preset representation filtering algorithms, duplicates are removed from each of the second clusters, and settings are configured... Each second cluster selects 2×3 candidate samples, and then the candidate samples are filtered for expression diversity. A preset similarity threshold of 0.3 is set. After filtering, each second cluster retains 2~3 final samples with different expressions, resulting in a total of 320 final samples. Finally, a structured list is generated based on the 320 original question and answer sentences corresponding to all final samples.

[0086] The deduplication process achieves a deduplication rate of approximately 36%, while preserving semantic consistency (sentences within the same cluster have consistent core semantics) and expression diversity (no duplicate expressions). For example, both "How to get a refund using the API?" and "How to apply for an API refund?" belong to the category of "API refund consultation," while "How to get a refund using the API?" is filtered out due to its duplicate expression. At the same time, it satisfies the requirements for Chinese and English compatibility. For example, "How to get user_id?" and "How to get user_id?" are classified into the same cluster and retained because they have consistent semantics but significant differences in expression.

[0087] This application provides a deduplication device for mixed Chinese and English text based on density clustering and semantic verification. (See reference...) Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the Chinese-English mixed text deduplication device based on density clustering and semantic verification provided in this application. The Chinese-English mixed text deduplication device based on density clustering and semantic verification includes: The first clustering module 10 is used to convert each text into a semantic vector and obtain the first cluster based on the density clustering algorithm; The threshold optimization module 20 is used to iteratively optimize the first cluster based on the binary search framework combined with the semantic verification of the large model to obtain the cluster threshold. The second clustering module 30 is used to update the first clustering based on the clustering threshold to obtain multiple second clusterings; The filtering and deduplication module 40 is used to deduplicat each second cluster based on a preset semantic filtering algorithm and a preset expression filtering algorithm to obtain a deduplicated text list.

[0088] The first clustering module 10, the threshold optimization module 20, the second clustering module 30, and the deduplication module 40 interact to realize the deduplication process of mixed Chinese and English text based on density clustering and semantic verification. You can refer to the specific description of steps S10 to S40 above. The repeated parts will not be repeated here.

[0089] See Figure 4 , Figure 4This is a schematic diagram of the structure of an embodiment of the storage medium provided in this application.

[0090] The storage medium 400 stores program data 410, which, when executed by the processor, implements, as follows: Figure 1 The steps of the described method for deduplication of mixed Chinese and English text based on density clustering and semantic verification.

[0091] The program data 410 is stored in a storage medium 400 and includes several instructions for causing a network device (which may be a router, personal computer, server, or other network device) or processor to execute all or part of the steps of the methods described in the various embodiments of this application.

[0092] Optionally, the storage medium 400 can be any medium that can store program data, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), disk, or optical disc.

[0093] See Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the computer device provided in this application.

[0094] The device 500 includes a processor 520 and a memory 510 connected to each other. The memory 510 stores a computer program. When the processor 520 executes the computer program, it implements the steps of the above-described method for deduplication of mixed Chinese and English text based on density clustering and semantic verification.

[0095] Unlike existing technologies, this application discloses a method for deduplicating mixed Chinese and English text based on density clustering and semantic verification. This method uses a binary search framework combined with semantic verification of a large model to automatically iteratively optimize the first cluster to obtain the optimal clustering threshold. This eliminates the need for manual setting of the clustering threshold, adapts to mixed Chinese and English text with varying semantic densities, avoids over-clustering or under-clustering, and ensures semantic consistency within the same cluster. Furthermore, through preset semantic and expression filtering algorithms, samples in the second cluster are further filtered for deduplication, preserving semantic uniqueness and expression diversity while avoiding misjudgments of expression similarity due to differences in language structure.

[0096] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the storage medium embodiments and computer device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0097] This application can be used in a wide range of general-purpose or special-purpose computing system environments or configurations. For example: personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, distributed computing environments including any of the above systems or devices, etc.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative; multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for deduplicating mixed Chinese and English text based on density clustering and semantic verification, characterized in that, include: Each text is converted into a semantic vector, and the first cluster is obtained based on the density clustering algorithm; Based on the binary search framework combined with the semantic verification of the large model, the first clustering is iteratively optimized to obtain the clustering threshold; The first cluster is updated based on the clustering threshold to obtain multiple second clusters; Based on the preset semantic filtering algorithm and the preset expression filtering algorithm, duplicates are removed from each of the second clusters to obtain a list of deduplicated texts.

2. The method for deduplication of mixed Chinese and English text based on density clustering and semantic verification according to claim 1, characterized in that, The process of converting each text into a semantic vector and obtaining the first cluster based on a density clustering algorithm includes: Based on the text embedding model, each of the texts is converted into a corresponding semantic vector. Normalize each of the semantic vectors and calculate the cosine distance between the normalized semantic vectors; Based on the cosine distance and density clustering algorithms described above, the sample access order and multiple first clusters and their corresponding reachability distances are obtained.

3. The method for deduplication of mixed Chinese and English text based on density clustering and semantic verification according to claim 2, characterized in that, The semantic verification based on the binary search framework combined with the large model iteratively optimizes the first clustering to obtain the clustering threshold, including: Based on the binary search framework, iterative search is performed within a preset neighborhood radius, and the intermediate threshold is calculated in each iteration; Update each of the first clusters based on the intermediate threshold, the sample access order, and the reachability distance; The median value of the cluster diameter corresponding to the updated first cluster is used as the median diameter, and the two samples corresponding to the median diameter are input into the large model to obtain the semantic verification result. The neighborhood radius range is optimized based on the semantic verification results until the iteration is completed, and the clustering threshold is obtained.

4. The method for deduplication of mixed Chinese and English text based on density clustering and semantic verification according to claim 3, characterized in that, The process of optimizing the neighborhood radius range based on the semantic verification result until the iteration is complete, to obtain the clustering threshold, includes: In response to the semantic verification result being semantically consistent, the minimum neighborhood radius in the neighborhood radius range is updated to the intermediate threshold; In response to the semantic verification result being semantically inconsistent, the maximum neighborhood radius within the neighborhood radius range is updated to the intermediate threshold; The iteration continues until the neighborhood radius does not exceed a preset precision threshold or reaches a preset number of iterations. Once the iteration is complete, the minimum neighborhood radius is used as the clustering threshold.

5. The method for deduplication of mixed Chinese and English text based on density clustering and semantic verification according to claim 1, characterized in that, The preset semantic filtering algorithm and preset representation filtering algorithm are used to remove duplicates from each of the second clusters, resulting in a list of deduplicated texts, including: For each of the second clusters, the average of all the samples is calculated and normalized to obtain the cluster center; Based on the preset semantic filtering algorithm and the cluster centers, several candidate samples are selected from all the second clusters; The expression similarity between the candidate samples is calculated based on the preset expression filtering algorithm, and several final samples are selected. The original text corresponding to several final samples is converted into a structured list to obtain the deduplicated text list.

6. The method for deduplication of mixed Chinese and English text based on density clustering and semantic verification according to claim 5, characterized in that, Based on the preset semantic filtering algorithm and the cluster centers, several candidate samples are selected from all the second clusters, including: For each sample in each of the second clusters, calculate the cosine distance between the sample and the cluster centers of the remaining second clusters, and calculate the average cosine distance; In each of the second clusters, the top few samples with the largest average cosine distance are selected as candidate samples.

7. The method for deduplication of mixed Chinese and English text based on density clustering and semantic verification according to claim 5, characterized in that, The algorithm for calculating the expression similarity between candidate samples based on the preset expression filtering algorithm is used to filter out several final samples, including: Based on a character-level n-gram model, the similarity of the expressions among the candidate samples is calculated. In response to the existence of candidate samples whose expression similarity does not exceed a preset similarity threshold, several candidate samples whose expression similarity does not exceed the preset similarity threshold are used as the final samples; In response to the absence of a sample whose representation similarity does not exceed a preset similarity threshold, the candidate sample with the largest average cosine distance is selected as the final sample.

8. A deduplication device for mixed Chinese and English text based on density clustering and semantic verification, characterized in that, include: The first clustering module is used to convert each text into a semantic vector and obtain the first cluster based on the density clustering algorithm; The threshold optimization module is used to iteratively optimize the first clustering based on the binary search framework combined with the semantic verification of the large model to obtain the clustering threshold. The second clustering module is used to update the first clustering based on the clustering threshold to obtain multiple second clusterings; The filtering and deduplication module is used to deduplicat each of the second clusters based on a preset semantic filtering algorithm and a preset expression filtering algorithm to obtain a deduplicated list of texts.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method for deduplication of mixed Chinese and English text based on density clustering and semantic verification as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for deduplication of mixed Chinese and English text based on density clustering and semantic verification as described in any one of claims 1-7.

Images