Data deduplication method, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a data deduplication method, electronic equipment and a computer readable storage medium. The method comprises the following steps: generating a plurality of sample points based on semantic features respectively corresponding to a plurality of data; determining transition probability information between each sample point and other sample points according to the semantic similarity between the semantic features corresponding to the multiple pieces of data; performing multi-step transfer in each sample point according to the transfer probability information and the simulation information to obtain a density value of each sample point, the density value representing the measurement of the information accumulation degree of each sample point after the multi-step transfer; determining transition probability distances among the plurality of sample points according to the transition probability information; and according to the transition probability distance among the plurality of sample points and the density value of each sample point, performing deduplication processing on the plurality of data to obtain a deduplication result. According to the method and the apparatus, the technical problems of limited duplicate removal capability and poor duplicate removal robustness of redundant data in related technologies are solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data deduplication method, electronic device, and computer-readable storage medium. Background Technology

[0002] With the development of large-scale pre-trained language models and their widespread application in tasks such as question answering and information retrieval, Retrieval Augmentation (RAG) technology has become a core paradigm for improving the accuracy, real-time performance, and data reliability of large-scale model-generated results. RAG systems generate answers by retrieving large-scale external knowledge bases in real time and using relevant text fragments as context input to the large model. Existing RAG systems face the challenge of deduplication due to large-scale semantically repetitive data. This includes issues such as repeated collection of the same information from multiple sources, multiple expressions of the same fact, mirrored web page content, iterative updates of information versions, and duplicates generated during web crawling. Highly similar documents or text fragments negatively impact the performance of RAG systems, manifesting as decreased retrieval efficiency, redundant and repetitive generated answers, increased response latency, and higher information storage costs.

[0003] The data deduplication methods provided in related technologies mostly rely on shallow semantic matching, which makes it difficult to effectively identify redundant information with diverse forms of expression but consistent semantics. In high-dimensional semantic embedding space, there are drawbacks such as fuzzy clustering boundaries and unstable redundancy judgment, resulting in limited ability to identify duplicate corpora and difficulty in improving the accuracy and efficiency of redundancy recognition. Summary of the Invention

[0004] This application provides a data deduplication method, an electronic device, and a computer-readable storage medium to alleviate or solve the technical problems of limited deduplication capability and poor robustness of related technologies for redundant data.

[0005] In a first aspect, embodiments of this application provide a data deduplication method, including: Based on the semantic features corresponding to multiple data points, multiple sample points are generated, and the multiple sample points correspond one-to-one with the multiple data points. Based on the semantic similarity between the semantic features corresponding to the multiple data points, the transfer probability information between each sample point and other sample points is determined, and the transfer probability information represents the probability of information transfer from each sample point to other sample points; According to the transfer probability information, the simulated information undergoes multiple transfers at each sample point to obtain the density value of each sample point. The density value represents a measure of the degree of information accumulation at each sample point after multiple transfers. Based on the transition probability information, the transition probability distance between the plurality of sample points is determined, and the transition probability distance is used to represent a measure of the similarity between samples; Based on the transition probability distance between the multiple sample points and the density value of each sample point, the multiple data are deduplicated to obtain the deduplication result.

[0006] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0008] Based on the data deduplication method described in the first aspect above, this application has at least the following beneficial effects or advantages: the density value obtained by simulating multi-step transitions can effectively reflect the degree of clustering of sample points in the semantic space, providing a basis for determining cluster centers. Compared with the direct semantic association in related technologies, which is easily affected by differences in expression forms, this method can effectively capture redundant data with different expression forms but consistent semantics. Furthermore, common neighbor sample points can reflect the indirect association of samples in the semantic space. By utilizing the transition probability of common neighbors to enhance the separation of samples, a more accurate data deduplication effect can be achieved.

[0009] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0010] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0011] Figure 1 A flowchart of a data deduplication method according to an embodiment of this application is shown; Figure 2 This illustration shows an information transfer diagram of the data deduplication method according to an embodiment of this application; Figure 3 A flowchart of a data deduplication apparatus according to an embodiment of this application is shown; Figure 4 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0012] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0013] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0014] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0015] The following terms will be used in the following text: Retrieval-Augmented Generation (RAG) is a technique that combines information retrieval with generative AI. Before generating an answer, it retrieves information related to the question from an external knowledge base and then uses this information as context input to the generative model to help generate more accurate and reliable content.

[0016] Large-scale pre-trained language models (LLMs) are large-scale artificial intelligence models trained on massive amounts of data. They learn language rules, semantic relationships, and world knowledge, and can understand and generate semantic text.

[0017] To provide more comprehensive, real-time, and authoritative knowledge support, current RAG systems often need to access data sources from various heterogeneous structures, such as PDF documents, web page content, and database forms, to build a large-scale semantic vector index library. However, the construction process inevitably introduces a large amount of redundant information, such as multiple expressions of the same knowledge content, web page mirroring, version updates, crawler errors, sentence truncation, and noise injection. This semantic redundancy and low-quality data not only increase the size of the vector index library, increase the storage and computational burden of the vector database, and delay response speed, but also cause duplicate content to crowd out the limited context window of the LLM (e.g., GPT-4 only has 128K tokens), diluting key information and making the model susceptible to noise interference, resulting in contradictory outputs.

[0018] How to efficiently and accurately identify and remove highly similar and duplicate content from massive retrieval data sources while ensuring the breadth and depth of relevant information, so as to provide more refined and information-density contextual input to large models and significantly improve their understanding, reasoning, and generation quality and efficiency, has become a key technical challenge that urgently needs to be addressed in optimizing the performance of RAG systems. Existing data deduplication schemes all suffer from limitations in recognition capabilities and poor clustering robustness when dealing with semantic redundancy issues in RAG knowledge base construction. Especially in practical applications, when facing millions of unstructured data points, relying solely on vector thresholding or static clustering schemes is insufficient to accurately model the complex nonlinear relationships between semantic expressions. There is also the risk of mistakenly deleting semantically complementary variant data, leading to a decrease in the completeness and diversity of responses. Therefore, a novel algorithmic framework is urgently needed that can more effectively model high-dimensional semantic structures and improve the accuracy and computational efficiency of redundancy identification.

[0019] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0020] Figure 1 A flowchart of a data deduplication method according to an embodiment of this application is shown, such as... Figure 1 As shown, the method may include steps S101 to S105.

[0021] Step S101: Based on the semantic features corresponding to multiple data points, generate multiple sample points, with each sample point corresponding to one of the multiple data points; Step S102: Based on the semantic similarity between the semantic features corresponding to multiple data points, determine the transfer probability information between each sample point and other sample points. The transfer probability information represents the probability of information transfer from each sample point to other sample points. Step S103: According to the transition probability information, simulate the information to undergo multiple steps of transition at each sample point to obtain the density value of each sample point. The density value represents the measure of the degree of information accumulation at each sample point after multiple steps of transition. Step S104: Determine the transition probability distance between multiple sample points according to the transition probability information. The transition probability distance is used to represent the similarity between samples. Step S105: Based on the transition probability distance between multiple sample points and the density value of each sample point, perform deduplication processing on multiple data to obtain the deduplication result.

[0022] In the embodiments provided in this application, semantic features are extracted from multiple data points and transformed into corresponding vectors to generate sample points. Based on the similarity between the semantic features of each data point, the probability of each sample point transitioning to other sample points is calculated, yielding transition probability information. This probability transition information can be in matrix form, where each matrix element represents the probability of a "random walk" from sample i to sample j, intuitively reflecting the neighbor distribution structure of sample i. The density values ​​obtained from multi-step transitions reflect the degree of sample clustering, and the transition probability distance accurately measures sample similarity. Combining these two methods for deduplication captures deep semantic relationships between data, helping to identify redundant data with different expressions but consistent semantics. This overcomes the problem of unstable deduplication performance in complex semantic spaces caused by traditional methods, improving the accuracy and robustness of deduplication and providing more refined and information-dense data for scenarios such as RAG systems.

[0023] For example, the aforementioned execution entity is the RAG system, which can be applied to the data cleaning stage in offline knowledge base construction and data retrieval post-processing.

[0024] According to the embodiments provided in this application, the multiple data in step S101 can be multimodal data, and the data types can include text, images, videos, etc. One-to-one corresponding sample points are generated based on their respective semantic features. For example, semantic features of text are extracted using a pre-trained language model, while visual semantic features of images and videos can be extracted based on a pre-trained visual model. This not only identifies data with different expressions but the same substantive semantics, but also enables the deduplication of redundant data from different modalities. The following explanation uses text data as an example. The aforementioned text data can be unstructured data, such as sentences, paragraphs, documents, etc. A general open-source pre-trained word embedding model is used to convert the original text data into a fixed-dimensional dense vector representation.

[0025] Each data point of different data types is processed to obtain a fixed d-dimensional feature vector. For a dataset containing N data points, a feature matrix of dimension N x d is generated, capturing the semantic information of the data for subsequent calculations.

[0026] According to the embodiments provided in this application, the semantic features corresponding to multiple data points can form a feature matrix. This feature matrix is ​​used to construct transition probability information. By quantifying the semantic associations between samples, the semantic similarity of the data is transformed into a probability distribution that can be used to simulate information transfer. Each matrix element in the feature matrix represents a semantic feature of a data point, corresponding to a sample point. Based on the feature matrix, the initial similarity is obtained by calculating the cosine similarity between all sample pairs. Following the example above, with N×N sample pairs, an initial similarity matrix of dimension N×N can be formed. Cosine similarity reflects the degree of semantic similarity by measuring the cosine value of the angle between two vectors, providing the basis for the original semantic association. Each row of vectors in the initial similarity matrix represents the similarity of a sample point to all other sample points.

[0027] The initial similarity matrix is ​​normalized to ensure that the sum of each row's elements is 1, transforming it into a probability distribution and yielding the state transition matrix P. This state transition matrix can be in matrix form, representing the transition probability information. The normalization process can be achieved by applying Softmax or other normalization methods along the row direction. In the state transition matrix, the matrix element P[i, j] represents the transition probability of a random walk from sample i to sample j, intuitively reflecting the neighbor distribution structure of sample i, i.e., which samples the semantic associations of sample i tend to concentrate on.

[0028] The embodiments provided in this application can perform density estimation for fast kernel diffusion (Neighborhood-Diffusion-based Density, NDDe) in the following manner. The density value of each sample point is calculated by simulating the multi-step diffusion process of information on the sample map. The core idea is that the density of a sample point depends not only on the similarity of its direct neighbor sample points, but also on its higher-order neighbors, which are neighbor sample points that can be reached through multi-step transitions.

[0029] According to some embodiments provided in this application, in step S103: the simulation information is transferred in multiple steps at each sample point according to the transition probability information to obtain the density value of each sample point, which may include the following steps: Based on semantic similarity, neighboring sample points are determined for each sample point. The neighboring sample points are a predetermined number of sample points with the highest semantic similarity to each sample point. The simulated information is transferred from each sample point to its corresponding neighbor sample point, and the single-step value of each sample point after performing a single-step transfer is obtained. Repeat the single-step transfer process until the single-step values ​​corresponding to the multiple sample points after the transfer meet the predetermined convergence conditions. Then, use the single-step values ​​corresponding to the multiple sample points after the transfer as the density values ​​corresponding to the multiple sample points.

[0030] In the embodiments provided in this application, based on the semantic similarity between sample points, a predetermined number of sample points with the highest semantic similarity are selected as neighbor sample points for each sample point, namely the TOP-K nearest neighbor sample points. This focuses on the samples most closely related to the target sample, providing a reliable range for information transfer. The density value is used to measure the degree of information accumulation of a sample point after multiple transfers. Since a sample point will transfer information to its neighbor sample points, its neighbor sample points will also be affected by the transfers from other sample points, which are equivalent to the higher-order neighbor sample points of that sample point. The higher-order neighbor sample points influence the sample point through its neighbor sample points. For each step in the information transfer process, information is simulated to transfer from each sample point to its corresponding neighbor sample point, and the single-step value of each sample point after performing a single-step transfer is obtained. The influence of the higher-order neighbor sample points is integrated, and the above processing is repeated until the single-step values ​​corresponding to multiple sample points meet the predetermined convergence condition. At this time, the single-step values ​​of each sample point after the transfer are determined as their corresponding density values.

[0031] By selecting a predetermined number of neighboring sample points with the highest semantic similarity, the system can focus on samples most closely associated with the target sample, providing a reliable range for information transfer. Simulating multi-step information transfer and using the converged single-step value as the density value fully integrates the influence of higher-order neighbors, making the density value more accurately reflect the degree of sample clustering in the semantic space and avoiding local biases caused by relying solely on direct neighbors. The density value provides a basis for subsequently determining cluster centers, thereby improving the accuracy of deduplication, making the deduplicated data more refined, effectively supporting the efficient retrieval needs of scenarios such as RAG systems, and reducing the interference of redundant information on system performance.

[0032] According to the embodiments provided in this application, the above-mentioned simulation information is transferred from each sample point to its corresponding neighbor sample point to obtain the single-step value of each sample point after performing a single-step transfer, which may include the following processing: For any first sample point among multiple sample points, based on the current value of the first sample point, update the current value of the corresponding neighbor sample point according to the third probability indicated by the transition probability information, and determine the single-step value of the neighbor sample point corresponding to the first sample point; the third probability is the probability of information transfer between the first sample point and the corresponding neighbor sample point. Based on the current value of the corresponding neighbor sample point, update the current value of the first sample point according to the third probability to determine the single-step value of the first sample point; The single-step value of each sample point after single-step transition is obtained by obtaining the single-step value of the first sample point and the single-step value of the neighboring sample points corresponding to the first sample point.

[0033] In the embodiments provided in this application, multiple sample points each correspond to a preset initial density value, which is updated step by step during the multi-step transition. The third probability can be obtained from the aforementioned transition probability information, representing the semantic similarity between the first sample point and its corresponding neighbor sample point. It can be understood that the higher the semantic similarity, the higher the probability of information transfer. For any first sample point among the multiple sample points, based on the current value of the first sample point in the current step of the transition, the density value after information diffusion from the first sample point to the corresponding neighbor sample point is determined according to the aforementioned third probability, and the current value of the neighbor sample point is updated, thus obtaining the single-step value of the neighbor sample point. Similarly, neighbor sample points will also undergo information transfer to the first sample point. Based on the current value of the neighbor sample point, the current value of the first sample point is also updated according to the third probability, thus obtaining the single-step value of the first sample point in the current step of the transition. This method of determining single-step values ​​can be used for multiple sample points to obtain the single-step values ​​of all sample points after performing single-step transitions. Figure 2 This illustration shows an information transfer diagram of the data deduplication method according to an embodiment of this application, such as... Figure 2 As shown, sample points A, B, C, D, and E are used for illustration. In addition to the diffusion relationship with its neighbors B, C, and D, sample point A is also indirectly affected by sample point E through sample point C.

[0034] Calculating single-step values ​​using bidirectional information transfer based on transition probability helps capture semantic relationships between sample points, allowing the single-step value to reflect the degree of mutual influence. The semantic relationships of higher-order neighbor sample points can be diffused to the first sample point through its neighbor sample points via multi-step information transfer. This process makes the single-step value of each sample point more closely reflect its actual association state in the semantic network, thereby improving the accuracy of the density value in measuring the degree of sample information accumulation and helping subsequent deduplication processes more accurately identify redundant data.

[0035] For example, the above-mentioned rapid nuclear diffusion process can be carried out in the following manner:

[0036] in, Sample points In the The current density value in the step transition (i.e., the current value mentioned above), where I represents the identity matrix, is the density initialized from uniform. Initially, each sample carries initial information at the start of diffusion. The diffusion process continuously assigns the density value of each sample point to its top-K nearest neighbors, following the corresponding transition probabilities in the state transition matrix P (i.e., the matrix form of transition probability information), until it converges to a steady-state distribution. Because kernel diffusion density estimation, through a multi-step information diffusion process, makes semantically closely related sample points tend to have consistent density values, it can effectively reflect the degree of clustering of sample points in the semantic space, providing a basis for determining cluster centers, and enabling the sample points at the cluster center to better represent the core information of the samples within the cluster.

[0037] According to the embodiments provided in this application, in step S104: determining the transition probability distance between multiple sample points based on the transition probability information may include the following processing: Based on semantic similarity, neighboring sample points are determined for each sample point. The neighboring sample points are a predetermined number of sample points that have the highest semantic similarity to each sample point. Based on the transition probability information, determine the first probability of information transfer between each sample point and its common neighbor points, and the second probability of information transfer between other sample points and their common neighbor points; common neighbor points are neighbor sample points shared by each sample point and other sample points. Based on the first probability and the second probability, the transition probability distance between multiple sample points is determined.

[0038] In the embodiments provided in this application, for each sample point and other sample points, common neighbor sample points are found. A first probability of information transfer from the sample point to the common neighbor point is determined, as well as a second probability of information transfer from other sample points to the same common neighbor point. This captures the indirect semantic connections established between samples through intermediate nodes. Combining the first and second probabilities, the transition probability distance between the sample point and other sample points is determined. Transition Probability-based Distance (TPDi) is a method for measuring the distance of similarity between samples. The core of TPDi lies in using the transition probability information of common neighbors between samples to define their similarity.

[0039] Specifically, TPDi calculates the similarity distance using the transition probabilities of common neighbors between samples. For sample point i and sample point j, each sample point has some neighboring sample points, and the neighboring sample points of sample point i are denoted as... The neighboring sample points of sample point j are denoted as , Let represent the common neighbor samples of the top-K nearest neighbors of sample points i and j. The first probability can be represented as the transition probability between sample i and its common neighbor samples, denoted as . The second probability can be represented as the transition probability between sample j and its common neighbor sample points, denoted as... c represents A common neighbor sample point in the array. The transition probability distance between sample i and sample j can be denoted as: , can be defined as: .

[0040] It should be noted that for data deduplication, the goal is to retain representative core semantic data and eliminate redundant data, rather than strictly defining clearly defined cluster structures. Therefore, a complete clustering step is not necessary. The aforementioned NDDe calculates sample density values ​​by simulating multi-step information diffusion, accurately capturing the semantic accumulation of samples in the semantic space. Samples with higher density can be considered key accumulation points of semantic association within a local area. The aforementioned TPDi calculates the distance between samples using the transition probability of common neighbor sample points, measuring the separation degree between a sample and higher-density samples. Combining these two methods can filter out sample points with semantic coreness and semantic independence as cluster center sample points to be retained, without relying on explicit cluster division. The relationship between other data points and cluster centers is not the focus of this application's embodiments. The elimination of redundant data does not depend on the cluster division logic and can achieve the deduplication goal without performing a complete clustering operation.

[0041] In the embodiments provided in this application, two sample points whose transition probability distance is less than or equal to a predetermined distance threshold can be considered as positive sample pairs, which are highly similar semantically and should belong to the same cluster if clustering is performed. On the other hand, negative sample pairs whose transition probability distance is greater than the predetermined distance threshold should belong to different clusters if clustering is performed. The transition probability distance TPDi enhances the ability to distinguish between positive and negative sample pairs.

[0042] Compared to related techniques that rely solely on the cosine distance of semantic features, the advantage of transition probability distance lies in the higher separation between positive and negative sample pairs in the generated similarity distribution. This makes it easier to choose a uniform and robust similarity threshold to determine whether samples belong to the same cluster, resulting in more accurate clustering results.

[0043] Suppose clustering is required to assign multiple sample points to different clusters. Within the same cluster, the density levels need to be balanced so that samples within the same cluster tend to have similar density values, thus facilitating the identification of local cluster centers. Density differences should be maintained between different clusters to eliminate density fluctuations in different representations of the same topic, highlighting the true semantic centers. Generally, it is believed that density imbalances easily highlight cluster centers, making clustering methods that rely on the density of direct neighbors suitable. However, these methods are susceptible to noise interference in high-dimensional semantic spaces, leading to large density fluctuations within clusters and making it difficult to distinguish true centers from redundant samples.

[0044] Assuming clustering is still required, the diffusion process in the embodiments provided in this application allocates density to higher-order neighbors through transition probabilities, making the sample densities within the same cluster similar. Achieving intra-cluster balance does not eliminate differences, but rather makes the density characteristics of samples within the cluster more uniform, avoiding interference from local density abrupt changes in center identification. Although the cluster center has a similar density to other samples within the cluster, it is farther away from other clusters, and therefore can still be screened out for identification.

[0045] According to the embodiments provided in this application, determining the neighboring sample points of each sample point based on semantic similarity may include the following processing: A sample graph is generated by using multiple sample points as nodes and the semantic similarity between multiple sample points as edges. In the sample graph, a predetermined number of sample points are selected as neighbor sample points according to their semantic similarity with each other from high to low.

[0046] In the embodiments provided in this application, a sample graph can be constructed based on the aforementioned multiple sample points. The generated sample points serve as nodes, and the edges between sample points are determined based on the cosine similarity between the semantic features of the data. The higher the similarity, the greater the weight of the edge, and the more likely it is to become a neighboring sample point. Neighboring sample points are determined based on the semantic similarity between sample points. A predetermined number of sample points with the highest semantic similarity to each sample point are selected as its neighbors, thereby defining a range of samples with close semantic relationships. The neighboring sample points can be represented as the top-K nearest neighbors, which can be used for graph-based diffusion and information transfer simulation.

[0047] According to the embodiments provided in this application, in step S105: based on the transition probability distance between multiple sample points and the density value of each sample point, multiple data are deduplicated to obtain the deduplication result, including: Based on the density value and transition probability distance of each sample point, the semantic hub degree corresponding to multiple sample points is determined. The semantic hub degree represents the semantic representativeness of the corresponding sample point. Among multiple sample points, sample points with a semantic hub degree greater than a predetermined condition are identified and used as cluster center sample points. Delete all sample points except the cluster center sample point from multiple sample points to obtain the deduplication result.

[0048] In the embodiments provided in this application, the density value of each sample point reflects the degree of information accumulation, and the transition probability distance is used to measure the similarity between samples. By fusing these two features, the semantic hub degree corresponding to multiple sample points is determined. This semantic hub degree is used to represent the representativeness of the corresponding sample point at the semantic level and its separation degree from other sample points. Sample points with a semantic hub degree greater than a predetermined condition are selected from all sample points and designated as cluster center sample points. Other sample points besides the cluster center sample points are deleted to obtain the final deduplication result. The semantic hub degree, through the combination of density value and transition probability distance, quantifies the representativeness of the sample. A high density value indicates that the sample is an information convergence point, while the transition probability distance ensures its distinguishability from other core samples. Cluster centers can be directly determined and deduplication can be performed using the semantic hub degree as the standard, without relying on the cluster division of complete clustering, thus preserving core information and effectively eliminating redundancy. According to the embodiments provided in this application, the semantic hub degree corresponding to multiple sample points is determined based on the density value and transition probability distance of each sample point, including: For any second sample point among multiple sample points, there is at least one sample point whose density value is greater than that of the second sample point among the multiple sample points; Determine the minimum transition probability distance between the second sample point and at least one sample point, as the minimum distance; The semantic hub degree of the second sample point is determined based on its density value and minimum distance. The semantic pivot degree of multiple sample points is determined by using the method of determining the semantic pivot degree of the second sample point.

[0049] In the embodiments provided in this application, at least one sample point with a density value greater than that of the second sample point is found among all sample points. The sample point with the smallest transition probability distance to the second sample point is denoted as the minimum distance. The semantic hub degree of the second sample point is determined by combining its own density value and the minimum distance. The same method is used to determine the semantic hub degree corresponding to multiple sample points to complete deduplication. The minimum transition probability distance between the second sample point and the sample point with higher density can measure its relative independence in the semantic space. Combined with its own density value, the semantic hub degree reflects both the information aggregation ability of the sample and its distinguishability from other samples, which helps to select cluster center sample points with strong semantic representativeness. This method directly determines the density and distance relationship between samples without relying on complete clustering processing, achieving efficient data deduplication. For example, the density value of each sample point is calculated using step S103. and the transition probability distance defined in step S104 This allows us to further solve for the local cluster centers of the data samples, which can then be used as the deduplicated data.

[0050] As sample points The minimum transition probability distance with any other sample point j with a higher density, i.e., the aforementioned minimum distance, can be calculated as follows:

[0051] The cluster center sample point is surrounded by neighboring sample points with lower local density. By adding minimum distance... This allows the cluster center sample point to be relatively far from any locally high-density sample point, thus increasing semantic pivotality. The properties of the cluster center can be represented by the product of the density value of the sample point and the minimum distance:

[0052] in, Indicates the semantic hubness of sample point i. This represents the density value of sample point i. This represents the minimum distance to sample point i.

[0053] Based on the above embodiments and optional embodiments, this application also provides an optional implementation method, which is applied to the data cleaning stage in the offline knowledge base construction and data retrieval post-processing of the RAG system, and may include the following process.

[0054] Step S1, Data Feature Extraction. Based on a general open-source pre-trained word embedding model, the original data samples are converted into dense vector representations of fixed dimensions. Each sample is processed to obtain a d-dimensional feature vector. For a dataset containing N samples, a feature matrix of dimension N x d is finally generated. This matrix effectively captures the semantic information of the samples, laying the foundation for subsequent similarity calculations.

[0055] Step S2: Construct the state transition matrix. The state transition matrix can be viewed as a matrix of transition probability information. Based on the feature matrix obtained in step S1, calculate the cosine similarity between all sample pairs to form an initial similarity matrix. Then, normalize each row (representing the similarity of a sample to all other samples) (e.g., by applying row-wise Softmax or other normalization methods) to transform it into a probability distribution. This normalized matrix is ​​the state transition matrix P (N x N). In this matrix, the element P[i, j] represents the transition probability from sample i to sample j via a "random walk," intuitively reflecting the neighbor distribution structure of sample i.

[0056] Step S3, density estimation based on fast kernel diffusion NDDe. The density value of each sample point is calculated by simulating the multi-step diffusion process of information on the sample map. The density of a sample point depends not only on the similarity of its direct neighbor sample points, but also on its higher-order neighbors, which are neighbor sample points reachable through multi-step transitions.

[0057] The aforementioned rapid nuclear diffusion process can take the following forms:

[0058] in, Sample points In the The current density value in the step transition (i.e., the current value mentioned above), where I represents the identity matrix, is the density initialized from uniform. Initially, each sample carries initial information at the start of diffusion. The diffusion process continuously assigns the density value of each sample point to its top-K nearest neighbors, following the corresponding transition probabilities in the state transition matrix P (i.e., the matrix form of transition probability information), until it converges to a steady-state distribution. Because kernel diffusion density estimation, through a multi-step information diffusion process, makes semantically closely related sample points tend to have consistent density values, it can effectively reflect the degree of clustering of sample points in the semantic space, providing a basis for determining cluster centers, and enabling the sample points at the cluster center to better represent the core information of the samples within the cluster.

[0059] Step S4, distance metric based on transition probability TPDi. The similarity between samples is defined using the transition probability information of their common neighbors, reflecting the distance metric of sample similarity. For sample point i and sample point j, each sample point has some neighboring sample points. The neighboring sample points of sample point i are denoted as... The neighboring sample points of sample point j are denoted as , Let represent the common neighbor samples of the top-K nearest neighbors of sample points i and j. The first probability can be represented as the transition probability between sample i and its common neighbor samples, denoted as . The second probability can be represented as the transition probability between sample j and its common neighbor sample points, denoted as... c represents A common neighbor sample point in the array. The transition probability distance between sample i and sample j can be denoted as: , can be defined as: .

[0060] Compared to traditional cosine distance, TPDi significantly enhances the ability to distinguish between "positive sample pairs" (which should belong to the same cluster) and "negative sample pairs" (which should belong to different clusters). Its advantage lies in the higher separation between positive and negative sample pairs in the generated similarity distribution, which makes it easier to select a uniform and robust similarity threshold to determine whether samples belong to the same cluster in subsequent clustering steps.

[0061] Step S5: Data deduplication. The sample density is calculated using the data from step S3. The distance between the TPDi defined in step S4 can be used to further solve for the local cluster centers of the data samples, which can then be used as the deduplicated data.

[0062] calculate , as data points The minimum distance between it and any other point with a higher density.

[0063] calculate Based on the properties of local cluster centers: cluster centers are surrounded by neighbors with lower local density, and these neighbors are relatively far from any points with higher local density. This property applies to all cluster centers. Perform descending sorting and test hyperparameters. Settings, select The local cluster centers are used as the data retained after deduplication.

[0064] Through the above processing, the robustness of data deduplication is effectively improved by using techniques such as density estimation based on kernel diffusion and distance metric based on transition probability to address the problems of limited ability to identify redundant information and poor deduplication robustness in large-scale RAG system data processing.

[0065] Corresponding to the application scenarios and methods provided in the embodiments of this application, Figure 3 A flowchart of a data deduplication device according to an embodiment of this application is shown, such as... Figure 3 The embodiments shown in this application also provide a data deduplication device, including: The feature extraction module 301 is used to generate multiple sample points based on the semantic features corresponding to multiple data points, and the multiple sample points correspond one-to-one with the multiple data points. The transition probability determination module 302 is used to determine the transition probability information between each sample point and other sample points according to the semantic similarity between the semantic features corresponding to multiple data points. The transition probability information represents the probability of information transfer from each sample point to other sample points. The density determination module 303 is used to simulate information to perform multi-step transfers at each sample point according to the transfer probability information, and obtain the density value of each sample point. The density value represents a measure of the degree of information accumulation at each sample point after multi-step transfers. The distance determination module 304 is used to determine the transition probability distance between multiple sample points according to the transition probability information. The transition probability distance is used to represent the similarity between samples. The deduplication module 305 is used to deduplicatize multiple data based on the transition probability distance between multiple sample points and the density value of each sample point, and obtain the deduplication result.

[0066] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0067] Figure 4 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 4 As shown, the electronic device includes a memory 401 and a processor 402. The memory 401 stores a computer program that can run on the processor 402. When the processor 402 executes the computer program, it implements the method described in the above embodiments. The number of memories 401 and processors 402 can be one or more. In a specific implementation, the electronic device may also include a communication interface 403 for communicating with external devices and performing data exchange and transmission.

[0068] In practical implementation, if the memory 401, processor 402, and communication interface 403 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0069] Optionally, in a specific implementation, if the memory 401, processor 402 and communication interface 403 are integrated on a single chip, the memory 401, processor 402 and communication interface 403 can communicate with each other through an internal interface.

[0070] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0071] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0072] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0073] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0074] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0075] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0076] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0077] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0078] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0079] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0080] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0082] The above are merely exemplary embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data deduplication method, characterized in that, include: Based on the semantic features corresponding to multiple data points, multiple sample points are generated, and the multiple sample points correspond one-to-one with the multiple data points. Based on the semantic similarity between the semantic features corresponding to the multiple data points, the transfer probability information between each sample point and other sample points is determined, and the transfer probability information represents the probability of information transfer from each sample point to other sample points; Based on the transition probability information, the transition probability distance between the plurality of sample points is determined, and the transition probability distance is used to represent a measure of the similarity between samples; According to the transfer probability information, the simulated information undergoes multiple transfers at each sample point to obtain the density value of each sample point. The density value represents a measure of the degree of information accumulation at each sample point after multiple transfers. Based on the transition probability distance between the multiple sample points and the density value of each sample point, the multiple data are deduplicated to obtain the deduplication result.

2. The method according to claim 1, characterized in that, Determining the transition probability distance between the plurality of sample points according to the transition probability information includes: Based on the semantic similarity, neighboring sample points are determined for each sample point, and the neighboring sample points are a predetermined number of sample points with the highest semantic similarity to each sample point. Based on the transfer probability information, a first probability of information transfer between each sample point and a common neighbor point, and a second probability of information transfer between other sample points and the common neighbor point are determined; the common neighbor point is a neighbor sample point shared by each sample point and other sample points. Based on the first probability and the second probability, the transition probability distance between the plurality of sample points is determined.

3. The method according to claim 1, characterized in that, The step of simulating information through multiple transitions at each sample point according to the transition probability information to obtain the density value of each sample point includes: Based on the semantic similarity, neighboring sample points are determined for each sample point, and the neighboring sample points are a predetermined number of sample points with the highest semantic similarity to each sample point. The simulated information is transferred from each sample point to its corresponding neighbor sample point to obtain the single-step value of each sample point after performing a single-step transfer. Repeat the single-step transfer process until the single-step values ​​corresponding to the multiple sample points after the transfer meet the predetermined convergence conditions. Then, use the single-step values ​​corresponding to the multiple sample points after the transfer as the density values ​​corresponding to the multiple sample points.

4. The method according to claim 3, characterized in that, The simulated information is transferred from each sample point to its corresponding neighbor sample point to obtain the single-step value of each sample point after performing a single-step transfer, including: For any first sample point among the plurality of sample points, based on the current value of the first sample point, the current value of the corresponding neighbor sample point is updated according to the third probability indicated by the transfer probability information, and the single-step value of the neighbor sample point corresponding to the first sample point is determined; the third probability is the probability that information transfer occurs between the first sample point and the corresponding neighbor sample point. Based on the current value of the corresponding neighbor sample point, update the current value of the first sample point according to the third probability, and determine the single-step value of the first sample point; The single-step value of each sample point after single-step transition is obtained by obtaining the single-step value of the first sample point and the single-step value of the neighboring sample points corresponding to the first sample point.

5. The method according to any one of claim 2 or 3, characterized in that, The step of determining the neighboring sample points of each sample point based on the semantic similarity includes: Using the multiple sample points as nodes and the semantic similarity between the multiple sample points as edges, a sample graph is generated. In the sample graph, a predetermined number of sample points are selected as the neighbor sample points according to the semantic similarity between them and the sample points, from high to low.

6. The method according to claim 1, characterized in that, The process of deduplicating the multiple data points based on the transition probability distance between the multiple sample points and the density value of each sample point to obtain the deduplication result includes: Based on the density value and transition probability distance of each sample point, the semantic hub degree corresponding to each of the multiple sample points is determined, and the semantic hub degree represents the semantic representativeness of the corresponding sample point. Among the plurality of sample points, the sample points whose semantic hubness is greater than a predetermined condition are determined as cluster center sample points; The deduplication result is obtained by deleting all sample points except the cluster center sample point from the plurality of sample points.

7. The method according to claim 6, characterized in that, The determination of the semantic hub degree corresponding to each of the multiple sample points based on the density value and transition probability distance of each sample point includes: For any second sample point among the plurality of sample points, there is at least one sample point among the plurality of sample points whose density value is greater than that of the second sample point; The minimum transition probability distance between the second sample point and the at least one sample point is determined as the minimum distance; The semantic hub degree of the second sample point is determined based on the density value of the second sample point and the minimum distance. The semantic pivot degree corresponding to each of the plurality of sample points is determined by using the method of determining the semantic pivot degree of the second sample point.

8. The method according to any one of claims 1 to 4, 6 or 7, characterized in that, The data types of the multiple data include text, image, and voice.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images