Data deduplication method and system based on incremental calculation and feature clustering and electronic equipment

Abstract

The invention discloses a data deduplication method and system based on incremental calculation and feature clustering and electronic equipment, and the method comprises the steps: receiving multi-source heterogeneous text, image, audio and video data streams in real time, adding a timestamp to each data unit, and generating an input data set with a time attribute; performing multi-modal feature extraction on the input data set, and dynamically adjusting feature weights of the extracted multi-modal features through a sliding window model and a time decay factor to obtain an increment feature vector set with the weights; executing two-stage clustering based on the feature vector set to generate a plurality of fine-grained data clusters; and comparing the intra-cluster data of each fine-grained data cluster in pairs by adopting a composite similarity model to determine intra-cluster repeated data, and performing data deduplication on the input data set according to the intra-cluster repeated data to obtain a deduplicated input data set. According to the method, the processing efficiency of the real-time data is improved, and the storage and calculation cost is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the field of data storage and processing technology, and in particular to a data deduplication method, system and electronic device based on incremental computing and feature clustering. Background Art

[0002] Data deduplication is a data streamlining process that eliminates duplicate information and reduces redundant data. Its goal is to reduce an organization's ongoing storage needs, improve storage utilization, and lower costs. The most significant advantage of data deduplication is that removing irrelevant data reduces the total amount of data an organization must store and manage. By reducing the storage space occupied by data, it effectively improves an organization's storage capacity. Common data deduplication techniques include database deduplication, programmatic deduplication, and specialized data deduplication tools.

[0003] There are generally two ways in which data deduplication works. At the most basic level, it uses automated functions to identify duplicate content in data blocks and then delete it. By performing this across the data layer, unique data blocks can be analyzed and then saved to the data blocks that are worth saving. Another way to deduplicate data is to operate at the file level. Single-instance data stores compare complete copies within file data, but do not compare data blocks. Similar to the basic level method, file deduplication also relies on retaining the original file and deleting redundant copies. Traditional data deduplication methods face significant bottlenecks when dealing with dynamic data streams, multimodal heterogeneous data, and high-dimensional feature processing, as shown below:

[0004] Limitations of static algorithms: Traditional hash algorithms (such as the SHA series) rely on the full amount of data to calculate a unique identifier and cannot adapt to the feature drift problem in incremental data scenarios, resulting in an increased false positive rate for duplicate data.

[0005] Inefficient real-time processing: Similarity detection algorithms (such as Sim'Hash') have a time complexity of O(n 2 ), it is difficult to cope with high-throughput real-time data streams, such as IoT sensor data, with sampling rates greater than 1kHz.

[0006] Inefficient processing of high-dimensional features: Existing clustering algorithms (such as DBSCAN) face the curse of dimensionality when processing high-dimensional data. Memory usage and computation time increase exponentially with the dimension, making them unable to meet the needs of dynamic updates of large-scale data.

[0007] In response to the problems existing in the prior art mentioned above, this patent proposes a data deduplication method based on incremental computing and feature clustering. Summary of the Invention

[0008] In order to solve the technical problems existing in the background technology, the present invention proposes a data deduplication method based on incremental calculation and feature clustering.

[0009] In a first aspect, the present invention proposes a data deduplication method based on incremental computing and feature clustering, comprising the following steps:

[0010] S1, receiving text, image, audio and video data streams from multiple heterogeneous sources in real time, attaching timestamps to each data unit, and generating input datasets with time attributes;

[0011] S2. performing multimodal feature extraction on the input data set, and dynamically adjusting the feature weights of the extracted multimodal features using a sliding window model and a time decay factor to obtain a weighted incremental feature vector set;

[0012] S3. Perform two-stage clustering based on the feature vector set to generate multiple fine-grained data clusters, where the two-stage clustering includes Canopy coarse clustering and DBSCAN fine clustering;

[0013] S4. Using a composite similarity model, perform pairwise comparisons on the data within each fine-grained data cluster to identify duplicate data within the cluster, and perform data deduplication on the input dataset based on the duplicate data within the cluster to obtain a deduplicated input dataset.

[0014] S5. Write the deduplicated input data set into the storage system in timestamp order or push it to the downstream processing module.

[0015] Preferably, the performing multimodal feature extraction on the input data set specifically includes:

[0016] When the input data set is specifically text data, extracting term feature vectors using an improved TF-IDF algorithm, wherein the improved TF-IDF algorithm includes a term life cycle correction term;

[0017] When the input data set is specifically image data, the color histogram, gray level co-occurrence matrix texture features and shape features of the image data are extracted;

[0018] When the input data set is specifically audio data, extracting the Mel-frequency cepstral coefficients of the audio data;

[0019] When the input data set is specifically video data, inter-frame differences, optical flow motion features, and key frame content features of the video data are extracted.

[0020] Preferably, the feature weights of the extracted multimodal features are dynamically adjusted by a sliding window model and a time decay factor to obtain a weighted incremental feature vector set, specifically including:

[0021] Use the sliding window model to limit the real-time processing range and set the window time span Δ;

[0022] Within the window time, the feature weights of the extracted multimodal features are dynamically adjusted to generate an incremental feature vector set with temporal weights. The weights of text features are attenuated at the term level, while the weights of image or audio features are attenuated as a whole according to the data unit. Each feature vector in the incremental feature vector set contains the original features, weights and timestamps.

[0023] Preferably, step S3 specifically includes:

[0024] Using the Canopy coarse clustering algorithm to perform coarse clustering on the feature vector set to generate preliminary data clusters;

[0025] Based on the DBSCAN fine clustering algorithm, the preliminary data clusters are finely clustered to generate multiple fine-grained data clusters. Incremental updates are achieved by dynamically adjusting the core radius ∈ and the minimum number of samples MinPts. The adjustment formula is:

[0026] ∈ new =∈ old ×(1+γ×Δd);

[0027] MinPts new =MinPts old ×(1+δ×Δd);

[0028] Where γ and δ represent adjustment coefficients.

[0029] Preferably, the method of using the Canopy coarse clustering algorithm to perform coarse clustering on the feature vector set to generate preliminary data clusters specifically includes:

[0030] Calculate the similarity between the data points in the feature vector set based on the Jaccard similarity or cosine similarity method;

[0031] The similarity between the data points is compared with the preset similarity threshold, and the data points in the feature vector set are preliminarily divided into multiple canopies based on the comparison results. The formula is:

[0032]

[0033] Where D represents the dataset, C represents the Canopy center set, sim(d,c) represents the similarity between data point d and center c; τ is the preset similarity threshold.

[0034] Preferably, the method of using a composite similarity model to perform pairwise comparison on the data within each fine-grained data cluster to determine duplicate data within the cluster specifically includes:

[0035] Set the threshold θ of composite similarity;

[0036] The data within each fine-grained data cluster are compared pairwise. For each pair of data d1 and d2 within the cluster, the composite similarity is calculated using the formula:

[0037] Sim(d1,d2)=θ1×Jaccard(d1,d2)+θ2×Cosine(d1,d2)+θ3×TD(d1,d2);

[0038]

[0039] Among them, β represents the time attenuation coefficient; θ1, θ2, θ3 represent the weight coefficients;

[0040] If the composite similarity Sim(d1, d2) is higher than the threshold θ, the data d1 and d2 are determined to be duplicate data within the cluster.

[0041] Preferably, deduplicating the input data set according to the duplicate data within the cluster to obtain a deduplicated input data set specifically includes:

[0042] Determine the data stream corresponding to the duplicate data within the cluster in the input data set;

[0043] The input data set is deduplicated. For the data streams corresponding to the duplicate data in the cluster in the input data set, only the data stream with the latest timestamp is retained to obtain the deduplicated input data set.

[0044] Preferably, the improved TF-IDF algorithm specifically includes:

[0045] The improved TF-IDF algorithm introduces a term lifecycle correction factor into the calculation of the traditional TF-IDF algorithm. It dynamically adjusts the weight through the difference Δt between the term generation time and the current time and the decay coefficient α, so that new terms obtain higher weights, old terms decay over time, and the high discrimination characteristics of active terms in the window are retained.

[0046] In a second aspect, the present invention proposes a data deduplication system based on incremental computing and feature clustering, comprising:

[0047] The data acquisition module is used to receive text, image, audio and video data streams from multiple heterogeneous sources in real time, attach a timestamp to each data unit, and generate an input data set with time attributes;

[0048] an incremental feature generation module, configured to extract multimodal features from the input data set, and dynamically adjust the feature weights of the extracted multimodal features through a sliding window model and a time decay factor to obtain a weighted incremental feature vector set;

[0049] A clustering module, configured to perform two-stage clustering based on the feature vector set to generate a plurality of fine-grained data clusters, wherein the two-stage clustering includes Canopy coarse clustering and DBSCAN fine clustering;

[0050] A deduplication execution module is used to perform pairwise comparison of the intra-cluster data of each fine-grained data cluster using a composite similarity model to determine the duplicate data within the cluster, and to deduplicate the input data set based on the duplicate data within the cluster to obtain a deduplicated input data set;

[0051] The output module is used to write the deduplicated input data set into the storage system in timestamp order or push it to the downstream processing module.

[0052] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0053] The memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0055] In the present invention, the data deduplication method, system and electronic device based on incremental computing and feature clustering are proposed. Through dynamic feature weight adjustment and hybrid clustering mechanism, they can realize real-time and efficient processing of high-concurrency data streams, accurately identify redundant data, and dynamically adapt to changes in data distribution. It improves the response speed and processing efficiency of real-time data, significantly reduces storage and computing costs, and is adaptable to multimodal data scenarios such as text, images, audio and video. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a structural diagram of the workflow of a data deduplication method based on incremental calculation and feature clustering proposed by the present invention;

[0057] Figure 2 A schematic diagram of an implementation process of a data deduplication method based on incremental calculation and feature clustering proposed by the present invention;

[0058] Figure 3 This is a schematic diagram of an implementation architecture of a data deduplication method based on incremental computing and feature clustering proposed by the present invention;

[0059] Figure 4 This is a schematic diagram of the system architecture of a data deduplication system based on incremental computing and feature clustering proposed by the present invention;

[0060] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] Reference Figure 1-4 The present invention proposes a data deduplication method based on incremental calculation and feature clustering, comprising the following steps:

[0062] S1. Receive text, image, audio, and video data streams from multiple heterogeneous sources in real time, attach a timestamp to each data unit, and generate an input dataset with time attributes.

[0063] This embodiment can process data in multiple modalities, including text, images, audio, and video. For text data, a modified TF-IDF algorithm is used to extract term feature vectors; for images, color histograms, texture features, and shape features are extracted; for audio data, features such as Mel-Frequency Cepstral Coefficients (MFCCs) are extracted; and for video data, inter-frame differences, optical flow motion features, and keyframe content features are extracted. These diverse feature extraction methods ensure that the system can comprehensively capture key data information, providing a solid foundation for subsequent clustering and deduplication operations.

[0064] S2. Perform multimodal feature extraction on the input data set, and dynamically adjust the feature weights of the extracted multimodal features through a sliding window model and a time decay factor to obtain a weighted incremental feature vector set.

[0065] In this embodiment, multimodal feature extraction is performed on the input data set, specifically including:

[0066] When the input dataset is text data, the term feature vector is extracted using the improved TF-IDF algorithm, which includes a term life cycle correction term.

[0067] When the input data set is specifically image data, the color histogram, gray level co-occurrence matrix texture features and shape features of the image data are extracted;

[0068] When the input data set is specifically audio data, extracting the Mel-frequency cepstral coefficients of the audio data;

[0069] When the input data set is specifically video data, inter-frame differences, optical flow motion features, and key frame content features of the video data are extracted.

[0070] Specifically, the improved TF-IDF algorithm includes:

[0071] The improved TF-IDF algorithm introduces a term lifecycle correction factor into the calculation of the traditional TF-IDF algorithm. It dynamically adjusts the weight through the difference Δt between the term generation time and the current time and the decay coefficient α, so that new terms obtain higher weights, old terms decay over time, and the high discrimination characteristics of active terms in the window are retained.

[0072] In this embodiment, for text data, an improved TF-IDF algorithm is used to extract term feature vectors. The specific steps are as follows:

[0073] (1) Text domain processing: Perform preprocessing operations such as word segmentation, stop word removal, and stemming on text data to obtain a clean word sequence;

[0074] (2) Calculate word frequency (TF): Count the frequency of each word in the document. The formula is:

[0075]

[0076] (3) Calculate the inverse document frequency (IDF): This measures the prevalence of a term in a document collection. The formula is:

[0077]

[0078] Where N is the total number of documents and DF(t) is the number of documents containing term t.

[0079] (4) Calculate the term life cycle correction term (LC): Consider the time difference between the term generation time and the current time, the formula is:

[0080]

[0081] in, represents the decay coefficient, and Δt represents the time difference between the generation time of term t in document d and the current time.

[0082] (5) Calculate the improved TF-IDF value: Multiply TF, IDF, and LC to obtain the improved TF-IDF value. The formula is:

[0083] TF=IDF'(t,d)=TF(t,d)×IDF(t)×LC(t,d).

[0084] In this embodiment, for image data, color histogram, texture features, shape features, etc. are extracted. The specific steps are as follows:

[0085] (1) Color histogram extraction: Calculate the pixel distribution of each color channel in the image to obtain the color histogram features.

[0086] (2) Texture feature extraction: Gray-level co-occurrence matrix (GLCM) is used to calculate the texture features of the image, such as contrast, correlation, energy, and entropy.

[0087] (3) Shape feature extraction: Through edge detection and contour analysis, the shape features of objects in the image, such as area, perimeter, circularity, etc., are extracted.

[0088] In this embodiment, for audio data, features such as Mel-Frequency Cepstral Coefficients (MFCC) are extracted. The specific steps are as follows:

[0089] (1) Audio preprocessing: Perform preprocessing operations such as framing and windowing on the audio signal to obtain audio frames suitable for feature extraction.

[0090] (2) Calculate MFCC: Extract the MFCC features of the audio through Fourier transform, Mel filter bank and discrete cosine transform.

[0091] In this embodiment, for video data, inter-frame difference, optical flow motion features and key frame content features are comprehensively extracted. The specific steps are as follows:

[0092] (1) Inter-frame difference extraction: Calculate the pixel difference between adjacent video frames to obtain inter-frame difference features to reflect the dynamic changes of the video.

[0093] (2) Optical flow motion feature extraction: Use the optical flow algorithm to calculate the motion trajectory of objects in the video and obtain the optical flow motion features.

[0094] (3) Key frame content feature extraction: Extract key frames from the video and perform image feature extraction on the key frames, such as color histogram, texture features, etc., to obtain key frame content features.

[0095] In this embodiment, the feature weights of the extracted multimodal features are dynamically adjusted through a sliding window model and a time decay factor to obtain a weighted incremental feature vector set, specifically including:

[0096] Use the sliding window model to limit the real-time processing range and set the window time span Δ;

[0097] Within the window time, the feature weights of the extracted multimodal features are dynamically adjusted to generate an incremental feature vector set with temporal weights. The weights of text features are attenuated at the term level, while the weights of image or audio features are attenuated as a whole according to the data unit. Each feature vector in the incremental feature vector set contains the original features, weights and timestamps.

[0098] Specifically, based on the sliding window model and the time decay factor, the feature weights are dynamically adjusted. The specific steps are as follows:

[0099] (1) Set the sliding window: define the time range of data processing as [t-Δ,t], where t represents the current time and Δ represents the window size.

[0100] (2) Calculate the time decay factor: Calculate the time decay factor λ based on the time interval of the data. The formula is:

[0101] λ=e -βΔt ;

[0102] Where β represents the attenuation coefficient and Δt represents the time interval of the data.

[0103] (3) Update feature weights: For new data, update feature weights according to the time decay factor. The formula is:

[0104] W new =W old ×λ Δt ;

[0105] Among them, W new Represents the feature weight of new data, W old Represents the feature weight of old data.

[0106] (4) Improved TF-IDF algorithm: During the incremental calculation process, the improved TF-IDF algorithm is continued to be used, and the term life cycle correction term is added to improve the accuracy of semantic similarity detection.

[0107] In this embodiment, the system can process data streams in real time by combining a sliding window model with a time decay factor. The sliding window defines the time range for data processing, ensuring that the system focuses only on recent, temporally relevant data. The time decay factor increases the feature weight of new data and decreases the weight of older data, allowing the system to quickly adapt to data changes and improving both responsiveness and processing efficiency for real-time data.

[0108] The improved TF-IDF algorithm incorporates a term lifecycle correction term to more accurately capture the changes in term importance over time. This enables the system to efficiently extract discriminative and representative features when processing large amounts of text data, reducing data processing redundancy and improving overall processing efficiency.

[0109] S3. Perform two-stage clustering based on the feature vector set to generate multiple fine-grained data clusters. The two-stage clustering includes Canopy coarse clustering and DBSCAN fine clustering.

[0110] In this embodiment, step S3 specifically includes:

[0111] The Canopy coarse clustering algorithm is used to perform coarse clustering on the feature vector set to generate preliminary data clusters;

[0112] Based on the DBSCAN fine clustering algorithm, the preliminary data clusters are finely clustered to generate multiple fine-grained data clusters. Incremental updates are achieved by dynamically adjusting the core radius ∈ and the minimum number of samples MinPts. The adjustment formula is:

[0113] ∈ new =∈ old ×(1+γ×Δd);

[0114] MinPts new =MinPts old ×(1+δ×Δd);

[0115] Where γ and δ represent adjustment coefficients.

[0116] In this embodiment, the Canopy coarse clustering algorithm is used to perform coarse clustering on the feature vector set to generate preliminary data clusters, specifically including:

[0117] Calculate the similarity between data points in the feature vector set based on Jaccard similarity or cosine similarity method;

[0118] The similarity between the data points is compared with the preset similarity threshold, and the data points in the feature vector set are preliminarily divided into multiple canopies based on the comparison results. The formula is:

[0119]

[0120] Where D represents the dataset, C represents the Canopy center set, sim(d,c) represents the similarity between data point d and center c; τ is the preset similarity threshold.

[0121] Specifically, a hybrid model combining Canopy coarse clustering and DBSCAN fine clustering is used to achieve efficient clustering. DBSCAN clustering fine grouping:

[0122] (1) For each data point in Canopy, use the DBSCAN algorithm to perform fine clustering.

[0123] (2) Set the neighborhood radius ∈ and the minimum number of samples MinPts, and calculate the neighborhood of the data point. The formula is:

[0124] ∈-domain(d)={d'|d'∈D,dist(d,d')≤∈}

[0125] (3) According to the density of data points in the neighborhood, the data points are divided into core points, boundary points and noise points.

[0126] (4) Connect the data points that are density-reachable to form the final cluster.

[0127] According to the data distribution change rate Δd, the neighborhood radius ∈ and the minimum number of samples MinPts are dynamically adjusted.

[0128] S4. Use a composite similarity model to perform pairwise comparisons on the intra-cluster data of each fine-grained data cluster to determine the duplicate data within the cluster, and deduplicate the input data set based on the duplicate data within the cluster to obtain a deduplicated input data set.

[0129] In this embodiment, a composite similarity model is used to perform pairwise comparisons on the data within each fine-grained data cluster to determine duplicate data within the cluster, specifically including:

[0130] Set the threshold θ of composite similarity;

[0131] The data within each fine-grained data cluster are compared pairwise. For each pair of data d1 and d2 within the cluster, the composite similarity is calculated using the formula:

[0132] Sim(d1,d2)=θ1×Jaccard(d1,d2)+θ2×Cosine(d1,d2)+θ3×TD(d1,d2);

[0133]

[0134] Among them, β represents the time attenuation coefficient; θ1, θ2, θ3 represent the weight coefficients;

[0135] If the composite similarity Sim(d1, d2) is higher than the threshold θ, the data d1 and d2 are determined to be duplicate data within the cluster.

[0136] In this embodiment, deduplication of the input data set is performed based on the duplicate data within the cluster to obtain a deduplicated input data set, specifically including:

[0137] Determine the data stream corresponding to the duplicate data within the cluster in the input data set;

[0138] The input data set is deduplicated. For the data streams corresponding to the duplicate data in the cluster in the input data set, only the data stream with the latest timestamp is retained to obtain the deduplicated input data set.

[0139] Specifically, the threshold θ of the composite similarity is set according to the application scenario and data characteristics.

[0140] S5. Write the deduplicated input data set into the storage system in timestamp order or push it to the downstream processing module.

[0141] Specifically, the deduplicated data is output to the storage system or subsequent data processing link to complete the entire data deduplication process.

[0142] In this embodiment, Canopy clustering first performs a preliminary classification of the data to reduce the data size and improve processing efficiency; DBSCAN clustering connects data points based on density and can effectively identify clusters of any feature type. Through this hybrid model, the system can accurately classify similar data into one category in large-scale data, thereby improving the accuracy of deduplication. The dynamic update rules of the core radius and the minimum number of samples MinPts in the clustering module enable the system to automatically adjust the clustering parameters according to changes in data distribution. This dynamic adaptability enables the system to maintain good clustering effects without manual intervention when facing data from different fields and different characteristics, thereby enhancing the versatility and flexibility of the system. The system as a whole is versatile and flexible, and adapts to the characteristics of data in multiple fields. Whether it is text, image, audio or video data, the system can effectively process and deduplicate through corresponding feature extraction methods and clustering models, expanding a wide range of application scenarios.

[0143] In this embodiment, through precise feature extraction, efficient clustering, and accurate similarity determination, this patent can effectively identify and remove duplicate parts in the data. In large-scale data processing scenarios, such as finance and the Internet of Things, this efficient deduplication method can significantly reduce data storage requirements and lower storage costs. At the same time, it improves data purity and quality, providing a higher-quality data foundation for subsequent data analysis and applications.

[0144] Reference Figure 1-4 The present invention proposes a data deduplication system based on incremental computing and feature clustering, comprising:

[0145] The data acquisition module is used to receive text, image, audio and video data streams from multiple heterogeneous sources in real time, attach a timestamp to each data unit, and generate an input data set with time attributes;

[0146] The incremental feature generation module is used to extract multimodal features from the input data set and dynamically adjust the feature weights of the extracted multimodal features through a sliding window model and a time decay factor to obtain a weighted incremental feature vector set;

[0147] The clustering module is used to perform two-stage clustering based on the feature vector set to generate multiple fine-grained data clusters. The two-stage clustering includes Canopy coarse clustering and DBSCAN fine clustering.

[0148] The deduplication execution module is used to use a composite similarity model to perform pairwise comparison on the intra-cluster data of each fine-grained data cluster to determine the duplicate data within the cluster, and to deduplicate the input data set based on the duplicate data within the cluster to obtain a deduplicated input data set.

[0149] In this embodiment, the deduplication execution module uses a composite similarity determination model that integrates Jaccard similarity, cosine similarity, and a time decay correction term. By dynamically optimizing weights through an online learning mechanism, this multi-dimensional similarity calculation method can more comprehensively measure the similarity between data, effectively improving the accuracy of data deduplication.

[0150] The output module is used to write the deduplicated input data set into the storage system in timestamp order or push it to the downstream processing module.

[0151] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 3 The electronic device 20 may include a memory 21 and a processor 22. For example, the memory 21 and the processor 22 are interconnected via a bus 23.

[0152] The memory 21 is used to store computer-executable instructions;

[0153] The processor 22 is configured to execute the computer-executable instructions stored in the memory, so as to enable the control device 20 to execute the method shown in the above method embodiment.

[0154] The electronic device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principles and beneficial effects are similar, which will not be repeated here.

[0155] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A data deduplication method based on incremental calculation and feature clustering, characterized in that: The following steps are involved: S1, receiving text, image, audio and video data streams from multiple heterogeneous sources in real time, attaching timestamps to each data unit, and generating input datasets with time attributes; S2. performing multimodal feature extraction on the input data set, and dynamically adjusting the feature weights of the extracted multimodal features using a sliding window model and a time decay factor to obtain a weighted incremental feature vector set; S3. Perform two-stage clustering based on the feature vector set to generate multiple fine-grained data clusters, where the two-stage clustering includes Canopy coarse clustering and DBSCAN fine clustering; S4. Using a composite similarity model, perform pairwise comparisons on the data within each fine-grained data cluster to identify duplicate data within the cluster, and perform data deduplication on the input dataset based on the duplicate data within the cluster to obtain a deduplicated input dataset. S5. Write the deduplicated input data set into the storage system in timestamp order or push it to the downstream processing module.

2. The data deduplication method based on incremental calculation and feature clustering according to claim 1, characterized in that: The performing multimodal feature extraction on the input data set specifically includes: When the input data set is specifically text data, extracting term feature vectors using an improved TF-IDF algorithm, wherein the improved TF-IDF algorithm includes a term life cycle correction term; When the input data set is specifically image data, the color histogram, gray level co-occurrence matrix texture features and shape features of the image data are extracted; When the input data set is specifically audio data, extracting the Mel-frequency cepstral coefficients of the audio data; When the input data set is specifically video data, inter-frame differences, optical flow motion features, and key frame content features of the video data are extracted.

3. The data deduplication method based on incremental calculation and feature clustering according to claim 1, characterized in that: The feature weights of the extracted multimodal features are dynamically adjusted by the sliding window model and the time decay factor to obtain a weighted incremental feature vector set, specifically including: Use the sliding window model to limit the real-time processing range and set the window time span Δ; Within the window time, the feature weights of the extracted multimodal features are dynamically adjusted to generate an incremental feature vector set with temporal weights. The weights of text features are attenuated at the term level, while the weights of image or audio features are attenuated as a whole according to the data unit. Each feature vector in the incremental feature vector set contains the original features, weights and timestamps.

4. The data deduplication method based on incremental calculation and feature clustering according to claim 1, characterized in that: Step S3 specifically includes: Using the Canopy coarse clustering algorithm to perform coarse clustering on the feature vector set to generate preliminary data clusters; Based on the DBSCAN fine clustering algorithm, the preliminary data clusters are finely clustered to generate multiple fine-grained data clusters. Incremental updates are achieved by dynamically adjusting the core radius ∈ and the minimum number of samples MinPts. The adjustment formula is: ∈ new =∈ old ×(1+γ×Δd); MinPts new =MinPts old ×(1+δ×Δd); Where γ and δ represent adjustment coefficients.

5. The data deduplication method based on incremental calculation and feature clustering according to claim 4, characterized in that: The method of using the Canopy coarse clustering algorithm to perform coarse clustering on the feature vector set to generate preliminary data clusters specifically includes: Calculate the similarity between the data points in the feature vector set based on the Jaccard similarity or cosine similarity method; The similarity between the data points is compared with the preset similarity threshold, and the data points in the feature vector set are preliminarily divided into multiple canopies based on the comparison results. The formula is: Where D represents the dataset, C represents the Canopy center set, sim(d,c) represents the similarity between data point d and center c; τ is the preset similarity threshold.

6. The data deduplication method based on incremental calculation and feature clustering according to claim 1, characterized in that: The composite similarity model is used to perform pairwise comparison of the data within each fine-grained data cluster to determine the duplicate data within the cluster, specifically including: Set the threshold θ of composite similarity; The data within each fine-grained data cluster are compared pairwise. For each pair of data d1 and d2 within the cluster, the composite similarity is calculated using the formula: Sim(d1,d2)=θ1×Jaccard(d1,d2)+θ2×Cosine(d1,d2)+θ3×TD(d1,d2); Among them, β represents the time attenuation coefficient; θ1, θ2, θ3 represent the weight coefficients; If the composite similarity Sim(d1, d2) is higher than the threshold θ, the data d1 and d2 are determined to be duplicate data within the cluster.

7. The data deduplication method based on incremental calculation and feature clustering according to claim 1, characterized in that: Deduplicating the input data set according to the duplicate data within the cluster to obtain a deduplicated input data set specifically includes: Determine the data stream corresponding to the duplicate data within the cluster in the input data set; The input data set is deduplicated. For the data streams corresponding to the duplicate data in the cluster in the input data set, only the data stream with the latest timestamp is retained to obtain the deduplicated input data set.

8. The data deduplication method based on incremental calculation and feature clustering according to claim 2, characterized in that: The improved TF-IDF algorithm specifically includes: The improved TF-IDF algorithm introduces a term lifecycle correction factor into the calculation of the traditional TF-IDF algorithm. It dynamically adjusts the weight through the difference Δt between the term generation time and the current time and the decay coefficient α, so that new terms obtain higher weights, old terms decay over time, and the high discrimination characteristics of active terms in the window are retained.

9. A data deduplication system based on incremental computing and feature clustering, characterized in that: include: The data acquisition module is used to receive text, image, audio and video data streams from multiple heterogeneous sources in real time, attach a timestamp to each data unit, and generate an input data set with time attributes; an incremental feature generation module, configured to extract multimodal features from the input data set, and dynamically adjust the feature weights of the extracted multimodal features through a sliding window model and a time decay factor to obtain a weighted incremental feature vector set; A clustering module, configured to perform two-stage clustering based on the feature vector set to generate a plurality of fine-grained data clusters, wherein the two-stage clustering includes Canopy coarse clustering and DBSCAN fine clustering; A deduplication execution module is used to perform pairwise comparison of the intra-cluster data of each fine-grained data cluster using a composite similarity model to determine the duplicate data within the cluster, and to deduplicate the input data set based on the duplicate data within the cluster to obtain a deduplicated input data set; The output module is used to write the deduplicated input data set into the storage system in timestamp order or push it to the downstream processing module.

10. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

Images