Data deduplication method and device based on large model and local hash algorithm, equipment and medium

Abstract

The invention discloses a data deduplication method and device based on a large model and a local hash algorithm, equipment and a medium, and relates to the technical field of data deduplication, and the method comprises the steps: carrying out the preprocessing operation of obtained to-be-detected data to obtain target data, and carrying out the semantic feature extraction of the target data based on a preset large model, obtaining a target high-dimensional semantic feature vector corresponding to the target data; processing the dimensionality-reduced target high-dimensional semantic feature vectors by using a preset locality sensitive hash algorithm to obtain target hash values respectively corresponding to the dimensionality-reduced target high-dimensional semantic feature vectors; determining expected similar data pairs from the target data based on the target hash values, and determining a Hamming distance between the expected similar data pairs according to the target hash values corresponding to two pieces of data in the expected similar data pairs; and judging whether the data in the expected similar data pair is repeated or not by utilizing the Hamming distance to obtain a judgment result, and performing data deduplication on the repeated data in the to-be-detected data based on the judgment result.

Description

Technical Field

[0001] This application relates to the technical field of data deduplication, and particularly to a data deduplication method, device, equipment and medium based on a large model and a local hashing algorithm. Background Art

[0002] With the rapid development of information technology, large AI models have been widely applied in fields such as aviation, transportation, agriculture, education, and government affairs. In order to improve the application quality of large models in various fields, it is necessary to search for a large amount of industry-related data for training. However, a large amount of duplicate data will appear in the massive data, and the large amount of duplicate data will lead to a decline in the performance of the model.

[0003] Traditional data deduplication is usually based on keyword matching or simple hashing algorithms. The traditional methods have the following limitations: Insufficient deduplication accuracy: The method based on keyword matching cannot deeply understand the intrinsic semantics of the data and is easily affected by factors such as polysemous words and synonyms, resulting in misjudgment or missed judgment. Low processing efficiency: With the rapid growth of the data scale, when traditional methods process large-scale data, the computational complexity is relatively high, resulting in a long time-consuming deduplication process. For example, in the case of simple hashing algorithms, when facing high-dimensional data, it is easily affected by the "curse of dimensionality", further reducing the processing efficiency. Lack of generality: Traditional methods are often designed for specific types of data and lack broad applicability. In addition, when traditional methods process multi-language and multi-format data, a large amount of customization is also required. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a data deduplication method, device, equipment and medium based on a large model and a local hashing algorithm, which extracts the deep semantic features of the data through the large model and combines the locality-sensitive hashing algorithm to achieve efficient and accurate data deduplication. The specific solutions are as follows: In the first aspect, this application provides a data deduplication method based on a large model and a local hashing algorithm, including: Performing a preprocessing operation on the obtained data to be detected to obtain target data, and extracting semantic features of the target data based on a preset large model to obtain a target high-dimensional semantic feature vector corresponding to the target data; Processing the reduced-dimensional target high-dimensional semantic feature vector by using a preset locality-sensitive hashing algorithm to obtain target hash values corresponding to the reduced-dimensional target high-dimensional semantic feature vectors respectively; Determining expected similar data pairs from the target data based on the target hash values, and determining the Hamming distance between the expected similar data pairs according to the target hash values respectively corresponding to the two data in the expected similar data pairs; Use the Hamming distance to determine whether the data in the expected similar data pairs is repeated to obtain a judgment result, and perform data deduplication on the duplicate data in the data to be detected based on the judgment result.

[0005] Optionally, the preprocessing operation on the obtained data to be detected to obtain target data includes: For the text data in the data to be detected, perform text cleaning and word segmentation on the text data to obtain first text data, remove special characters and HTML tags in the first text data to obtain second text data, and unify the encoding format of the second text data using a preset encoding format to obtain the target text data corresponding to the text data; wherein, the special characters include punctuation marks and stop words; For the image data in the data to be detected, adjust the image data based on a preset resolution to obtain first image data, adjust the color mode of the first image data to a preset color mode to obtain second image data, and perform normalization processing on the pixel values of the second image data to obtain the target image data corresponding to the image data; For the audio data in the data to be detected, adjust the audio sampling rate in the audio data to a fixed value based on a preset sampling rate, and perform noise removal processing on the adjusted audio data to obtain the target audio data corresponding to the audio data.

[0006] Optionally, the processing of the dimension-reduced target high-dimensional semantic feature vector using a preset locality-sensitive hashing algorithm to obtain target hash values corresponding to the dimension-reduced target high-dimensional semantic feature vector respectively includes: Adjust the dimension-reduced target high-dimensional semantic feature vector based on a preset target dimension to obtain a target feature vector; Use the Simhash algorithm to determine the target hash values corresponding to the target feature vector respectively, store the target hash values and the corresponding data to be detected in the form of key-value pairs to establish an index, and query and deduplicate the data based on the index.

[0007] Optionally, after adjusting the dimension-reduced target high-dimensional semantic feature vector based on a preset target dimension to obtain a target feature vector, it further includes: Perform block processing on the target feature vector, and use the Simhash algorithm in parallel on the block-processed target feature vector to determine the target hash values corresponding to the block-processed target feature vector.

[0008] Optionally, the data deduplication method based on the large model and the local hashing algorithm further includes: For any target hash value, divide the target hash value into a preset number of hash value segments, and store each of the hash value segments using KV encoding; wherein, each of the hash value segments after KV encoding consists of a K value and a V value.

[0009] Optionally, the determining of the expected similar data pairs from the target data based on the target hash value includes: For any two target hash values, compare the two hash value segments in each pair of hash values at the same hash value segment position in the two target hash values; If there is a first pair of hash values with the same K value among the pairs of hash values, compare whether there is a second pair of hash values with the same V value among the first pairs of hash values; If there is a second pair of hash values with the same V value, use the two data to be detected corresponding to the two target hash values as the expected similar data pairs. If there is no second pair of hash values with the same V value, determine whether the difference between the V values of the two target hash values exceeds the expected difference threshold. If the difference between the V values of the two target hash values does not exceed the expected difference threshold, use the two data to be detected corresponding to the two target hash values as the expected similar data pairs.

[0010] Optionally, the using of the Hamming distance to determine whether the data in the expected similar data pairs is repeated to obtain a judgment result includes: If the Hamming distance is less than the preset distance threshold, determine that the data in the expected similar data pairs is repeated, and obtain a judgment result indicating data repetition; If the Hamming distance is not less than the preset distance threshold, determine that the data in the expected similar data pairs is not repeated, and obtain a judgment result indicating data non - repetition.

[0011] In a second aspect, the present application provides a data deduplication device based on a large model and a local hashing algorithm, including: A vector acquisition module, configured to perform a pre - processing operation on the acquired data to be detected to obtain target data, and extract semantic features of the target data based on a preset large model to obtain a target high - dimensional semantic feature vector corresponding to the target data; A hash value acquisition module, configured to process the reduced - dimensional target high - dimensional semantic feature vector using a preset locality - sensitive hashing algorithm to obtain target hash values respectively corresponding to the reduced - dimensional target high - dimensional semantic feature vector; A Hamming distance determination module, configured to determine expected similar data pairs from the target data based on the target hash value, and determine the Hamming distance between the expected similar data pairs according to the target hash values respectively corresponding to the two data in the expected similar data pairs; A data deduplication module, configured to determine whether the data in the expected similar data pairs is duplicated by using the Hamming distance to obtain a judgment result, and perform data deduplication on the duplicated data in the data to be detected based on the judgment result.

[0012] In a third aspect, the present application provides an electronic device, including: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the foregoing data deduplication method based on a large model and a locality-sensitive hashing algorithm.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program, wherein when the computer program is executed by a processor, the foregoing data deduplication method based on a large model and a locality-sensitive hashing algorithm is implemented.

[0014] In the present application, preprocessing operations are performed on the acquired data to be detected to obtain target data, and semantic feature extraction is performed on the target data based on a preset large model to obtain a target high-dimensional semantic feature vector corresponding to the target data; a preset locality-sensitive hashing algorithm is used to process the dimension-reduced target high-dimensional semantic feature vector to obtain target hash values corresponding to the dimension-reduced target high-dimensional semantic feature vectors respectively; expected similar data pairs are determined from the target data based on the target hash values, and the Hamming distance between the expected similar data pairs is determined according to the target hash values respectively corresponding to the two data in the expected similar data pairs; the Hamming distance is used to determine whether the data in the expected similar data pairs is duplicated to obtain a judgment result, and data deduplication is performed on the duplicated data in the data to be detected based on the judgment result. As can be seen from the above, the present application performs semantic feature extraction on target data based on a preset large model to obtain a target high-dimensional semantic feature vector corresponding to the target data, which can accurately identify duplicated data with similar semantics but different expression forms in the target data, significantly improving the deduplication accuracy; by combining a preset locality-sensitive hashing algorithm and a dimension reduction technique, the computational complexity is significantly reduced, realizing efficient processing of large-scale data; at the same time, by combining the general feature extraction ability of the large model and the locality-sensitive hashing algorithm, it can be applied to deduplication tasks of various data types such as text, images, and audio, with strong generality. Description of the Drawings

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0016] Figure 1 Flowchart of a data deduplication method based on a large model and a local hashing algorithm disclosed in this application; Figure 2 Schematic diagram of the pigeonhole principle; Figure 3 Schematic diagram of the design of hash values and Hamming distances disclosed in this application; Figure 4 Schematic diagram of the design of the Text-dedup function disclosed in this application; Figure 5 Schematic diagram of a specific data duplication checking and deduplication processing flow disclosed in this application; Figure 6 Schematic diagram of the structure of a data deduplication device based on a large model and a local hashing algorithm disclosed in this application; Figure 7 Schematic diagram of the structure of an electronic device disclosed in this application. Detailed implementation manners

[0017] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0018] Traditional data deduplication is usually based on keyword matching or simple hashing algorithms. The traditional methods have the following limitations: insufficient deduplication accuracy: Methods based on keyword matching cannot deeply understand the intrinsic semantics of data and are easily affected by factors such as polysemes and synonyms, resulting in misjudgments or missed judgments; low processing efficiency: With the rapid growth of data scale, traditional methods have a high computational complexity when processing large-scale data, resulting in a long time-consuming deduplication process; insufficient generality: Traditional methods are often designed for specific types of data and lack broad applicability. In addition, traditional methods also require a large amount of customized adjustments when processing multi-language and multi-format data. For this reason, this application provides a data deduplication method based on a large model and a local hashing algorithm, which can extract the deep semantic features of data through the large model and combine the locality-sensitive hashing algorithm to achieve efficient and accurate data deduplication.

[0019] See Figure 1 As shown, the embodiments of this application disclose a data deduplication method based on a large model and a local hashing algorithm, including: Step S11: Perform a preprocessing operation on the obtained data to be detected to obtain target data, and extract semantic features of the target data based on a preset large model to obtain a target high-dimensional semantic feature vector corresponding to the target data.

[0020] In this embodiment, first, preprocessing operations such as cleaning and standardizing the acquired data to be detected are performed. The purpose of data preprocessing is to remove noise, unify the data format, handle missing values, ensure data quality, etc., so as to lay a foundation for subsequent semantic feature extraction and data deduplication. For different data types, different preprocessing operations can be performed to obtain the target data, which may specifically include: For the text data in the data to be detected, text cleaning and word segmentation are performed on the text data to obtain the first text data, special characters and HTML (HyperText Markup Language) tags in the first text data are removed to obtain the second text data, and the encoding format of the second text data is unified using a preset encoding format to obtain the target text data corresponding to the text data; among them, special characters include punctuation marks and stop words. For example, libraries such as NLTK (Natural Language Toolkit), Jieba, and HanLP (Han Language Processing, a multilingual natural language processing toolkit for production environments) in Python (Python Programming Language, a high-level programming language) can be used to implement text cleaning and word segmentation; regular expressions (Regex, i.e., Regular Expression) can be used to remove special characters and HTML tags; the Pandas library can be used to handle missing values in the text data to ensure data consistency and coherence; finally, the text data can be uniformly converted to the UTF-8 (i.e., 8-bit Universal Character Set / Unicode Transformation Format) encoding format to unify the encoding format.

[0021] For the image data in the data to be detected, the image data is adjusted based on a preset resolution to obtain the first image data, and the color mode of the first image data is adjusted to a preset color mode to obtain the second image data. The pixel values of the second image data are normalized to obtain the target image data corresponding to the image data. For example, the image resolution can be uniformly adjusted to a fixed size (such as 224x224) to meet the input requirements of the preset large model; the image can be uniformly converted to the RGB (Red Green Blue) color mode to ensure color consistency, and the Pandas library can be used to handle missing values in the image data; finally, the pixel values of the image can be normalized to the range [0,1] or standardized to a distribution with a mean of 0 and a variance of 1 to accelerate the training speed of the preset large model and improve the generalization ability of the preset large model. Among them, the OpenCV (Open Source Computer Vision Library, an open-source computer vision library) or PIL (Python Imaging Library, an image processing library) library can be used to implement image size adjustment and color mode conversion; the NumPy (Numerical Python, a Python library for processing arrays) library can be used to implement pixel value normalization and standardization; the data augmentation tools of TensorFlow or PyTorch can be used to implement image augmentation, such as using TensorFlow or PyTorch to load the pre-trained preset vision large model to provide high-quality, diverse, and model-processable image data for the preset large model.

[0022] For the audio data in the data to be detected, the audio sampling rate in the audio data is adjusted to a fixed value based on a preset sampling rate for subsequent processing and analysis, and the adjusted audio data is denoised to obtain the target audio data corresponding to the audio data to improve the clarity and intelligibility of the audio and make the useful information in the audio more prominent. For example, the audio sampling rate can be uniformly adjusted to a fixed value (such as 16kHz) to ensure the same audio length; a noise reduction algorithm (such as spectral subtraction) can be used to remove background noise. Among them, the Librosa library can be used to implement audio sampling rate adjustment, noise removal, etc.; the NumPy library can be used to implement the normalization process of audio features to convert them into a format suitable for the input of the preset large model.

[0023] Then, based on the preset large model, deep semantic feature extraction is performed on the target data to capture the semantic information of the data, and the target data is converted into a high-dimensional semantic feature vector for subsequent dimensionality reduction and data deduplication. For example: For the target text data, a pre-trained preset large model can be utilized, and forward propagation and feature extraction of the preset large model can be implemented using PyTorch or TensorFlow to extract the semantic feature vector of the target text data and capture the deep semantic information of the text; then the extracted semantic feature vector can be saved using the NumPy library.

[0024] For the target image data, a convolutional neural network (CNN, i.e., Convolutional Neural Networks) or a pre-trained vision large model, such as ResNet (Residual Network) or VGG (Visual Geometry Group, a deep convolutional neural network model), can be used to extract the visual feature vector of the image; then the extracted visual feature vector can be saved using the NumPy library.

[0025] For the target audio data, a large model for audio feature extraction (such as VGGish) or an audio encoder (such as Wav2Vec) can be used to extract the acoustic features of the audio, or the acoustic features of the audio can be extracted using the Librosa library; then the extracted acoustic feature vector can be saved using the NumPy library.

[0026] It should be noted that in this embodiment, rapid integration of new large models, such as Transformer-based vision models and audio models, is also supported.

[0027] In this way, by combining multiple pre-trained large models (such as ResNet and VGG), deep semantic feature extraction of various data types such as text, image, and audio is achieved, and the target high-dimensional semantic feature vector corresponding to the target data is obtained, which has high efficiency, generality, and scalability, and can provide high-quality semantic feature input for subsequent dimensionality reduction and data deduplication processing; at the same time, it can accurately identify duplicate data with similar semantics but different expression forms in the target data, significantly improving the deduplication accuracy. For example, for text data, it can identify "I like apples" and "Apples are my favorite" as duplicate content, for image data, it can identify the same image under different resolutions or lighting conditions, and for audio data, it can identify the same audio under different sampling rates or background noises.

[0028] Step S12: Process the dimensionality-reduced target high-dimensional semantic feature vector using a preset locality-sensitive hashing algorithm to obtain the target hash values corresponding to the dimensionality-reduced target high-dimensional semantic feature vector respectively.

[0029] In this embodiment, the preset locality-sensitive hashing algorithm is used to process the dimension-reduced target high-dimensional semantic feature vector to obtain the target hash values corresponding to the dimension-reduced target high-dimensional semantic feature vector respectively, which may include: First, the dimension-reduced target high-dimensional semantic feature vector is adjusted based on a preset target dimension to obtain a target feature vector; then the Simhash algorithm is used to determine the target hash values corresponding to the target feature vectors respectively, and the target hash values and the corresponding data to be detected are stored in the form of key-value pairs to establish an index, so as to query and deduplicate the data based on the index. Among them, after the dimension-reduced target high-dimensional semantic feature vector is adjusted based on a preset target dimension to obtain a target feature vector, it may further include: performing block processing on the target feature vector, so as to parallelly use the Simhash algorithm to determine the target hash values corresponding to the block-processed target feature vectors for the block-processed target feature vectors.

[0030] It should be noted that for any target hash value, the target hash value can be divided into a preset number of hash value segments, and each hash value segment is stored using KV encoding; among them, each hash value segment after KV encoding consists of a K value and a V value.

[0031] Before using the Simhash algorithm to determine the target hash values corresponding to the target feature vectors respectively, first, it should be checked whether the dimensions of the target feature vectors are consistent (such as 64 dimensions or 128 dimensions), and it should be ensured that the values of the target feature vectors have been normalized to a fixed range (such as [0, 1]). Among them, the NumPy library can be used to check the dimensions and formats of the target feature vectors; the multi-process or multi-thread library of Python is used to implement the block processing of the target feature vectors.

[0032] It can be understood that the Simhash algorithm belongs to a kind of locality-sensitive hashing algorithm, which can efficiently calculate the similarity of data, is suitable for the deduplication task of large-scale data, and uses the Simhash algorithm for the target feature vectors to generate target hash values to ensure that similar data is mapped to similar hash values.

[0033] Among them, the implementation of the Simhash algorithm is as follows: First, perform vector weighting to assign weights to each dimension of the target feature vector. The weights can be dynamically adjusted according to the importance of the features. Then, perform binary conversion to convert the weighted target feature vector into a binary representation. For example, for each dimension, if the value is greater than a preset threshold (such as 0.5), it is set to 1; otherwise, it is set to 0. Finally, generate a hash value by performing a weighted sum on the binary feature vector to generate an intermediate hash value. For each bit of the intermediate hash value, if it is greater than 0, it is set to 1; otherwise, it is set to 0, and finally, a fixed-length Simhash value (such as 64 bits or 128 bits), that is, the target hash value, is generated. Among them, the NumPy library can be used to implement the weighting and binary conversion of the target feature vector, and the bit operation library of Python can be used to implement the generation of the target hash value.

[0034] Finally, store the generated Simhash value. An efficient data structure (such as a hash table or a database) can be used to store the Simhash value in the form of a binary string or an integer, and the Simhash value and the corresponding data to be detected are stored in a key-value pair manner to establish an inverted index for quickly querying similar data. At the same time, the Locality Sensitive Hashing (LSH) technology can be used to accelerate the similarity search. It can be understood that the traditional index is based on the original key of the data (such as document number, unique identifier of the data record, etc.) to find the corresponding data content, while the inverted index, on the contrary, uses the feature of the data (Simhash value) as the key and the data to be detected corresponding to the Simhash value as the value.

[0035] In this way, through the Simhash algorithm, this embodiment realizes efficient hash calculation for the dimension-reduced target feature vector, has high efficiency, flexibility, and scalability, and can provide high-quality Simhash value input for subsequent data deduplication determination. At the same time, by combining the Simhash algorithm and the dimension reduction technology, the computational complexity is significantly reduced, the efficient processing of large-scale data is realized, and the accuracy and efficiency of data similarity detection are greatly improved. For example, for text data, the dimension of the feature vector after dimension reduction is reduced from 768 dimensions to 64 dimensions, and the computational efficiency is increased by about 30%. For image data, the Hamming distance calculation can be accelerated through the GPU, and the processing speed is increased by about 50%. For audio data, a distributed computing framework (such as Spark) can be used to process large-scale data, and the calculation time is reduced by about 40%.

[0036] Step S13: Determine the expected similar data pairs from the target data based on the target hash value, and determine the Hamming distance between the expected similar data pairs according to the target hash values corresponding to the two data in the expected similar data pairs.

[0037] In this embodiment, determining the expected similar data pairs from the target data based on the target hash values may include: for any two target hash values, comparing the two hash value segments in each hash value pair located at the same hash value segment position among the two target hash values; if there is a first hash value pair with the same K value in each hash value pair, comparing whether there is a second hash value pair with the same V value in each first hash value pair; if there is a second hash value pair with the same V value, taking the two data to be detected corresponding to the two target hash values as the expected similar data pairs, if there is no second hash value pair with the same V value, determining whether the difference between the V values of the two target hash values exceeds the expected difference threshold, and if the difference between the V values of the two target hash values does not exceed the expected difference threshold, taking the two data to be detected corresponding to the two target hash values as the expected similar data pairs.

[0038] It should be noted that in this embodiment, the drawer principle is used to determine the expected similar data pairs, reducing the number of comparisons and improving the efficiency. According to the drawer principle, at least one segment of the Simhash values in the expected similar data pairs is exactly the same. For example Figure 2 As shown, both text 1 and text 2 are composed of 64-bit Simhash values, and both text 1 and text 2 are divided into four segments; among them, there is 1 different position in the first three segments of text 1 and text 2, and there is no different position in the fourth segment. Therefore, text 1 and text 2 are expected similar data pairs. Therefore, for any two target hash values A and B, they are divided into several hash value segments by position, and the hash value segments at the same position are compared. If there is a pair of hash value segments with the same K value, continue to compare the V values of this pair of hash value segments. If the V values are also the same, take the two data to be detected corresponding to A and B as the expected similar data pairs; if the V values are different, compare whether the difference between the two target hash values A and B is within the expected difference threshold. If it is within the expected difference threshold, take the corresponding two data to be detected as the expected similar data pairs as well.

[0039] For example, the Simhash value is 64 bits, and the Simhash value is divided into 4 segments for encoding. Each Simhash value is stored in 4 segments of 16-bit KV encoding. First, compare K-K and then V-V. Suppose there are 2^30 (1 billion) Simhash values and at most 2^16 kinds of K. Then the maximum number of candidates returned by each K-V is 2^(30 - 16) = 16384, and the total result of 4 Ks is 16384 * 4 = 65536, about 660,000 comparisons; for 2^25 (more than 33 million) Simhash values, there are 2^(25 - 16) = 512 candidates, and 512 * 4 = 2048 comparisons are made, greatly reducing the number of data comparisons.

[0040] Furthermore, determine the Hamming distance between the expected similar data pairs according to the target hash values respectively corresponding to the two data in the expected similar data pairs. Specifically, the target hash value is usually represented by a binary string or an integer. By comparing the number of differences in the binary bits or digits at the same positions of the two target hash values, the Hamming distance between them can be obtained. For example Figure 3 as shown, the two target hash values represented in binary are 1001110001 and 1001100011 respectively, and their Hamming distance is 2 because the digits at the 6th and 9th positions are different.

[0041] Step S14: Use the Hamming distance to determine whether the data in the expected similar data pairs are repeated to obtain a judgment result, and based on the judgment result, perform data deduplication on the duplicate data in the data to be detected.

[0042] In this embodiment, using the Hamming distance to determine whether the data in the expected similar data pairs are repeated to obtain a judgment result may include: if the Hamming distance is less than a preset distance threshold, it is determined that the data in the expected similar data pairs are repeated, and a judgment result indicating data repetition is obtained; if the Hamming distance is not less than the preset distance threshold, it is determined that the data in the expected similar data pairs are not repeated, and a judgment result indicating data non-repetition is obtained.

[0043] It can be understood that the Hamming distance reflects the number of different characters at the corresponding positions of two equal-length target hash values. The smaller the Hamming distance, the more similar the two target hash values corresponding to the data, indicating a high similarity in features of the original data; on the contrary, the larger the Hamming distance, the greater the difference between the data. Through this judgment method based on the Hamming distance and the preset threshold, it is possible to objectively determine whether the data in the expected similar data pairs are repeated, which helps to clean up duplicate data and improve data quality and processing efficiency.

[0044] Finally, based on the judgment result, perform data deduplication on the duplicate data in the data to be detected. For example, the Text-dedup tool can be used to implement the deduplication processing of large-scale data. This tool is designed for the Spark cluster environment, ensuring efficient operation in a high-concurrency and distributed computing environment. Specifically, by making Text-dedup compatible with the Spark cluster, large-scale text data processing tasks can be decomposed and executed in parallel on multiple nodes in the cluster. Each node is responsible for processing a part of the data, which can significantly improve the processing speed. For example, when processing a large amount of text data, if only a single machine is used for deduplication and duplicate checking operations, the processing time is long and the task may not be completed due to insufficient memory. With the help of the Spark cluster, multiple nodes work simultaneously, greatly shortening the processing time and improving the overall performance of the system.

[0045] It should be noted that Text-dedup natively supports data deduplication, which can effectively identify and eliminate duplicate data to ensure data uniqueness. It is also possible to customize the Text-dedup source code so that Text-dedup uses the data duplication checking method based on Simhash values in this application to implement the data duplication checking function, meeting more detailed data quality control requirements.

[0046] For example Figure 4 As shown, first read the data set, perform a hash algorithm on the data set to convert the data in the data set into hash values, and construct an undirected graph connected set based on the hash values. Initially screen out duplicate data through the undirected graph connected set to prepare for subsequent deduplication or duplication checking operations. The user side can choose to deduplicate or check the data set. If deduplicating, perform a duplication checking operation on the data set according to the undirected graph connected set and using the data duplication checking method based on Simhash values in this application to obtain a duplication checking result, and filter the data set based on the duplication checking result to generate a deduplicated data set, and save the deduplicated data set; if checking for duplicates, perform a duplication checking operation on the data set according to the undirected graph connected set and using the data duplication checking method based on Simhash values in this application to obtain a duplication checking result, filter the data set based on the duplication checking result to generate a duplicate data set, and then similarity processing can be performed on the duplicate data set based on UDF to further analyze the similarity degree between the duplicate data to obtain duplicate similar data, and save the duplicate similar data. For deduplication, finally, monitoring metrics such as deduplication metrics and time-consuming metrics can be calculated. For duplicate checking, finally, monitoring metrics such as duplicate checking metrics and time-consuming metrics can be calculated, which helps to understand the working conditions of the data processing process, provides a basis for optimizing the algorithm and adjusting resource allocation, and ensures the efficiency and accuracy of the data processing process.

[0047] Among them, UDF (User-Defined Function) can flexibly define the similarity calculation logic according to specific requirements to further accurately analyze the similarity degree between duplicate data, not limited to simple hash similarity judgment, so as to distinguish completely duplicate and partially similar data.

[0048] As can be seen from the above, in this embodiment, semantic feature extraction is performed on target data based on a preset large model to obtain a target high-dimensional semantic feature vector corresponding to the target data, which can accurately identify duplicate data with similar semantics but different expression forms in the target data, significantly improving the deduplication accuracy; by combining a preset locality-sensitive hashing algorithm and dimensionality reduction technology, the computational complexity is significantly reduced, enabling efficient processing of large-scale data and greatly enhancing the accuracy and efficiency of data similarity detection; seamlessly integrated into the Spark cluster, making full use of distributed computing resources to achieve rapid processing of large-scale data; at the same time, combining the general feature extraction ability of the large model and the locality-sensitive hashing algorithm, it can be applied to deduplication tasks of various data types such as text, images, and audio, with strong generality; the source code is open, supporting customized development according to actual needs to meet the deduplication and duplicate checking requirements in different scenarios, and each module is independent of each other, being easy to expand and integrate. Through the method in this embodiment, the deduplication and duplicate checking of large-scale text data can be efficiently and accurately achieved, providing a solid data foundation for subsequent data analysis and mining work.

[0049] See Figure 5 shown below. Taking the specific data duplicate checking and deduplication processing process as an example, the technical solution in this application will be described.

[0050] First, randomly sample 100,000 data from the large dataset for duplicate checking. Among them, the first benchmark similarity can be set to 0.7. The higher the benchmark similarity is set, the stricter the system's duplicate checking of the data. If it is set to 0.7, it means that only data with a similarity of 0.7 or above will be determined as duplicate data. For example, when checking the sampled data, it is necessary to calculate the similarity for each pair of data considered likely to be duplicate, and then compare it with the benchmark similarity. If the similarity between the sampled data is greater than or equal to the benchmark similarity, it is determined as duplicate data; otherwise, it is not determined as duplicate data. This step aims to preliminarily evaluate the duplicate situation of the dataset at a relatively low computational cost. Then, conduct a manual sampling inspection on the duplicate checking results of the sampling inspection, that is, conduct a manual sampling inspection on the duplicate data obtained from the duplicate checking of the above 100,000 data. When the sampled data is 100% duplicate, the similarity can be reduced by 1% - 10%. This is to further verify whether there are any missed duplicate data. Because the current relatively high benchmark similarity may cause some data that actually have a certain similarity and should be determined as duplicate to not be recognized due to the similarity being slightly lower than the benchmark value, that is, there may be a "false negative" situation. By reducing the similarity and expanding the scope of duplicate checking, these originally misjudged duplicate data can be discovered, improving the comprehensiveness and accuracy of duplicate checking. When the sampled data is not 100% duplicate, if the sampled data is not more than 95% duplicate, it indicates that there is a "false positive misjudgment" at this time, that is, the data itself is not truly duplicate data, but due to the unreasonable setting of the system's benchmark similarity, these data are misjudged as duplicate. For example, in text duplicate checking, two pieces of text with similar themes but different contents are determined as duplicate due to a low benchmark similarity, but in fact, they express different viewpoints. At this time, the similarity can be increased by 1% - 10%, and a second system duplicate checking and manual sampling inspection can be performed until the data duplication rate is greater than 95% to reduce such misjudgments and make the duplicate checking results more accurate; if the sampled data is more than 95% duplicate, a second manual sampling inspection can be performed at this time. If the data is still more than 95% duplicate after the second manual sampling inspection, it indicates that the benchmark similarity is set reasonably at this time. At this time, the entire dataset (such as 1 billion data) can be fully checked for duplicates and fully deduplicated based on the system with the adjusted benchmark similarity. Among them, for duplicate data, the duplicate data with a low similarity can be further analyzed, and the duplicate data with a low similarity can be stored in the database together with the non-duplicate data and the non-duplicate data after deduplication.

[0051] As can be seen from the above, in this embodiment, first, 100,000 data are randomly selected from the large dataset for duplicate checking processing, and then a manual sampling inspection is conducted on the sampling duplicate checking results. After determining that the manual sampling inspection results meet the expectations, the large-scale full dataset is checked for duplicates and deduplicated. In this way, the efficient and accurate execution of duplicate checking and deduplication for the large dataset is realized, providing a solid data foundation for subsequent data analysis and mining work, and at the same time minimizing the performance overhead.

[0052] See Figure 6 As shown, the embodiment of the present application also discloses a data deduplication device based on a large model and a local hashing algorithm, including: A vector acquisition module 11, configured to perform preprocessing operations on the acquired data to be detected to obtain target data, and extract semantic features of the target data based on a preset large model to obtain a target high-dimensional semantic feature vector corresponding to the target data; A hash value acquisition module 12, configured to process the reduced-dimensional target high-dimensional semantic feature vector by using a preset locality-sensitive hashing algorithm to obtain target hash values respectively corresponding to the reduced-dimensional target high-dimensional semantic feature vector; A Hamming distance determination module 13, configured to determine an expected similar data pair from the target data based on the target hash value, and determine the Hamming distance between the expected similar data pairs according to the target hash values respectively corresponding to the two data in the expected similar data pair; A data deduplication module 14, configured to use the Hamming distance to determine whether the data in the expected similar data pair is repeated to obtain a judgment result, and perform data deduplication on the duplicate data in the data to be detected based on the judgment result.

[0053] As can be seen from the above, the present application extracts semantic features of target data based on a preset large model to obtain a target high-dimensional semantic feature vector corresponding to the target data, which can accurately identify duplicate data with similar semantics but different expression forms in the target data, significantly improving the deduplication accuracy; by combining a preset locality-sensitive hashing algorithm and a dimensionality reduction technique, the computational complexity is significantly reduced, realizing efficient processing of large-scale data; at the same time, by combining the general feature extraction ability of the large model and the locality-sensitive hashing algorithm, it can be applied to deduplication tasks of various data types such as text, images, and audio, and has strong versatility.

[0054] In some specific embodiments, the vector acquisition module 11 includes: A target text data acquisition unit, configured to perform text cleaning and word segmentation on the text data in the data to be detected to obtain first text data, remove special characters and HTML tags in the first text data to obtain second text data, and unify the encoding format of the second text data by using a preset encoding format to obtain target text data corresponding to the text data; wherein, the special characters include punctuation marks and stop words; A target image data acquisition unit, which is used for the image data in the data to be detected, adjusts the image data based on a preset resolution to obtain first image data, adjusts the color mode of the first image data to a preset color mode to obtain second image data, and normalizes the pixel values of the second image data to obtain the target image data corresponding to the image data; A target audio data acquisition unit, which is used for the audio data in the data to be detected, adjusts the audio sampling rate in the audio data to a fixed value based on a preset sampling rate, and performs noise removal processing on the adjusted audio data to obtain the target audio data corresponding to the audio data.

[0055] In some specific embodiments, the hash value acquisition module 12 includes: A feature vector acquisition sub-module, which is used for adjusting the dimension-reduced target high-dimensional semantic feature vector based on a preset target dimension to obtain a target feature vector; A first storage unit, which is used for using the Simhash algorithm to determine the target hash values corresponding to the target feature vectors respectively, storing the target hash values and the corresponding data to be detected in the form of key-value pairs to establish an index, and querying and de-duplicating data based on the index.

[0056] In some specific embodiments, the feature vector acquisition sub-module further includes: A block processing unit, which is used for performing block processing on the target feature vector, and parallelly using the Simhash algorithm on the block-processed target feature vector to determine the target hash values corresponding to the block-processed target feature vector.

[0057] In some specific embodiments, the data deduplication device based on the large model and the local hash algorithm further includes: A second storage unit, which is used for any target hash value, divides the target hash value into a preset number of hash value segments, and stores each of the hash value segments using KV encoding; wherein, each of the KV-encoded hash value segments consists of a K value and a V value.

[0058] In some specific embodiments, the Hamming distance determination module 13 includes: A comparison unit, which is used for any two target hash values, compares the two hash value segments in each hash value pair located at the same hash value segment position in the two target hash values; A first determination unit, which is used for if there is a first hash value pair with the same K value in each hash value pair, comparing whether there is a second hash value pair with the same V value in each of the first hash value pairs; A second judgment unit, configured to, if there are second hash value pairs with the same V value, use the two data to be detected corresponding to the two target hash values as expected similar data pairs; if there are no second hash value pairs with the same V value, determine whether the difference between the V values of the two target hash values exceeds an expected difference threshold. If the difference between the V values of the two target hash values does not exceed the expected difference threshold, use the two data to be detected corresponding to the two target hash values as expected similar data pairs.

[0059] In some specific embodiments, the data deduplication module 14 includes: A third judgment unit, configured to, if the Hamming distance is less than a preset distance threshold, determine that the data in the expected similar data pair is duplicated, and obtain a judgment result indicating data duplication; A fourth judgment unit, configured to, if the Hamming distance is not less than the preset distance threshold, determine that the data in the expected similar data pair is not duplicated, and obtain a judgment result indicating data non-duplication.

[0060] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 7 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure should not be regarded as any limitation on the scope of use of the present application.

[0061] Figure 7 It is a schematic structural diagram of an electronic device 20 provided by an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the data deduplication method based on a large model and a local hash algorithm disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0062] In this embodiment, the power supply 23 is used to provide operating voltages for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.

[0063] In addition, the memory 22, as the carrier for storing resources, can be a read-only memory, a random access memory, a magnetic disk, an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0064] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the data deduplication method based on the large model and the local hashing algorithm executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

[0065] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the data deduplication method based on the large model and the local hashing algorithm disclosed above. For the specific steps of this method, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be repeated here.

[0066] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and reference can be made to the description of the method part for related parts.

[0067] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0068] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0069] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0070] The technical solutions provided in this application have been introduced in detail above. Specific examples are used in this text to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A data deduplication method based on a large model and a local hashing algorithm, characterized in that, Including: Performing a preprocessing operation on the obtained data to be detected to obtain target data, and extracting semantic features from the target data based on a preset large model to obtain a target high-dimensional semantic feature vector corresponding to the target data; Processing the reduced-dimensional target high-dimensional semantic feature vector by using a preset locality-sensitive hashing algorithm to obtain target hash values corresponding to the reduced-dimensional target high-dimensional semantic feature vector respectively; Determining expected similar data pairs from the target data based on the target hash values, and determining the Hamming distance between the expected similar data pairs according to the target hash values corresponding to the two data in the expected similar data pairs; Using the Hamming distance to determine whether the data in the expected similar data pairs are repeated to obtain a judgment result, and performing data deduplication on the duplicate data in the data to be detected based on the judgment result.

2. The data deduplication method based on a large model and a local hashing algorithm according to claim 1, wherein The performing a preprocessing operation on the obtained data to be detected to obtain target data includes: For the text data in the data to be detected, cleaning and segmenting the text data to obtain first text data, removing special characters and HTML tags in the first text data to obtain second text data, and unifying the encoding format of the second text data by using a preset encoding format to obtain target text data corresponding to the text data; wherein, the special characters include punctuation marks and stop words; For the image data in the data to be detected, adjusting the image data based on a preset resolution to obtain first image data, adjusting the color mode of the first image data to a preset color mode to obtain second image data, and normalizing the pixel values of the second image data to obtain target image data corresponding to the image data; For the audio data in the data to be detected, adjusting the audio sampling rate in the audio data to a fixed value based on a preset sampling rate, and performing noise removal processing on the adjusted audio data to obtain target audio data corresponding to the audio data.

3. The data deduplication method based on the large model and the local hashing algorithm according to claim 1, wherein The processing the reduced-dimensional target high-dimensional semantic feature vector by using a preset locality-sensitive hashing algorithm to obtain target hash values corresponding to the reduced-dimensional target high-dimensional semantic feature vector respectively includes: Adjusting the reduced-dimensional target high-dimensional semantic feature vector based on a preset target dimension to obtain a target feature vector; Using the Simhash algorithm to determine the target hash values corresponding to the target feature vector respectively, storing the target hash values and the corresponding data to be detected in the form of key-value pairs to establish an index, and querying and deduplicating data based on the index.

4. The data deduplication method based on the large model and the local hashing algorithm according to claim 3, wherein After the adjusting the reduced-dimensional target high-dimensional semantic feature vector based on a preset target dimension to obtain a target feature vector, it further includes: Performing a block processing on the target feature vector, and parallelly using the Simhash algorithm on the block-processed target feature vector to determine the target hash values corresponding to the block-processed target feature vector.

5. The data deduplication method based on a large model and a local hashing algorithm according to claim 1, wherein It also includes: For any target hash value, divide the target hash value into a preset number of hash value segments, and store each of the hash value segments using KV encoding; wherein, each of the hash value segments after KV encoding consists of a K value and a V value.

6. The data deduplication method based on the large model and the local hashing algorithm according to claim 5, wherein The determining of the expected similar data pairs from the target data based on the target hash value includes: For any two target hash values, compare the two hash value segments in each pair of hash values at the same hash value segment position in the two target hash values; If there is a first pair of hash values with the same K value among the pairs of hash values, compare whether there is a second pair of hash values with the same V value among the first pairs of hash values; If there is a second pair of hash values with the same V value, take the two data to be detected corresponding to the two target hash values as the expected similar data pairs. If there is no second pair of hash values with the same V value, determine whether the difference between the V values in the two target hash values exceeds the expected difference threshold. If the difference between the V values in the two target hash values does not exceed the expected difference threshold, take the two data to be detected corresponding to the two target hash values as the expected similar data pairs.

7. The data deduplication method based on a large model and a local hashing algorithm according to any one of claims 1 to 6, characterized in that The using of the Hamming distance to determine whether the data in the expected similar data pairs are repeated to obtain a judgment result includes: If the Hamming distance is less than the preset distance threshold, determine that the data in the expected similar data pairs are repeated, and obtain a judgment result indicating data repetition; If the Hamming distance is not less than the preset distance threshold, determine that the data in the expected similar data pairs are not repeated, and obtain a judgment result indicating data non-repetition.

8. A data deduplication device based on a large model and a local hashing algorithm, characterized in that, Includes: A vector acquisition module, configured to perform preprocessing operations on the acquired data to be detected to obtain target data, and extract semantic features of the target data based on a preset large model to obtain a target high-dimensional semantic feature vector corresponding to the target data; A hash value acquisition module, configured to process the dimension-reduced target high-dimensional semantic feature vector using a preset locality-sensitive hashing algorithm to obtain target hash values corresponding to the dimension-reduced target high-dimensional semantic feature vector respectively; A Hamming distance determination module, configured to determine expected similar data pairs from the target data based on the target hash value, and determine the Hamming distance between the expected similar data pairs according to the target hash values corresponding to the two data in the expected similar data pairs; A data deduplication module, configured to use the Hamming distance to determine whether the data in the expected similar data pairs are repeated to obtain a judgment result, and perform data deduplication on the duplicate data in the data to be detected based on the judgment result.

9. An electronic device, characterized in that, Includes: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the data deduplication method based on a large model and a local hashing algorithm according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, For storing a computer program, the computer program, when executed by a processor, implements the data deduplication method based on a large model and a local hashing algorithm according to any one of claims 1 to 7.

Images