Feature extraction data deduplication method and device, electronic equipment and storage medium

By using the word frequency-inverse text frequency TF-IDF method in the data deduplication method to construct feature vectors and classify and process them, the problem of low efficiency of large-scale data deduplication in the prior art is solved, and efficient and accurate data deduplication is achieved. And reduces the system's dependence on the central server.

CN119005139APending Publication Date: 2024-11-22INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411019181.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing data deduplication methods are highly complex and inefficient in processing large-scale data, especially text that cannot accurately judge semantically similar but not exactly the same text, resulting in low data deduplication efficiency.

Method used

The feature vector is constructed based on word frequency-inverse text frequency TF-IDF method, and the text data to be processed are classified through the feature vector, and the same category of data is allocated to the same distributed server for deduplication, avoiding the real-time calculation of the central server .

Benefits of technology

It improves the efficiency and accuracy of data deduplication, makes full use of the parallel processing capabilities of distributed systems, reduces the risk of hash conflicts, and improves the availability and resource utilization of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005139A_ABST
    Figure CN119005139A_ABST
Patent Text Reader

Abstract

The invention provides a feature extraction data deduplication method and device, electronic equipment and a storage medium, and the method comprises the steps: constructing a to-be-processed text data set based on received to-be-processed text data; based on a TF-IDF method, determining a TF-IDF weight of each word in the to-be-processed text data set, and based on the TF-IDF weight, determining a feature vector of each piece of to-be-processed text data; classifying the to-be-processed text data in the to-be-processed text data set based on the feature vectors to obtain a plurality of categories of to-be-processed text data; and based on the plurality of categories, distributing the to-be-processed text data of the to-be-processed text data set to a plurality of distributed servers for de-duplication processing. And distributing the to-be-processed text data of the same category to the same distributed server. Each distributed server can independently perform deduplication processing without relying on real-time calculation of a central server, so that the data deduplication efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for data deduplication by feature extraction. Background Art

[0002] Existing data deduplication methods are mainly based on the principle of comparison and matching, detecting duplicates by comparing data one by one.

[0003] Existing data deduplication methods based on comparison and matching face the challenges of high computational complexity and low efficiency when processing large-scale data. In addition, for text data, simple comparison methods may not accurately judge texts that are semantically similar but not identical, resulting in low data deduplication efficiency. Summary of the invention

[0004] The present invention provides a method, device, electronic device and storage medium for data deduplication based on feature extraction, so as to improve the efficiency of data deduplication.

[0005] The present invention provides a data deduplication method for feature extraction, comprising the following steps: Building a set of text data to be processed based on the received text data to be processed sent by the multiple distributed servers and the text data to be processed sent by the multiple data sources; Based on the term frequency-inverse text frequency TF-IDF method, determine the TF-IDF weight of each word in the text data set to be processed, and based on the TF-IDF weight, determine the feature vector of each text data to be processed; Based on the feature vector of each of the text data to be processed, classifying the text data to be processed in the text data set to be processed to obtain multiple categories of text data to be processed; Based on the multiple categories, the to-be-processed text data of the to-be-processed text data set are distributed to the multiple distributed servers, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

[0006] According to a data deduplication method for feature extraction provided by the present invention, based on the feature vector of each text data to be processed, the text data to be processed in the text data set to be processed is classified to obtain multiple categories of text data to be processed, including: Determine the feature vector of each piece of text data to be processed in the set of text data to be processed based on a formula for determining the feature vector of the text data to be processed; The formula for determining the feature vector of the text data to be processed is determined based on the feature vector of each word in the text data to be processed, the number of all words contained in the text data to be processed, and the word frequency-inverse text frequency TF-IDF weight calculated in the text data to be processed; Determining similar text data to be processed in the set of text data to be processed based on the values ​​of the feature vectors of the text data to be processed; Similar text data to be processed in the text data set to be processed are grouped into one category, so as to classify the text data to be processed in the text data set to be processed and obtain multiple categories of text data to be processed.

[0007] According to a data deduplication method for feature extraction provided by the present invention, the formula for determining the feature vector of the text data to be processed is: ; in, Text data to be processed The feature vector of n is the text data to be processed The number of all words contained in Text data to be processed Words in In the text data to be processed The TF-IDF weight in .

[0008] According to a data deduplication method for feature extraction provided by the present invention, the TF-IDF weight of each word in the text data set to be processed is determined based on the word frequency-inverse text frequency TF-IDF method, including: Based on the term frequency-inverse document frequency TF-IDF method, determining the term frequency of each term in the text data set to be processed and the inverse document frequency of each term; Based on the word frequency and the inverse document frequency, the TF-IDF weight of each word in the text data set to be processed is determined.

[0009] According to a data deduplication method for feature extraction provided by the present invention, the method of allocating the to-be-processed text data of the to-be-processed text data set to the multiple distributed servers based on the multiple categories includes: Determining the amount of the to-be-processed text data in each category among the multiple categories of to-be-processed text data; The resource utilization rates of the plurality of distributed servers are obtained, and based on the resource utilization rates of the distributed servers and the amount of the text data to be processed in each category, the text data to be processed of the text data set to be processed are distributed to the plurality of distributed servers.

[0010] According to a data deduplication method for feature extraction provided by the present invention, after constructing the text data set to be processed, the method further includes: Performing data preprocessing on the text data to be processed in the text data set to be processed; The data preprocessing includes text segmentation, removal of stop words and removal of punctuation marks.

[0011] According to a data deduplication method for feature extraction provided by the present invention, the feature vector of the text data to be processed is constructed based on taking the TF-IDF weight of each word in the text data to be processed as a vector dimension.

[0012] The present invention also provides a data deduplication device for feature extraction, comprising the following modules: A collection construction module, used to construct a collection of text data to be processed based on the received text data to be processed sent by multiple distributed servers and the text data to be processed sent by multiple data sources; A weight determination module, used to determine the TF-IDF weight of each word in the text data set to be processed based on the word frequency-inverse text frequency TF-IDF method, and determine the feature vector of each text data to be processed based on the TF-IDF weight; A classification module, used for classifying the to-be-processed text data in the to-be-processed text data set based on the feature vector of each to-be-processed text data, to obtain a plurality of categories of to-be-processed text data; A distribution processing module is used to distribute the to-be-processed text data of the to-be-processed text data set to the multiple distributed servers based on the multiple categories, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a data deduplication method for feature extraction as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data deduplication method for feature extraction as described in any one of the above.

[0015] The feature extraction data deduplication method, device, electronic device and storage medium provided by the present invention construct a feature vector of each word by determining the TF-IDF weight of each word in the text data set to be processed. Based on the feature vector of each word, the text data to be processed in the text data set to be processed is classified, a plurality of categories are determined, and based on the determined plurality of categories, the text data to be processed in the text data set to be processed is distributed to a plurality of distributed servers, so that the text data to be processed of the same category is distributed to the same distributed server. Each distributed server can perform deduplication processing independently without relying on the real-time calculation of the central server, thereby improving the efficiency of data deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the data deduplication method for feature extraction provided by the present invention.

[0018] Figure 2 This is a deduplication processing flow chart provided by the present invention.

[0019] Figure 3 The present invention provides a schematic diagram of the structure of a device for applying the method provided by the present invention.

[0020] Figure 4 It is a schematic diagram of the structure of the data deduplication device for feature extraction provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Data deduplication is an important problem often encountered when processing large-scale data. With the advent of the information age, a large amount of data is created and stored, which may contain a large amount of duplicate data. Duplicate data not only wastes storage space, but also increases the complexity of data processing and analysis. Therefore, effective data deduplication methods are of great significance for data management and analysis.

[0024] One of the related methods is based on the principle of comparison and matching, which detects duplicates by comparing data one by one. However, this method faces the challenges of high computational complexity and low efficiency when processing large-scale data. In addition, for text data, simple comparison methods may not accurately judge texts that are semantically similar but not identical, resulting in low data deduplication efficiency.

[0025] The second related method uses a central server to deduplicate data. The central server uses a first hash algorithm to obtain a first hash value corresponding to the current data to be processed, and distributes the first hash value to the corresponding distributed server. The distributed server uses a second hash algorithm to hash the first hash value to obtain a second hash value, and the distributed server deduplicates the current data to be processed according to the second hash value. The deduplication method in the second related method directly uses the second hash algorithm to obtain the second hash value for the data to be processed, and the distributed server then deduplicates the data to be processed.

[0026] This data deduplication method is that the central server uses the first hash algorithm to hash the data and distributes the hash value to the corresponding distributed server, and the distributed server then uses the second hash algorithm to hash and dedupe the data. Some disadvantages are: Hash collision problem: When using hash algorithms for data deduplication, there is a possibility of hash collision, that is, different data may generate the same hash value, which may lead to misjudgment and data loss.

[0027] Distributed servers rely on central servers: Distributed servers need to rely on the allocation of central servers to obtain data and perform deduplication processing. If the central server fails or does not work properly, the distributed servers may be unable to perform deduplication processing, thus affecting the accuracy and availability of the entire data deduplication system.

[0028] In view of the defects in the related methods, the present invention provides a data deduplication method for feature extraction. Figure 1 is a flow chart of the data deduplication method for feature extraction provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 110, constructing a set of text data to be processed based on the received text data to be processed sent by the multiple distributed servers and the text data to be processed sent by the multiple data sources; Step 120, determining the TF-IDF weight of each word in the text data set to be processed based on the term frequency-inverse text frequency TF-IDF method, and determining the feature vector of each text data to be processed based on the TF-IDF weight; Step 130, classifying the text data to be processed in the text data set to be processed based on the feature vector of each text data to be processed, to obtain multiple categories of text data to be processed; Step 140, based on the multiple categories, distribute the to-be-processed text data of the to-be-processed text data set to the multiple distributed servers, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

[0029] The execution subject of the feature extraction data deduplication method provided by the present invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), or a personal computer (PC), etc., which is not specifically limited by the present invention.

[0030] The following takes the data deduplication method for feature extraction provided by the present invention executed by a computer as an example to explain the technical solution of the present invention in detail.

[0031] In step 110, a set of text data to be processed is constructed based on the received text data to be processed sent by multiple distributed servers and the received text data to be processed sent by multiple data sources; Based on the central server, the text data to be processed sent by multiple distributed servers connected to the central server and the text data to be processed sent by multiple data sources can be received. The central server constructs a set of text data to be processed based on the received text data to be processed.

[0032] The central server collects data from various data sources, including but not limited to distributed servers, databases, file systems, or other data streams. The role of the central server is to coordinate and manage the entire deduplication process, including data reception, preprocessing, feature extraction, and assigning tasks to distributed servers for further processing.

[0033] In step 120, based on the term frequency-inverse text frequency TF-IDF method, the TF-IDF weight of each word in the text data set to be processed is determined, and based on the TF-IDF weight, the feature vector of each text data to be processed is determined.

[0034] TF-IDF (Term Frequency-Inverse Document Frequency) is a commonly used weighting technique for information retrieval and text mining. It is used to evaluate the importance of a term to a document set or a document in a corpus. The importance of a term increases in direct proportion to the number of times it appears in a document, but decreases in inverse proportion to the frequency of its appearance in the corpus.

[0035] The process of determining the feature vector based on TF-IDF may include: For each text data to be processed Perform feature extraction. The algorithm includes the following steps: Text data set to be processed Contains multiple text data to be processed .

[0036] Calculate word frequency (TF): for each text data to be processed , calculate the frequency of each word Indicates word j in the text data to be processed The frequency of occurrence in . The formula is as follows: ; Calculate the inverse document frequency (IDF): For each word j, calculate the inverse document frequency , indicating that word j is in the text data set to be processed The inverse document frequency in . The formula is as follows: ; calculate Weight: Multiply the term frequency and the inverse document frequency to get the TF-IDF weight .

[0037] The formula is as follows: ; Construct feature vector: Take the TF-IDF weight of each word as the dimension of feature vector and construct ; Eigenvector , indicating the text data to be processed Each dimension of the feature vector corresponds to a word. For example, the feature vector The jth dimension of represents the weight of word j .

[0038] use The text feature extraction algorithm extracts features from the data to be processed and converts the data into a feature vector with high discrimination. It comprehensively considers the frequency of words in the text and the inverse document frequency in the entire document collection, and can accurately measure the importance of words.

[0039] In step 130, based on the feature vector of each of the text data to be processed, the text data to be processed in the text data set to be processed is classified to obtain multiple categories of text data to be processed.

[0040] Specifically, documents with similar feature vectors can be assigned to one category according to the content of the feature vector. By classifying the to-be-processed text data in the to-be-processed text data set, multiple categories of to-be-processed text data can be obtained.

[0041] In step 140, based on the multiple categories, the to-be-processed text data of the to-be-processed text data set are distributed to the multiple distributed servers, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

[0042] According to the obtained multiple categories, each to-be-processed text data in the to-be-processed text data set is assigned to a corresponding category, so as to obtain multiple categories of to-be-processed text data.

[0043] According to the categories, the to-be-processed text data of the same category are distributed to the same distributed server, and the distribution process of the to-be-processed text data of all categories in the to-be-processed text data set is completed.

[0044] In a distributed system, there are usually multiple servers available to process data. In order to optimize the processing process, it is necessary to select an appropriate server allocation strategy based on the classification results. Once the data is assigned to the corresponding servers, each server can start to deduplicate the text data of the same category it receives.

[0045] Optionally, assume that the text dataset to be processed contains three documents: document A, document B, and document C. The specific process of implementing deduplication can be as follows: Figure 2 The deduplication process provided by the present invention is shown in the flowchart, which specifically includes: The goal is to deduplicate these three documents, that is, to find the duplicate content among them.

[0046] First, read the data to be processed and extract its features. Convert documents into feature vectors.

[0047] Assume that the document has been preprocessed to remove stop words and punctuation.

[0048] Based on the feature extraction module, calculate the TF-IDF feature vector: First, calculate the frequency of each word in each document.

[0049] Suppose you want to calculate the frequency of the word "apple" in document A, document B, and document C.

[0050] Then, calculate the inverse document frequency of each word in the entire document collection

[0051] ;

[0052] Suppose you want to calculate the word "apple" in document A ; Similarly, we can calculate the TF-IDF weight of the word "apple" in document B and document C. and .

[0053] By calculating the TF-IDF weights of all words in a document, we can get the TF-IDF feature vector of each document.

[0054]

[0055] ; Where n represents the number of words in document A. Represents the TF-IDF weight of the word "word_i" in document A.

[0056] Based on the allocation module, documents with similar feature vectors can be allocated to the same distributed server according to the content of the feature vectors.

[0057] Each distributed server can perform deduplication processing independently without relying on real-time calculations of the central server.

[0058] After ensuring that all distributed servers have completed processing, output the deduplicated data.

[0059] Through the above implementation plan, the document collection can be effectively deduplicated and the accuracy and adaptability of data deduplication can be improved.

[0060] By distributing the text data to be processed to multiple distributed servers for deduplication according to the classification results, the parallel processing capabilities of the distributed system can be fully utilized to improve data processing efficiency and resource utilization. This process requires a carefully designed data distribution strategy, an efficient deduplication algorithm, and real-time performance monitoring and optimization.

[0061] Optionally, after the allocation is completed, each distributed server may also implement a data deduplication process for the text data to be processed based on a predetermined feature vector of each word in the text data to be processed.

[0062] The data deduplication method based on TF-IDF features has many advantages. First, it can transform the text data to be processed into feature vectors with high discrimination, thereby improving the accuracy of deduplication. Second, the TF-IDF method is flexible to adapt to different types of text data and can meet the needs of deduplication of large-scale data. In addition, the TF-IDF-based method also has good scalability and can adapt to data changes and the expansion of distributed servers.

[0063] The data deduplication method for feature extraction provided by the present invention constructs a feature vector of each word by determining the TF-IDF weight of each word in the text data set to be processed. Based on the feature vector of each word, the text data to be processed in the text data set to be processed is classified, a plurality of categories are determined, and based on the determined plurality of categories, the text data to be processed in the text data set to be processed is distributed to a plurality of distributed servers, so that the text data to be processed of the same category is distributed to the same distributed server. Each distributed server can independently perform deduplication processing without relying on the real-time calculation of the central server, thereby improving the efficiency of data deduplication.

[0064] In one embodiment, based on the feature vector of each text data to be processed, the text data to be processed in the text data set to be processed is classified to obtain multiple categories of text data to be processed, including: determining the feature vector of each text data to be processed in the text data set to be processed based on a determination formula of the feature vector of the text data to be processed; the determination formula of the feature vector of the text data to be processed is determined based on the feature vector of each word in the text data to be processed, the number of all words contained in the text data to be processed, and the word frequency-inverse text frequency TF-IDF weight calculated for the words in the text data to be processed in the text data to be processed; based on the value of the feature vector of each text data to be processed, similar text data to be processed in the text data set to be processed is determined; similar text data to be processed in the text data set to be processed is classified into one category to classify the text data to be processed in the text data set to be processed to obtain multiple categories of text data to be processed.

[0065] Among them, the formula for determining the feature vector of the text data to be processed is: ; in, Text data to be processed The feature vector of n is the text data to be processed The number of all words contained in Text data to be processed Words in In the text data to be processed The TF-IDF weight in .

[0066] Specifically, based on the formula for determining the feature vector of the text data to be processed, the feature vector of each text data to be processed in the text data set to be processed is determined.

[0067] Based on the values ​​of the feature vectors of each to-be-processed text data, similar to-be-processed text data in the to-be-processed text data set are determined, and those with similar values ​​can be regarded as similar to-be-processed text data.

[0068] After determining the mutually similar text data to be processed, the similar text data to be processed in the text data set to be processed are classified into one category, so as to classify the text data to be processed in the text data set to be processed.

[0069] In one embodiment, based on the term frequency-inverse text frequency TF-IDF method, the TF-IDF weight of each word in the text data set to be processed is determined, including: based on the term frequency-inverse text frequency TF-IDF method, the term frequency of each word in the text data set to be processed and the inverse document frequency of each word are determined; based on the term frequency and the inverse document frequency, the TF-IDF weight of each word in the text data set to be processed is determined.

[0070] The specific implementation process may include: For each text data to be processed Perform feature extraction. The algorithm includes the following steps: Text data set to be processed Contains multiple text data to be processed .

[0071] Calculate word frequency (TF): for each text data to be processed , calculate the frequency of each word Indicates word j in the text data to be processed The frequency of occurrence in . The formula is as follows: ; Calculate the inverse document frequency (IDF): For each word j, calculate the inverse document frequency , indicating that word j is in the text data set to be processed The inverse document frequency in . The formula is as follows: ; calculate Weight: Multiply the term frequency and the inverse document frequency to get the TF-IDF weight .

[0072] In one embodiment, based on the multiple categories, the to-be-processed text data of the to-be-processed text data set are distributed to the multiple distributed servers, including: determining the number of to-be-processed text data of each category among the multiple categories of to-be-processed text data; obtaining resource utilization of the multiple distributed servers, and based on the resource utilization of each distributed server and the number of to-be-processed text data of each category, distributing the to-be-processed text data of the to-be-processed text data set to the multiple distributed servers.

[0073] The resource utilization of distributed servers is one of the key factors that determine task allocation. Resource utilization includes but is not limited to CPU usage, memory usage, disk I / O performance, and network bandwidth. Real-time monitoring: Obtain the resource usage of each server in real time through the monitoring system. Historical data analysis: Analyze historical data to predict possible peaks and troughs in resource demand in the future. Performance evaluation: Evaluate the processing capabilities of each server, including its speed and efficiency in processing text data. Specifically, historical data can be analyzed based on time series to achieve the prediction process of resource utilization. Time series prediction is a commonly used prediction method that takes into account factors such as trends, seasonality, and periodicity in time series. In server resource utilization prediction, time series models can capture the pattern of resource usage changes over time.

[0074] After obtaining the amount of text data in each category and the resource utilization of each server, an intelligent task allocation strategy can be formulated to ensure that tasks can be efficiently and evenly allocated to each server.

[0075] Specifically, when the amount of text data to be processed in a certain category is large, the text data to be processed in this category can be allocated to a server with a lower resource utilization rate. When the amount of text data to be processed in a certain category is small, the text data to be processed in this category can be allocated to a server with a higher resource utilization rate.

[0076] In one embodiment, after constructing the to-be-processed text data set, the method further includes: performing data preprocessing on the to-be-processed text data in the to-be-processed text data set; the data preprocessing includes text segmentation, removal of stop words, and removal of punctuation marks.

[0077] Specific data preprocessing can be: Input: text data set to be processed ; Output: preprocessed text data set to be processed; ; Data preprocessing includes the following steps: Text segmentation: Split the text data to be processed into a set of words.

[0078] For each text data to be processed Perform word segmentation operation.

[0079] Remove stop words: Remove common meaningless words such as prepositions, conjunctions, etc. Use a stop word list , through the function Remove stop words from each text $d \in D$.

[0080] Remove punctuation: Remove punctuation from text. Use the function Remove each text Punctuation marks in .

[0081] In one embodiment, the feature vector of the text data to be processed is constructed based on taking the TF-IDF weight of each word in the text data to be processed as a vector dimension.

[0082] TF-IDF weight is the product of TF and IDF, which is used to indicate the importance of a word in a document.

[0083] For the constructed feature vector, each dimension of the feature vector corresponds to a word. For example, the feature vector The jth dimension of represents the weight of word j .

[0084] The specific national construction process is as follows: First, a vocabulary needs to be determined. This vocabulary can be all unique words extracted from all documents, or a set of words filtered according to certain rules (such as word frequency threshold, TF-IDF threshold, etc.).

[0085] Calculate TF-IDF: For each word in the vocabulary, calculate its TF-IDF weight in each document.

[0086] Constructing vectors: For each document, the TF-IDF weights of the words in its vocabulary are used as the dimensions of the feature vector to construct a high-dimensional numerical vector. If a word does not appear in a document, the corresponding dimension value in the feature vector of the document is 0.

[0087] The following is a schematic diagram of a device structure for applying the method provided by the present invention to illustrate the technical solution of the present invention. Figure 3 As shown, the device includes a central server and a distributed server. The central server includes a feature extraction module and an allocation module.

[0088] The feature extraction module uses The text feature extraction algorithm extracts features from the data to be processed, determines the TF-IDF weight of each word in the text data set to be processed, and determines the feature vector of each text data to be processed; based on the feature vector of each text data to be processed, the text data to be processed in the text data set to be processed is classified to obtain multiple categories of text data to be processed.

[0089] The distribution module of the central server distributes the to-be-processed text data of the to-be-processed text data set to multiple distributed servers based on multiple categories, so that the to-be-processed text data of the same category are distributed to the same distributed server.

[0090] The allocation module of the central server can allocate data to distributed servers with similar feature vectors for deduplication processing based on the feature vectors, thus realizing the advantages and flexibility of distributed processing.

[0091] The following is a description of the data deduplication device for feature extraction provided by the present invention. The data deduplication device for feature extraction described below and the data deduplication method for feature extraction described above can be referred to in correspondence with each other. Figure 4 As shown, the device comprises: A set construction module 410 is used to construct a set of text data to be processed based on the received text data to be processed sent by multiple distributed servers and the text data to be processed sent by multiple data sources; A weight determination module 420 is used to determine the TF-IDF weight of each word in the text data set to be processed based on the word frequency-inverse text frequency TF-IDF method, and determine the feature vector of each text data to be processed based on the TF-IDF weight; A classification module 430, for classifying the text data to be processed in the text data set to be processed based on the feature vector of each text data to be processed, to obtain multiple categories of text data to be processed; The distribution processing module 440 is used to distribute the to-be-processed text data of the to-be-processed text data set to the multiple distributed servers based on the multiple categories, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

[0092] The data deduplication device for feature extraction provided by the present invention constructs a feature vector of each word by determining the TF-IDF weight of each word in the text data set to be processed. Based on the feature vector of each word, the text data to be processed in the text data set to be processed is classified to determine multiple categories, and based on the determined multiple categories, the text data to be processed in the text data set to be processed is distributed to multiple distributed servers, so that the text data to be processed in the same category is distributed to the same distributed server. Each distributed server can perform deduplication processing independently without relying on the real-time calculation of the central server, thereby improving the efficiency of data deduplication.

[0093] In one embodiment, the classification module 430 is specifically configured to: The method of classifying the text data to be processed in the text data set to be processed based on the feature vector of each text data to be processed to obtain multiple categories of text data to be processed includes: Determine the feature vector of each piece of text data to be processed in the set of text data to be processed based on a formula for determining the feature vector of the text data to be processed; The formula for determining the feature vector of the text data to be processed is determined based on the feature vector of each word in the text data to be processed, the number of all words contained in the text data to be processed, and the word frequency-inverse text frequency TF-IDF weight calculated in the text data to be processed; Determining similar text data to be processed in the set of text data to be processed based on the values ​​of the feature vectors of the text data to be processed; Similar text data to be processed in the text data set to be processed are grouped into one category, so as to classify the text data to be processed in the text data set to be processed and obtain multiple categories of text data to be processed.

[0094] In one embodiment, the classification module 430 is further configured to: The formula for determining the feature vector of the text data to be processed is: ; in, Text data to be processed The feature vector of n is the text data to be processed The number of all words contained in Text data to be processed Words in In the text data to be processed The TF-IDF weight in .

[0095] In one embodiment, the weight determination module 420 is specifically used to: The method based on term frequency-inverse text frequency TF-IDF is used to determine the TF-IDF weight of each word in the text data set to be processed, including: Based on the term frequency-inverse document frequency TF-IDF method, determining the term frequency of each term in the text data set to be processed and the inverse document frequency of each term; Based on the word frequency and the inverse document frequency, the TF-IDF weight of each word in the text data set to be processed is determined.

[0096] In one embodiment, the distribution processing module 440 is specifically used to: The method of allocating the to-be-processed text data of the to-be-processed text data set to the multiple distributed servers based on the multiple categories includes: Determining the amount of the to-be-processed text data in each category among the multiple categories of to-be-processed text data; The resource utilization rates of the plurality of distributed servers are obtained, and based on the resource utilization rates of the distributed servers and the amount of the text data to be processed in each category, the text data to be processed of the text data set to be processed are distributed to the plurality of distributed servers.

[0097] In one embodiment, the set construction module 410 is specifically used to: After constructing the text data set to be processed, the process further includes: Performing data preprocessing on the text data to be processed in the text data set to be processed; The data preprocessing includes text segmentation, removal of stop words and removal of punctuation marks.

[0098] In one embodiment, the weight determination module 420 is specifically used to: The feature vector of the text data to be processed is constructed based on taking the TF-IDF weight of each word in the text data to be processed as a vector dimension.

[0099] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communications interface 520 and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the data deduplication method for feature extraction, the method comprising: constructing a set of text data to be processed based on the received text data to be processed sent by multiple distributed servers and the text data to be processed sent by multiple data sources; Based on the term frequency-inverse text frequency TF-IDF method, determine the TF-IDF weight of each word in the text data set to be processed, and based on the TF-IDF weight, determine the feature vector of each text data to be processed; Based on the feature vector of each of the text data to be processed, classifying the text data to be processed in the text data set to be processed to obtain multiple categories of text data to be processed; Based on the multiple categories, the to-be-processed text data of the to-be-processed text data set are distributed to the multiple distributed servers, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

[0100] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0101] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the data deduplication method for feature extraction provided by the above methods, the method comprising: based on the received text data to be processed sent by multiple distributed servers and the text data to be processed sent by multiple data sources, construct a set of text data to be processed; Based on the term frequency-inverse text frequency TF-IDF method, determine the TF-IDF weight of each word in the text data set to be processed, and based on the TF-IDF weight, determine the feature vector of each text data to be processed; Based on the feature vector of each of the text data to be processed, classifying the text data to be processed in the text data set to be processed to obtain multiple categories of text data to be processed; Based on the multiple categories, the to-be-processed text data of the to-be-processed text data set are distributed to the multiple distributed servers, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

[0102] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the data deduplication method for feature extraction provided by the above methods, the method comprising: constructing a set of text data to be processed based on the text data to be processed sent by multiple distributed servers and the text data to be processed sent by multiple data sources; Based on the term frequency-inverse text frequency TF-IDF method, determine the TF-IDF weight of each word in the text data set to be processed, and based on the TF-IDF weight, determine the feature vector of each text data to be processed; Based on the feature vector of each of the text data to be processed, classifying the text data to be processed in the text data set to be processed to obtain multiple categories of text data to be processed; Based on the multiple categories, the to-be-processed text data of the to-be-processed text data set are distributed to the multiple distributed servers, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

[0103] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data deduplication method for feature extraction, characterized in that: The method comprises: Building a set of text data to be processed based on the received text data to be processed sent by the multiple distributed servers and the text data to be processed sent by the multiple data sources; Based on the term frequency-inverse text frequency TF-IDF method, determine the TF-IDF weight of each word in the text data set to be processed, and based on the TF-IDF weight, determine the feature vector of each text data to be processed; Based on the feature vector of each of the text data to be processed, classifying the text data to be processed in the text data set to be processed to obtain multiple categories of text data to be processed; Based on the multiple categories, the to-be-processed text data of the to-be-processed text data set are distributed to the multiple distributed servers, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

2. The data deduplication method for feature extraction according to claim 1, characterized in that: The method of classifying the text data to be processed in the text data set to be processed based on the feature vector of each text data to be processed to obtain multiple categories of text data to be processed includes: Determine the feature vector of each piece of text data to be processed in the set of text data to be processed based on a formula for determining the feature vector of the text data to be processed; The formula for determining the feature vector of the text data to be processed is determined based on the feature vector of each word in the text data to be processed, the number of all words contained in the text data to be processed, and the word frequency-inverse text frequency TF-IDF weight calculated in the text data to be processed; Determining similar text data to be processed in the set of text data to be processed based on the values ​​of the feature vectors of the text data to be processed; Similar text data to be processed in the text data set to be processed are grouped into one category, so as to classify the text data to be processed in the text data set to be processed and obtain multiple categories of text data to be processed.

3. The data deduplication method for feature extraction according to claim 2, characterized in that: The formula for determining the feature vector of the text data to be processed is: ; in, Text data to be processed The feature vector of n is the text data to be processed The number of all words contained in Text data to be processed Words in In the text data to be processed The TF-IDF weight in .

4. The data deduplication method for feature extraction according to claim 1, characterized in that: The method based on term frequency-inverse text frequency TF-IDF is used to determine the TF-IDF weight of each word in the text data set to be processed, including: Based on the term frequency-inverse document frequency TF-IDF method, determining the term frequency of each term in the text data set to be processed and the inverse document frequency of each term; Based on the word frequency and the inverse document frequency, the TF-IDF weight of each word in the text data set to be processed is determined.

5. The data deduplication method for feature extraction according to claim 1, characterized in that: The method of allocating the to-be-processed text data of the to-be-processed text data set to the multiple distributed servers based on the multiple categories includes: Determining the amount of the to-be-processed text data in each category among the multiple categories of to-be-processed text data; The resource utilization rates of the plurality of distributed servers are obtained, and based on the resource utilization rates of the distributed servers and the amount of the text data to be processed in each category, the text data to be processed of the text data set to be processed are distributed to the plurality of distributed servers.

6. The method for deduplication of feature extraction according to claim 1, characterized in that: After constructing the text data set to be processed, the process further includes: Performing data preprocessing on the text data to be processed in the text data set to be processed; The data preprocessing includes text segmentation, removal of stop words and removal of punctuation marks.

7. The data deduplication method for feature extraction according to claim 1, characterized in that: The feature vector of the text data to be processed is constructed based on taking the TF-IDF weight of each word in the text data to be processed as a vector dimension.

8. A data deduplication device for feature extraction, characterized in that: include: A collection construction module, used to construct a collection of text data to be processed based on the received text data to be processed sent by multiple distributed servers and the text data to be processed sent by multiple data sources; A weight determination module, used to determine the TF-IDF weight of each word in the text data set to be processed based on the word frequency-inverse text frequency TF-IDF method, and determine the feature vector of each text data to be processed based on the TF-IDF weight; A classification module, used for classifying the to-be-processed text data in the to-be-processed text data set based on the feature vector of each to-be-processed text data, to obtain a plurality of categories of to-be-processed text data; A distribution processing module is used to distribute the to-be-processed text data of the to-be-processed text data set to the multiple distributed servers based on the multiple categories, so that the to-be-processed text data of the same category are distributed to the same distributed server, and each distributed server performs deduplication processing based on the received to-be-processed text data of the same category.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data deduplication method for feature extraction according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data deduplication method for feature extraction as described in any one of claims 1 to 7 is implemented.