A document deduplication method, system and storage medium based on metadata and sliding window

Abstract

The present invention belongs to the technical field of big data processing, and in particular, relates to a document deduplication method, system and storage medium based on metadata and sliding windows, comprising the following steps: extracting fingerprint values ​​from metadata of existing documents, storing metadata of documents to be added into a memory-disk hybrid storage system, calculating fingerprint values ​​of documents to be added according to feature value fields of the documents, and matching with fingerprint values ​​cached in distributed memory using a sliding window algorithm, thereby determining whether there are new documents. The present invention can solve the problem of lack of deduplication of repetitive data for metadata of digital resources from different sources in the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular relates to a document deduplication method, system and storage medium based on metadata and sliding windows. Background Art

[0002] With the continuous development of big data processing technology, the data generated by various information systems are becoming more and more relevant, thus forming information networks, such as social networks, mobile Internet, biomolecular relationship networks, digital resources, and knowledge graphs. In the library industry, research hotspots are mainly focused on digital resource information networks composed of digital resources, including digital resource organization, management, disclosure, use, and analysis. The purpose of this research hotspot is to improve service quality and service efficiency, that is, to provide readers with faster, better, and more convenient services in the entire document service process from user search requests to resource / knowledge acquisition. However, the current digital resource information network presents the following characteristics: (1) The amount of data is extremely large; with the explosive growth of global information, resource metadata is controlled by different digital resource providers, with a total amount of more than 10 billion items, and the storage cost is high. (2) The data duplication rate is high; due to commercial or technical barriers between digital resource providers, the resources of different providers have characteristics such as overlap, overlap, and complementarity, and the resource access quality is low. (3) Metadata is heterogeneous and multi-modal; almost all digital resource providers have their own specific standard open access methods, with uneven fields, different meanings, and different formats, making knowledge conversion difficult.

[0003] Based on the above problems, relevant scholars and institutions have carried out a lot of research work. Chongqing University Library explored the classification of digital resources based on subject catalogs, built a paper-electronic library catalog, and reorganized and integrated paper and digital document resources; Tsinghua University Library established the Tsinghua University Library Integrated Resource Management Platform, which includes construction ideas and data migration plans. Therefore, the library industry has tried to unify the management of resource metadata, but still does not have the ability to merge and process metadata from more than 10 sources, and lacks an implementation plan for deduplication of duplicate data. Summary of the invention

[0004] The technical problem solved by the present invention is to provide a document deduplication method, system and storage medium based on metadata and sliding windows to solve the problem of lack of repetitive data deduplication for digital resource metadata from different sources in the prior art.

[0005] The basic solution provided by the present invention is a document deduplication method based on metadata and sliding windows, comprising:

[0006] S1: Get the Simhash fingerprint value of the existing document metadata;

[0007] S2: Obtain the metadata of the document to be added, calculate the newhash fingerprint value of the metadata of the document to be added, and use the sliding window algorithm to match it with the Simhash fingerprint value, and form the matching results into a set 0;

[0008] S3: Determine whether there are new documents based on the number of elements in set 0;

[0009] The sliding window algorithm in S2 is specifically as follows: according to the document metadata type, the platform and time where the document first appears are obtained, and the time is used as a node or the platform and time are combined as a node. If the time is used as a node, the time is advanced forward by a preset time period, and the Simhash value within the time period is selected for matching; if the platform and time are combined as a node, the platform where the document first appears and the time of the first appearance are used, and the time is advanced forward by a certain time period, and the Simhash fingerprint values ​​of the document metadata of other platforms of the same type are read for matching.

[0010] The principle and advantage of the present invention are as follows: a document deduplication method based on metadata and sliding windows of the present invention first obtains the fingerprint value of the existing document metadata, calculates the fingerprint value of the document metadata to be added, and matches it with the existing document metadata through the sliding window. The algorithm of the sliding window is based on the type of document. There are 10 types of documents, which are different according to the type of document and the platform on which they are published. At the same time, according to the applicant's investigation, the continuous popularity of the document only lasts for a period of time, such as 5 years. After more than 5 years, the popularity of the document will drop immediately. As a technology known to all in the industry, basically no scholars will use the technology of the document to study and publish related documents. It is undoubtedly meaningless to match the data before this time through the server, and it will also increase To reduce the pressure on the server, a sliding window algorithm is used to match the document according to the type of document, the platform and time when the document first appeared. If the document cannot be traced back to the platform where it appeared, the time of its first appearance is used as the basis, and the preset time period is advanced forward. The Simhash fingerprint value cached in the distributed memory within the time period is selected for matching. If the document can be traced back to the platform and time when it first appeared, the time is used as the basis, and a certain time period is advanced forward. The Simhash fingerprint value of the document metadata of other platforms of the same type in the distributed memory is read for matching. Therefore, the above sliding window algorithm combines the time dimension and the space dimension, greatly improves the matching efficiency of the nwehash fingerprint value, and also reduces the computing pressure on the server.

[0011] Therefore, the advantages of this application are: (1) by using a sliding window algorithm to perform fingerprint matching of document metadata, it is possible to remove duplicate data for digital resource metadata from different sources; (2) a sliding window is used for matching, which improves the matching efficiency and reduces the computing pressure on the server.

[0012] Furthermore, the S1 further includes:

[0013] S1-1: Store the existing document metadata in the disk, extract the Simhash fingerprint value of the document in the document metadata, and cache the Simhash fingerprint value of the document metadata in the distributed memory.

[0014] Beneficial effect: Storing fingerprint values ​​in distributed memory can improve matching efficiency.

[0015] Further, the step S2 comprises:

[0016] S2-1: storing the metadata of the document to be added in the memory-disk hybrid storage system;

[0017] S2-2: Extract the feature values ​​of the metadata of the document to be added in the memory-disk hybrid storage system by field, and perform weight processing;

[0018] S2-3: Normalize the eigenvalue weights and smooth the discrete weight values;

[0019] S2-4: extract the read feature value into the memory, and use the same hash function to calculate an f-bit fingerprint value a for each feature value according to the 64-bit hash requirement;

[0020] S2-5: Weighted processing; if the i-th bit of the fingerprint value a is 1 (i is between 1 and f), the i-th bit of a is added with the weight of the feature; if the i-th bit of the fingerprint value a is 0 (i is between 1 and f), the i-th bit of a is subtracted with the weight of the feature;

[0021] S2-6: Accumulate the weights in S2-5 to form a sequence string with only one;

[0022] S2-7: Perform dimensionality reduction processing on the sequence string and output newhash as the fingerprint value;

[0023] S2-8: Use the sliding window algorithm to compare the newhash fingerprint value with the Simhash fingerprint value of the document in the distributed memory by Hamming distance, and find the newhash fingerprint value that meets the bit difference threshold. These newhash fingerprint values ​​constitute set 0.

[0024] Beneficial effect: Through step S2-1 to step S2-8, the improved Simhash algorithm is used to improve the accuracy of deduplication. The improved Simhash algorithm is based on the existing word segmentation technology, but the identification information data of the document is calculated by analyzing the characteristic values ​​of each field, and the multi-dimensional characteristic values ​​are normalized and smoothed according to their discrete characteristics. After processing, the nwehash fingerprint value of the document is obtained by fingerprint value calculation, wherein the characteristic values ​​of the document include title, signing organization, auxiliary field format and doi, etc. Compared with the method of deduplication by reading phrases through word segmentation technology in the prior art, the detection range of the read field characteristic value is wider and the accuracy is higher. For example: for deduplication of the difference in the auxiliary field format of two documents, the auxiliary field format of human papillomavirus in the first document is: human papillomavirus, HPV; and the auxiliary field format of human papillomavirus in another document is: HPV, so the auxiliary field formats of the term in the two comparative documents are converged, thereby greatly improving the accuracy of document deduplication; and the metadata of the document to be added is stored in a hybrid storage system of memory-disk to speed up the writing of document metadata and standardize the storage format of document metadata. At the same time, the calculation is performed in memory, which improves the execution efficiency of the algorithm and reduces the time complexity.

[0025] Further, the S3 includes:

[0026] S3-1: If the number of elements in set 0 is greater than 1, manual intervention is performed to determine whether the document represented by the newhash fingerprint value is the same as the document corresponding to the Simhash fingerprint value cached in the distributed memory;

[0027] S3-2: If the number of elements in set 0 = 1, it is determined that the document represented by the newhash fingerprint value is a duplicate of the document represented by the simhash fingerprint value;

[0028] S3-3: If the number of elements in set 0 = 0, then add a new non-duplicate document metadata;

[0029] S3-4: Merge duplicate document metadata based on the priority of the source database;

[0030] If there is a new document, the newhash fingerprint value of the document is cached in the distributed memory, and the metadata of the document is stored in the disk.

[0031] Beneficial effect: S3-1 is manually checked to confirm whether it is duplicate or non-duplicate, in order to ensure accuracy and recall rate; through S3-2 to S3-4, it can be determined whether there is new document metadata, and at the same time, data merging is performed for duplicate document metadata to ensure that there is no missing document metadata.

[0032] Furthermore, it also includes:

[0033] S4: After calculating a piece of document metadata, clear the document metadata stored in the memory-disk hybrid storage system;

[0034] S5: The metadata processing of this document is completed, and S2 is repeated until the metadata processing of the document to be added is complete.

[0035] Beneficial effect: After calculating a document metadata, the stored document metadata is cleaned up, which can alleviate the storage pressure of the hybrid storage system.

[0036] A document deduplication system based on metadata and sliding windows, including:

[0037] Data acquisition module: used to obtain the Simhash fingerprint value of existing document metadata;

[0038] Data processing module: used to obtain the metadata of the document to be added, calculate the newhash fingerprint value of the metadata of the document to be added, and use the sliding window algorithm to match it with the Simhash fingerprint value, and form the matching results into a set 0;

[0039] Judgment module: used to judge whether there are new documents according to the number of elements in set 0;

[0040] The sliding window algorithm is specifically as follows: according to the document type, the platform and time where the document first appears are obtained, and the time is used as a node or the platform and time are combined as a node. If the time is used as a node, the time is advanced forward by a certain period of time, and the Simhash value within the time period is selected for matching; if the platform and time are combined as a node, the platform where the document first appears and the time of the first appearance are used, and the Simhash fingerprint value of the document metadata of other platforms of the same type is read for matching.

[0041] Furthermore, the data acquisition module includes a distributed memory unit, and the distributed memory unit is used to cache the Simhash fingerprint value of the document metadata into the distributed memory when the Simhash fingerprint value of the existing document metadata is acquired.

[0042] Furthermore, the data processing module includes a hybrid storage unit, a data extraction unit and a calculation unit. The hybrid storage unit is used to store the document metadata to be added in the memory-disk hybrid storage system; the data extraction unit is used to extract the feature values ​​of the document metadata to be added in the memory-disk hybrid storage system in a field manner; the calculation unit is used to perform preprocessing according to the feature values ​​extracted by the data extraction unit, generate a newhash fingerprint value, and compare the newhash fingerprint value with the document Simhash fingerprint value in the distributed memory by a sliding window algorithm. Hamming distance comparison is performed to find the newhash fingerprint value that meets the bit difference threshold, and these newhash fingerprint values ​​constitute a set 0; the preprocessing includes feature value weight processing, hash calculation, weighted processing, weighted accumulation and dimensionality reduction processing.

[0043] Further, the judgment module includes a first judgment unit, a second judgment unit, a third judgment unit and a data merging unit, wherein the first judgment unit is used to manually intervene to judge whether the document represented by the newhash fingerprint value is a duplicate of the document corresponding to the Simhash fingerprint value cached in the distributed memory when the number of elements in the set 0 is greater than 1; the second judgment unit is used to judge whether the document represented by the newhash fingerprint value is a duplicate of the document represented by the Simhash fingerprint value when the number of elements in the set 0 is equal to 1; the third judgment unit is used to add a non-duplicate document metadata when the number of elements in the set 0 is equal to 0; and the data merging unit is used to merge duplicate document metadata according to the priority of the source database.

[0044] Furthermore, it also includes a distributed storage module and a clearing module. The distributed storage module is used to cache the newhash fingerprint value of a new document in the distributed memory and store the document metadata in the disk when there is a new document; the clearing module is used to clean up the document metadata stored in the hybrid storage unit after the data processing module processes the document metadata.

[0045] A document deduplication storage medium based on metadata and sliding windows stores a computer program. When the computer program is executed by a processor, the processor executes the document deduplication method based on metadata and sliding windows as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of an embodiment of the present invention;

[0047] Figure 2 is a functional block diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. At the same time, although a logical order is shown in the embodiments, in some cases, the steps shown or described may be performed in an order different from that shown herein.

[0049] The following is further described in detail through specific implementation methods:

[0050] The embodiment is basically as follows Figure 1 As shown: A document deduplication method based on metadata and sliding window, including:

[0051] S1: Get the Simhash fingerprint value of the existing document metadata;

[0052] Among them, S1 includes:

[0053] S1-1: Store the existing document metadata in the disk, extract the Simhash fingerprint value of the document in the document metadata, and cache the Simhash fingerprint value of the document metadata in the distributed memory.

[0054] In this embodiment, S1 is based on the advantage of fast read and write speed of distributed memory, so as to facilitate direct matching with Simhash fingerprint value in distributed memory in subsequent matching, thereby reducing matching time and reducing server pressure. In addition, in this embodiment, a file block Simhash fingerprint value set is created in distributed memory, wherein the Simhash fingerprint value set includes: one part is Simhash fingerprint values ​​appearing on the same type of platform, and the other part is according to the first appearance time of Simhash fingerprint value, for example, according to 5 years as the time node, Simhash fingerprint values ​​appearing from 2005 to 2010 are grouped into one file block.

[0055] S2: Obtain the metadata of the document to be added, calculate the newhash fingerprint value of the metadata of the document to be added, and use the sliding window algorithm to match it with the Simhash fingerprint value, and form the matching results into a set 0;

[0056] In this embodiment, the sliding window algorithm obtains the platform and time when the document first appears according to the document type. The document types include monographs, journals, books, newspapers, conference documents, scientific reports, standard documents, patent documents, dissertations and government publications. The current document platform usually does not include all documents. For example, patent documents usually use the State Intellectual Property Office as the main publication platform. Therefore, the documents belonging to each document platform are distinguished by file blocking in the distributed memory, which is convenient for the sliding window algorithm to identify and read.

[0057] The specific algorithm of the sliding window is: according to the document type, obtain the platform and time where the document first appears, and use the time as a node or the platform and time as a node. Specifically, according to the document type, traverse the platform and time where the document first appears in the distributed memory. There is a situation where the document cannot find the platform where it first appears or appears on multiple platforms at the same time. If this is the case, use the time as a node, advance the preset time period forward, and select the Simhash value within the time period for matching. In this embodiment, the preset time period is 5 years. For example, if a document first appears in 2010, 2010 is used as the node. Time node, traverse the Simhash fingerprint values ​​of documents between 2005 and 2010 in the distributed memory; if the platform and time where the document first appeared can be queried by traversing the distributed memory, then the platform and time are combined as nodes, and the platform and time where the document first appeared are advanced by a certain period of time, and the Simhash fingerprint values ​​of document metadata on other platforms of the same type are read for matching. For example, if the platform where a document first appeared is EI and the time of its first appearance is 2010, then traverse the Simhash fingerprint values ​​of all documents in the distributed memory between 2005 and 2010 except the EI platform.

[0058] In addition, the S2 comprises the following steps:

[0059] S2-1: storing the metadata of the document to be added in the memory-disk hybrid storage system;

[0060] S2-2: Extract the feature values ​​of the metadata of the document to be added in the memory-disk hybrid storage system by field, and perform weight processing;

[0061] S2-3: Normalize the eigenvalue weights and smooth the discrete weight values;

[0062] S2-4: extract the read feature value into the memory, and use the same hash function to calculate an f-bit fingerprint value a for each feature value according to the 64-bit hash requirement;

[0063] S2-5: Weighted processing; if the i-th bit of the fingerprint value a is 1 (i is between 1 and f), the i-th bit of a is added with the weight of the feature; if the i-th bit of the fingerprint value a is 0 (i is between 1 and f), the i-th bit of a is subtracted with the weight of the feature;

[0064] S2-6: Accumulate the weights in S2-5 to form a sequence string with only one. Specifically, according to S2-5, if the element at position i is greater than 0, then set the i-th position of newhash to 1; if the element at position i is less than 0, then set the i-th position of newhash to 0, thereby obtaining the newhash fingerprint value of the document.

[0065] S2-7: Perform dimensionality reduction processing on the sequence string and output newhash as the fingerprint value;

[0066] S2-8: Use the sliding window algorithm to compare the newhash fingerprint value with the document Simhash fingerprint value in the distributed memory by Hamming distance, and find the newhash fingerprint value that meets the difference threshold of the bit position. These newhash fingerprint values ​​constitute a set 0. In this embodiment, the difference threshold of the bit position is 3, and finding the newhash fingerprint value that meets the difference threshold of the bit position is finding the newhash fingerprint value with a bit position ≤ 3.

[0067] The Hamming distance comparison is specifically:

[0068] newhash=x1、x2……x i , Simhash=y1、y2……y i ;

[0069] The calculation formula of Hamming distance is:

[0070]

[0071] If D≤3, then all newhash fingerprint values ​​with D≤3 form set 0.

[0072] In this embodiment, in step S2-2, the feature values ​​of the metadata of the document to be added in the memory-disk hybrid storage system are extracted in a field manner, including the title, signing organization, auxiliary field format and doi of the document, and the weights of these feature values ​​are normalized and the discrete weight values ​​are smoothed, which can more fully retain the comprehensive feature values ​​of the fingerprint and make the identification of duplicate data more accurate.

[0073] S3: Determine whether there are new documents based on the number of elements in set 0;

[0074] The S3 includes:

[0075] S3-1: If the number of elements in set 0 is greater than 1, manual intervention is performed to determine whether the document represented by the newhash fingerprint value is the same as the document corresponding to the Simhash fingerprint value cached in the distributed memory;

[0076] S3-2: If the number of elements in set 0 = 1, it is determined that the document represented by the newhash fingerprint value is a duplicate of the document represented by the simhash fingerprint value;

[0077] S3-3: If the number of elements in set 0 = 0, then add a new non-duplicate document metadata;

[0078] S3-4: Merge duplicate document metadata based on the priority of the source database;

[0079] If there is a new document, the newhash fingerprint value of the document is cached in the distributed memory, and the metadata of the document is stored in the disk.

[0080] In this embodiment, S3-4 is specifically as follows: if the number of elements in set 0 is ≥ 1, duplicate document metadata is determined, and the document metadata is merged according to the source database of the document metadata, wherein the priority of the source database in this embodiment is EI>SCOPUS>others. For example, as shown in the following table, two documents before deduplication and merging:

[0081]

[0082] After deduplication and merging:

[0083]

[0084]

[0085] S4: After calculating a piece of document metadata, clear the document metadata stored in the memory-disk hybrid storage system;

[0086] S5: The metadata processing of this document is completed, and S2 is repeated until the metadata processing of the document to be added is complete.

[0087] In this embodiment, the distributed memory includes redis and memcache, and redis is used in this embodiment; the disk includes pgsql and mysql, and pgsql is used in this embodiment; the memory-disk hybrid storage system includes clickhouse and kylin, and clickhouse is used in this embodiment.

[0088] like Figure 2 As shown, in another embodiment of the present embodiment, a document metadata deduplication system based on metadata and sliding window is also included, including:

[0089] Data acquisition module: used to obtain the Simhash fingerprint value of existing document metadata; the data acquisition module includes a distributed memory unit, which is used to cache the Simhash fingerprint value of the document metadata into the distributed memory when the Simhash fingerprint value of the existing document metadata is obtained.

[0090] Data processing module: used to obtain the metadata of the document to be added, calculate the newhash fingerprint value of the metadata of the document to be added, and use the sliding window algorithm to match it with the Simhash fingerprint value, and form the matching results into set 0.

[0091] The sliding window algorithm is specifically as follows: according to the document type, the platform and time where the document first appears are obtained, and the time is used as a node or the platform and time are combined as a node. If the time is used as a node, the time is advanced forward by a certain period of time, and the Simhash value within the time period is selected for matching; if the platform and time are combined as a node, the platform where the document first appears and the time of the first appearance are used, and the Simhash fingerprint value of the document metadata of other platforms of the same type is read for matching.

[0092] The data processing module includes a hybrid storage unit, a data extraction unit and a calculation unit. The hybrid storage unit is used to store the document metadata to be added in the memory-disk hybrid storage system; the data extraction unit is used to extract the feature values ​​of the document metadata to be added in the memory-disk hybrid storage system in a field manner; the calculation unit is used to perform preprocessing according to the feature values ​​extracted by the data extraction unit, generate a newhash fingerprint value, and compare the newhash fingerprint value with the document Simhash fingerprint value in the distributed memory by a sliding window algorithm. Hamming distance comparison is performed to find the newhash fingerprint value that meets the difference threshold of the bit position, and these newhash fingerprint values ​​constitute a set 0; the preprocessing includes feature value weight processing, hash calculation, weighted processing, weighted accumulation and dimensionality reduction processing.

[0093] The judgment module is used to judge whether there is a new document according to the number of elements in the set 0; the judgment module includes a first judgment unit, a second judgment unit, a third judgment unit and a data merging unit, the first judgment unit is used to judge by manual intervention whether the document represented by the newhash fingerprint value is repeated with the document corresponding to the Simhash fingerprint value cached in the distributed memory when the number of elements in the set 0 is greater than 1; the second judgment unit is used to judge whether the document represented by the newhash fingerprint value is repeated with the document represented by the Simhash fingerprint value when the number of elements in the set 0 is equal to 1; the third judgment unit is used to add a non-duplicate document metadata when the number of elements in the set 0 is equal to 0; the data merging unit is used to merge the duplicate document metadata according to the priority of the source database.

[0094] Distributed storage module: used to cache the newhash fingerprint value of a new document in the distributed memory and store the metadata of the document in the disk.

[0095] Clearing module: used to clear a piece of document metadata stored in the hybrid storage unit after the data processing module has processed the piece of document metadata.

[0096] In another embodiment of the present embodiment, a document deduplication storage medium based on metadata and sliding window is also included, which stores a computer program. When the computer program is executed by a processor, the processor executes the document deduplication method based on metadata and sliding window as described above.

[0097] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A document deduplication method based on metadata and sliding window, characterized by: include: S1: Get the Simhash fingerprint value of the existing document metadata; S2: Obtain the metadata of the document to be added, calculate the newhash fingerprint value of the metadata of the document to be added, and use the sliding window algorithm to match it with the Simhash fingerprint value, and form the matching results into a set 0; S3: Determine whether there are new documents based on the number of elements in set 0; The sliding window algorithm in S2 is specifically as follows: according to the document metadata type, the platform and time where the document first appears are obtained, and the time is used as a node or the platform and time are combined as a node. If the time is used as a node, the time is advanced forward by a preset time period, and the Simhash value within the time period is selected for matching; if the platform and time are combined as a node, the platform where the document first appears and the time of the first appearance are used, and the Simhash fingerprint value of the document metadata of other platforms of the same type is read for matching; The step S2 comprises: S2-1: storing the metadata of the document to be added in the memory-disk hybrid storage system; S2-2: Extract the feature values ​​of the metadata of the document to be added in the memory-disk hybrid storage system by field, and perform weight processing; S2-3: Normalize the eigenvalue weights and smooth the discrete weight values; S2-4: extract the read feature value into the memory, and use the same hash function to calculate an f-bit fingerprint value a for each feature value according to the 64-bit hash requirement; S2-5: weighted processing; if the i-th bit of the fingerprint value a is 1, where i is between 1 and f, the i-th bit of a is added with the weight of the feature; if the i-th bit of the fingerprint value a is 0, the i-th bit of a is subtracted with the weight of the feature; S2-6: Accumulate the weights in S2-5 to form a sequence string with only one; S2-7: Perform dimensionality reduction processing on the sequence string and output newhash as the fingerprint value; S2-8: Use the sliding window algorithm to compare the newhash fingerprint value with the Simhash fingerprint value of the document in the distributed memory by Hamming distance, and find the newhash fingerprint value that meets the bit difference threshold. These newhash fingerprint values ​​constitute set 0.

2. The method for deduplicating documents based on metadata and sliding windows according to claim 1, characterized in that: The S1 further comprises: S1-1: Store the existing document metadata in the disk, extract the Simhash fingerprint value of the document in the document metadata, and cache the Simhash fingerprint value of the document metadata in the distributed memory.

3. The document deduplication method based on metadata and sliding window according to claim 2, characterized in that: The S3 includes: S3-1: If the number of elements in set 0 is greater than 1, manual intervention is performed to determine whether the document represented by the newhash fingerprint value is the same as the document corresponding to the Simhash fingerprint value cached in the distributed memory; S3-2: If the number of elements in set 0 = 1, it is determined that the document represented by the newhash fingerprint value is a duplicate of the document represented by the simhash fingerprint value; S3-3: If the number of elements in set 0 = 0, then add a new non-duplicate document metadata; S3-4: Merge duplicate document metadata based on the priority of the source database; If there is a new document, the newhash fingerprint value of the document is cached in the distributed memory, and the metadata of the document is stored in the disk.

4. The document deduplication method based on metadata and sliding window according to claim 3, characterized in that: Also includes: S4: After calculating a piece of document metadata, clear the document metadata stored in the memory-disk hybrid storage system; S5: The metadata processing of this document is completed, and S2 is repeated until the metadata processing of the document to be added is complete.

5. A document deduplication system based on metadata and sliding window, characterized by: include: Data acquisition module: used to obtain the Simhash fingerprint value of existing document metadata; Data processing module: used to obtain the metadata of the document to be added, calculate the newhash fingerprint value of the metadata of the document to be added, and use the sliding window algorithm to match it with the Simhash fingerprint value, and form the matching results into a set 0; Judgment module: used to judge whether there are new documents according to the number of elements in set 0; The sliding window algorithm is specifically as follows: according to the document type, the platform and time where the document first appears are obtained, and the time is used as a node or the platform and time are combined as a node. If the time is used as a node, the time is advanced a certain period of time, and the Simhash value within the time period is selected for matching; if the platform and time are combined as a node, the platform where the document first appears and the time of the first appearance are used, and the Simhash fingerprint value of the metadata of documents on other platforms of the same type is read for matching; The data acquisition module includes a distributed memory unit, and the distributed memory unit is used to cache the Simhash fingerprint value of the document metadata in the distributed memory when the Simhash fingerprint value of the existing document metadata is obtained; the data processing module includes a hybrid storage unit, a data extraction unit and a calculation unit, and the hybrid storage unit is used to store the document metadata to be added in the memory-disk hybrid storage system; The data extraction unit is used to extract the feature values ​​of the metadata of the document to be added in the memory-disk hybrid storage system in a field manner. The calculation unit is used to perform preprocessing according to the feature values ​​extracted by the data extraction unit to generate a newhash fingerprint value, and compare the newhash fingerprint value with the Simhash fingerprint value of the document in the distributed memory by a Hamming distance through a sliding window algorithm to find the newhash fingerprint value that meets the difference threshold of the bit position, and these newhash fingerprint values ​​constitute a set 0; the preprocessing includes feature value weight processing, hash calculation, weighted processing, weighted accumulation and dimensionality reduction processing.

6. The document deduplication system based on metadata and sliding window according to claim 5, characterized in that: The judgment module includes a first judgment unit, a second judgment unit, a third judgment unit and a data merging unit. The first judgment unit is used to manually intervene to judge whether the document represented by the newhash fingerprint value is repeated with the document corresponding to the Simhash fingerprint value cached in the distributed memory when the number of elements in the set 0 is greater than 1; the second judgment unit is used to judge whether the document represented by the newhash fingerprint value is repeated with the document represented by the Simhash fingerprint value when the number of elements in the set 0 is equal to 1; the third judgment unit is used to add a non-duplicate document metadata when the number of elements in the set 0 is equal to 0; the data merging unit is used to merge the duplicate document metadata according to the priority of the source database.

7. The document deduplication system based on metadata and sliding window according to claim 5, characterized in that: Also includes: It also includes a distributed storage module and a clearing module. The distributed storage module is used to cache the newhash fingerprint value of a new document in the distributed memory and store the document metadata in the disk when there is a new document; the clearing module is used to clean up the document metadata stored in the hybrid storage unit after the data processing module processes the document metadata.

8. A document deduplication storage medium based on metadata and sliding windows, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor executes a document deduplication method based on metadata and a sliding window as described in any one of claims 1 to 4.

Images