A document duplication checking method and terminal

Abstract

The present invention discloses a document duplicate checking method and terminal, which performs word segmentation processing on a document, takes each word segmentation as a feature vector, and calculates the feature value of each picture as the feature vector of the picture; takes the TF-IDF value of each word segmentation relative to the document as the weight value of each word segmentation, and multiplies the number of occurrences of each picture in the document by a preset basic weight to obtain the weight value of each picture; thereby generating a signature value of the document; and judging the similarity of the two documents according to the signature values ​​of the two documents to obtain a duplicate checking result. The present invention distinguishes between text and pictures for the weight value of each feature vector in the document, and retains the feature value of the picture as a feature vector when parsing the document. In the weighting stage, the text uses the TF-IDF algorithm to calculate the weight value, and the picture uses the number of occurrences*preset basic weight to calculate the weight value, that is, the picture and the text vocabulary are set to different weights to improve the accuracy of the document duplicate checking.

Description

Technical Field

[0001] The present invention relates to the technical field of document duplicate checking, and in particular to a document duplicate checking method and a terminal. Background Art

[0002] Google's Simhash algorithm is a dimensionality reduction algorithm that aims to represent high-dimensional vectors with lower-dimensional signatures. It is an efficient hashing technology for similar text detection. The Simhash algorithm generates a document signature value to represent the document, and the similarity between signature values ​​is determined by comparing the Hamming distance between documents. This distance is used to determine whether the documents are similar.

[0003] However, the traditional Simhash algorithm generally has two problems:

[0004] (1) Courseware documents contain a large number of pictures, which have different weights from texts. However, traditional Simhash does not distinguish between them, which affects the accuracy of duplicate checking.

[0005] (2) For documents that use templates, there may be a large number of repeated images or words, resulting in a loss of accuracy in the calculated fingerprint results. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a document duplication checking method and terminal to improve the accuracy of document duplication checking.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A document duplication checking method comprises the steps of:

[0009] S1. Perform word segmentation on the document, take each word segment as a feature vector, and calculate the feature value of each picture as the feature vector of the picture;

[0010] S2, taking the TF-IDF value of each word relative to the document as the weight value of each word, and multiplying the number of occurrences of each picture in the document by the preset basic weight to obtain the weight value of each picture;

[0011] S3, generating a signature value of the document according to the feature vector and the corresponding weight value;

[0012] S4. Determine the similarity between the two documents based on their signature values ​​to obtain a duplicate checking result.

[0013] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0014] A document duplication checking terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0015] S1. Perform word segmentation on the document, take each word segment as a feature vector, and calculate the feature value of each picture as the feature vector of the picture;

[0016] S2, taking the TF-IDF value of each word relative to the document as the weight value of each word, and multiplying the number of occurrences of each picture in the document by the preset basic weight to obtain the weight value of each picture;

[0017] S3, generating a signature value of the document according to the feature vector and the corresponding weight value;

[0018] S4. Determine the similarity between the two documents based on their signature values ​​to obtain a duplicate checking result.

[0019] The beneficial effects of the present invention are: a document duplicate checking method and terminal, which distinguishes text and pictures for the weight value of each feature vector in the document, and retains the feature value of the picture as the feature vector when parsing the document. In the weighting stage, the text uses the TF-IDF algorithm to calculate the weight value, and the picture uses the number of occurrences * preset basic weight to calculate the weight value, that is, setting different weights for pictures and text words to improve the accuracy of document duplicate checking. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the main process of a document duplication checking method according to an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a specific process of a document duplication checking method according to an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the structural components of a document duplication checking method according to an embodiment of the present invention;

[0023] Figure 4 The present invention is a schematic diagram of the structure of a document duplication checking terminal according to an embodiment of the present invention.

[0024] Description of labels:

[0025] 1. A document duplication checking terminal; 2. A processor; 3. A memory. DETAILED DESCRIPTION

[0026] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0027] Please refer to Figures 1 to 3 , a document duplication checking method, comprising the steps of:

[0028] S1. Perform word segmentation on the document, take each word segment as a feature vector, and calculate the feature value of each picture as the feature vector of the picture;

[0029] S2, taking the TF-IDF value of each word relative to the document as the weight value of each word, and multiplying the number of occurrences of each picture in the document by the preset basic weight to obtain the weight value of each picture;

[0030] S3, generating a signature value of the document according to the feature vector and the corresponding weight value;

[0031] S4. Determine the similarity between the two documents based on their signature values ​​to obtain a duplicate checking result.

[0032] From the above description, it can be seen that the beneficial effects of the present invention are: the weight value of each feature vector in the document distinguishes between text and pictures. When parsing the document, the feature value of the picture is retained as the feature vector. In the weighting stage, the text uses the TF-IDF algorithm to calculate the weight value, and the picture uses the number of occurrences * preset basic weight to calculate the weight value, that is, setting different weights for pictures and text words to improve the accuracy of document duplication checking.

[0033] Furthermore, the step S3 specifically includes the following steps:

[0034] S31, performing hash calculation on each of the feature vectors to obtain a hash value of each of the feature vectors;

[0035] S32, adding weight values ​​to each digit value in the hash value of each feature vector to obtain a weighted digital string of each feature vector, if a digit value in the hash value of the feature vector is 1, then the digit value of the weighted digital string of the feature vector is a positive weight value, if a digit value in the hash value of the feature vector is 0, then the digit value of the weighted digital string of the feature vector is a negative weight value;

[0036] S33, superimposing the weighted digital strings of all the feature vectors to obtain a weighted sequence string;

[0037] S34. Perform dimensionality reduction processing on the weighted sequence string to obtain the signature value of the document. If a certain digit value in the weighted sequence string is greater than 0, the value of this digit of the signature value of the document is 1. If a certain digit value in the weighted sequence string is less than or equal to 0, the value of this digit of the signature value of the document is 0.

[0038] From the above description, it can be seen that after the feature vector and the corresponding weight value of the document are processed as above, a signature value is finally used to compare the similarity, so as to facilitate the subsequent resource duplication check.

[0039] Furthermore, the step S1 specifically includes the following steps:

[0040] Use HanLP to perform word segmentation on the document and remove common stop words in the preset resource type to obtain the final word segmentation result, and use each word in the word segmentation result as a feature vector;

[0041] A grayscale value algorithm is used to calculate the hash value of each image as the feature vector of the image.

[0042] From the above description, it can be seen that in a preset resource type, common stop words that appear repeatedly are removed to ensure that the final word segmentation result can better express the document itself, thereby ensuring the accuracy of document duplication checking.

[0043] Furthermore, the step S2 further includes the following steps:

[0044] After obtaining the weight value of each word segment, determine whether the weight value of each word segment is greater than the preset text maximum weight value, and if so, use the preset text maximum weight value as the weight value of the word segment;

[0045] After obtaining the weight value of each picture, it is determined whether the weight value of each picture is greater than a preset maximum weight value of the picture. If so, the preset maximum weight value of the picture is used as the weight value of the picture.

[0046] From the above description, it can be seen that a maximum value is set for the weight of the feature vector. When the calculated weight of the feature vector is greater than the maximum weight, the maximum weight is taken for weighting, which improves the accuracy of duplicate checking for documents containing a large number of repeated words using templates.

[0047] Furthermore, each time a document is received, the steps S1 to S3 are sequentially executed on the document, and the signature value of the document outputted in step S3 is stored;

[0048] The step S4 specifically comprises the following steps:

[0049] The Hamming distances of the signature values ​​of the two documents are compared. If the Hamming distance is less than a preset value, the two documents are considered to be duplicates, otherwise they are considered not to be duplicates.

[0050] From the above description, it can be seen that by calculating the Hamming distance, the similarity of documents can be quickly and accurately determined.

[0051] Please refer to Figure 4A document duplication checking terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0052] S1. Perform word segmentation on the document, take each word segment as a feature vector, and calculate the feature value of each picture as the feature vector of the picture;

[0053] S2, taking the TF-IDF value of each word relative to the document as the weight value of each word, and multiplying the number of occurrences of each picture in the document by the preset basic weight to obtain the weight value of each picture;

[0054] S3, generating a signature value of the document according to the feature vector and the corresponding weight value;

[0055] S4. Determine the similarity between the two documents based on their signature values ​​to obtain a duplicate checking result.

[0056] From the above description, it can be seen that the beneficial effects of the present invention are: the weight value of each feature vector in the document distinguishes between text and pictures. When parsing the document, the feature value of the picture is retained as the feature vector. In the weighting stage, the text uses the TF-IDF algorithm to calculate the weight value, and the picture uses the number of occurrences * preset basic weight to calculate the weight value, that is, setting different weights for pictures and text words to improve the accuracy of document duplication checking.

[0057] Furthermore, the step S3 specifically includes the following steps:

[0058] S31, performing hash calculation on each of the feature vectors to obtain a hash value of each of the feature vectors;

[0059] S32, adding weight values ​​to each digit value in the hash value of each feature vector to obtain a weighted digital string of each feature vector, if a digit value in the hash value of the feature vector is 1, then the digit value of the weighted digital string of the feature vector is a positive weight value, if a digit value in the hash value of the feature vector is 0, then the digit value of the weighted digital string of the feature vector is a negative weight value;

[0060] S33, superimposing the weighted digital strings of all the feature vectors to obtain a weighted sequence string;

[0061] S34. Perform dimensionality reduction processing on the weighted sequence string to obtain the signature value of the document. If a certain digit value in the weighted sequence string is greater than 0, the value of this digit of the signature value of the document is 1. If a certain digit value in the weighted sequence string is less than or equal to 0, the value of this digit of the signature value of the document is 0.

[0062] From the above description, it can be seen that after the feature vector and the corresponding weight value of the document are processed as above, a signature value is finally used to compare the similarity, so as to facilitate the subsequent resource duplication check.

[0063] Furthermore, the step S1 specifically includes the following steps:

[0064] Use HanLP to perform word segmentation on the document and remove common stop words in the preset resource type to obtain the final word segmentation result, and use each word in the word segmentation result as a feature vector;

[0065] A grayscale value algorithm is used to calculate the hash value of each image as the feature vector of the image.

[0066] From the above description, it can be seen that in a preset resource type, common stop words that appear repeatedly are removed to ensure that the final word segmentation result can better express the document itself, thereby ensuring the accuracy of document duplication checking.

[0067] Furthermore, the step S2 further includes the following steps:

[0068] After obtaining the weight value of each word segment, determine whether the weight value of each word segment is greater than the preset text maximum weight value, and if so, use the preset text maximum weight value as the weight value of the word segment;

[0069] After obtaining the weight value of each picture, it is determined whether the weight value of each picture is greater than a preset maximum weight value of the picture. If so, the preset maximum weight value of the picture is used as the weight value of the picture.

[0070] From the above description, it can be seen that a maximum value is set for the weight of the feature vector. When the calculated weight of the feature vector is greater than the maximum weight, the maximum weight is taken for weighting, which improves the accuracy of duplicate checking for documents containing a large number of repeated words using templates.

[0071] Furthermore, each time the processor receives a document, it sequentially executes the steps S1 to S3 on the document, and stores the signature value of the document outputted in step S3;

[0072] The step S4 specifically comprises the following steps:

[0073] The Hamming distances of the signature values ​​of the two documents are compared. If the Hamming distance is less than a preset value, the two documents are considered to be duplicates, otherwise they are considered not to be duplicates.

[0074] From the above description, it can be seen that by calculating the Hamming distance, the similarity of documents can be quickly and accurately determined.

[0075] Please refer to Figures 1 to 3 , Embodiment 1 of the present invention is:

[0076] A document duplication checking method comprises the steps of:

[0077] S1. Perform word segmentation on the document, take each word segment as a feature vector, and calculate the feature value of each picture as the feature vector of the picture;

[0078] In this embodiment, step S1 specifically includes the following steps:

[0079] Use HanLP to segment the document and remove common stop words in the preset resource type to obtain the final segmentation result. Each segmentation word in the segmentation result is used as a feature vector.

[0080] The grayscale value algorithm is used to calculate the hash value of each image as the feature vector of the image.

[0081] Among them, HanLP is a common text segmentation tool, and the gray value algorithm is a common algorithm for image processing, which will not be further described in detail in this embodiment.

[0082] Among them, common stop words in preset resource types, such as educational resources, common stop words include courseware, lesson plan, study plan, etc., such as patent documents, common stop words include described, embodiment, invention, utility model, etc., and so on.

[0083] S2. The TF-IDF value of each word relative to the document is used as the weight value of each word, and the number of occurrences of each image in the document is multiplied by the preset basic weight to obtain the weight value of each image;

[0084] In this embodiment, step S2 further includes the following steps:

[0085] After obtaining the weight value of each word segmentation, determine whether the weight value of each word segmentation is greater than the preset text maximum weight value. If so, use the preset text maximum weight value as the weight value of the word segmentation;

[0086] After obtaining the weight value of each image, determine whether the weight value of each image is greater than the preset maximum weight value of the image. If so, use the preset maximum weight value of the image as the weight value of the image.

[0087] Therefore, a fixed preset basic weight is set for the image, and the weight value of the image is obtained by multiplying it according to the number of times. The weight value is represented by weight.

[0088] The preset maximum text weight value can be adjusted according to the TF-IDF value, such as 3.0 in this embodiment, and the preset maximum image weight value is 1.0 in this embodiment.

[0089] TF-IDF is the abbreviation of term frequency-inverse document frequency, which is a commonly used weighting technology used in information retrieval and data mining. TF is term frequency and IDF is inverse document frequency index.

[0090] S3, generating a signature value of the document according to the feature vector and the corresponding weight value;

[0091] In this embodiment, step S3 specifically includes the following steps:

[0092] S31, performing hash calculation on each feature vector to obtain a hash value of each feature vector;

[0093] That is, use the same hash function to perform hash calculation on each feature vector to obtain the hash value of each feature vector. i , where the hash value is i The fingerprint length is N, and in this embodiment, N is 64, where i=1, 2, 3, ..., n, i represents the i-th feature vector, and n represents the number of all feature vectors.

[0094] S32, adding weight values ​​to each digit value in the hash value of each feature vector to obtain a weighted digital string of each feature vector, if the value of a certain digit in the hash value of the feature vector is 1, then the value of this digit in the weighted digital string of the feature vector is a positive weight value, if the value of a certain digit in the hash value of the feature vector is 0, then the value of this digit in the weighted digital string of the feature vector is a negative weight value;

[0095] That is, a certain hash ij If the value is 1, let this bit be weight i , if a hash ij If the value is 0, this bit is set to -weight i , where j = 1, 2, 3, ..., 64, j represents the jth bit, then hash ij Represents the j-th value of the hash value of the i-th feature vector.

[0096] S33, superimposing the weighted digital strings of all feature vectors to obtain a weighted sequence string;

[0097] Among them, if there are originally n eigenvectors, there are n weighted digital strings. These n weighted digital strings are superimposed bit by bit to obtain a weighted sequence string.

[0098] S34. Perform dimensionality reduction processing on the weighted sequence string to obtain the signature value of the document. If a certain digit value in the weighted sequence string is greater than 0, the value of this digit of the signature value of the document is 1. If a certain digit value in the weighted sequence string is less than or equal to 0, the value of this digit of the signature value of the document is 0.

[0099] As a result, each document finally obtains a 64-bit signature value, which is also called Simhash signature value, Simhash value, hash fingerprint, etc.

[0100] S4. Determine the similarity between the two documents based on their signature values ​​to obtain a duplicate checking result.

[0101] Step S4 specifically includes the following steps:

[0102] The Hamming distances of the signature values ​​of the two documents are compared. If the Hamming distance is less than a preset value, the two documents are considered to be duplicates, otherwise they are considered not to be duplicates.

[0103] In this embodiment, the preset distance is 3, that is, if the Hamming distance is less than 3, the two documents are considered to be duplicates.

[0104] Therefore, the above method is applied to a specific scenario, such as the scenario of educational resources in this embodiment, combined with Figure 2 and Figure 3 It can be seen that the specific implementation process includes the following:

[0105] (1) When an operator creates or modifies a resource file of a resource in the Resource Portal, the text content of the resource file will be parsed by the unified file processing service, and the resource ID, resource type, subject, document text content, and document image content will be pushed to the Hadoop Distributed File System (HDFS).

[0106] (2) Using the T+1 format, start the Spark offline computing task and load the data pushed to HDFS every day into the resource text content table of Hive.

[0107] (3) Each time a document is received, steps S1 to S3 are executed on the document in sequence, and the signature value of the document outputted in step S3 is stored:

[0108] According to the word segmentation method of step S1, the resource text content and resource image content of the day are segmented, and the word segmentation results of each document are saved in the resource text word segmentation table of Hive.

[0109] According to the weight value calculation method of step S2, for text content, first use the IDF training method to train the IDF value of each word segment under the resource type and subject, save it to the resource text IDF table of Hive, and obtain the weight value of the text word segmentation based on the word segmentation result and the IDF training result, and calculate the TF-IDF value of each word segmentation in each document as the weight value. For image content, calculate the number of times it appears in the document and multiply it by the preset basic weight to get the weight value of each image;

[0110] Then, the top 500 word segments of each document are taken out for hash calculation according to step S3 to obtain the 64-bit hash fingerprint of the resource document. Finally, the fingerprint calculation result is saved in the resource fingerprint table of Hive.

[0111] (4) Compare the newly added fingerprints every day with the latest fingerprints of resources of the same resource type to obtain resources that are determined to be duplicates. Save the resources that are determined to be duplicates every day and their fingerprint data in MongoDB (a database based on distributed file storage), and provide an API (Application Programming Interface) for querying duplicate resources.

[0112] (5) A "Resource Duplicate Query" menu is added to the "Resource Portal". The front-end calls the API provided by the back-end to query duplicate resources. Duplicate resources under a certain library and duplicate resources of a certain resource type can be queried.

[0113] Please refer to Figure 4 , Embodiment 2 of the present invention is:

[0114] A document duplication checking terminal 1 includes a memory 3, a processor 2, and a computer program stored in the memory 3 and executable on the processor 2, wherein the processor 2 implements the steps in the above-mentioned embodiment 1 when executing the computer program.

[0115] In summary, the present invention provides a document duplicate checking method and terminal, which remove common stop words that may appear repeatedly in advance, and then distinguish text and pictures by the weight value of each feature vector in the document. When the document is parsed, the feature value of the picture is retained as the feature vector, wherein a maximum value limit is set for the weight of the feature vector. When the weight calculation result of the feature vector is greater than the maximum weight, the maximum weight is taken for weighting. In the weighting stage, the text uses the TF-IDF algorithm to calculate the weight value, and the picture uses the number of occurrences * preset basic weight to calculate the weight value, that is, the picture and text vocabulary are set to different weights to improve the accuracy of document duplicate checking.

[0116] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A document duplication checking method, characterized in that: Includes steps: S1. Perform word segmentation on the document, take each word segment as a feature vector, and calculate the feature value of each picture as the feature vector of the picture, including: Use HanLP to perform word segmentation on the document and remove common stop words in the preset resource type to obtain the final word segmentation result, and use each word in the word segmentation result as a feature vector; Using a grayscale algorithm to calculate the hash value of each image as a feature vector of the image; S2, taking the TF-IDF value of each word segment relative to the document as the weight value of each word segment, judging whether the weight value of each word segment is greater than the preset maximum text weight value, if so, taking the preset maximum text weight value as the weight value of the word segment; and multiplying the number of occurrences of each picture in the document by the preset basic weight to obtain the weight value of each picture, judging whether the weight value of each picture is greater than the preset maximum picture weight value, if so, taking the preset maximum picture weight value as the weight value of the picture; S3, generating a signature value of the document according to the feature vector and the corresponding weight value; S4. Determine the similarity between the two documents based on their signature values ​​to obtain a duplicate checking result.

2. A document duplication checking method according to claim 1, characterized in that: The step S3 specifically comprises the following steps: S31, performing hash calculation on each of the feature vectors to obtain a hash value of each of the feature vectors; S32, adding weight values ​​to each digit value in the hash value of each feature vector to obtain a weighted digital string of each feature vector, if a digit value in the hash value of the feature vector is 1, then the digit value of the weighted digital string of the feature vector is a positive weight value, if a digit value in the hash value of the feature vector is 0, then the digit value of the weighted digital string of the feature vector is a negative weight value; S33, superimposing the weighted digital strings of all the feature vectors to obtain a weighted sequence string; S34. Perform dimensionality reduction processing on the weighted sequence string to obtain the signature value of the document. If a certain digit value in the weighted sequence string is greater than 0, the value of this digit of the signature value of the document is 1. If a certain digit value in the weighted sequence string is less than or equal to 0, the value of this digit of the signature value of the document is 0.

3. A document duplication checking method according to claim 2, characterized in that: Each time a document is received, steps S1 to S3 are sequentially executed on the document, and the signature value of the document outputted in step S3 is stored; The step S4 specifically comprises the following steps: The Hamming distances of the signature values ​​of the two documents are compared. If the Hamming distance is less than a preset value, the two documents are considered to be duplicates, otherwise they are considered not to be duplicates.

4. A document duplication checking terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: S1. Perform word segmentation on the document, take each word segment as a feature vector, and calculate the feature value of each picture as the feature vector of the picture, including: Use HanLP to perform word segmentation on the document and remove common stop words in the preset resource type to obtain the final word segmentation result, and use each word in the word segmentation result as a feature vector; Using a grayscale algorithm to calculate the hash value of each image as a feature vector of the image; S2, taking the TF-IDF value of each word segment relative to the document as the weight value of each word segment, judging whether the weight value of each word segment is greater than the preset maximum text weight value, if so, taking the preset maximum text weight value as the weight value of the word segment; and multiplying the number of occurrences of each picture in the document by the preset basic weight to obtain the weight value of each picture, judging whether the weight value of each picture is greater than the preset maximum picture weight value, if so, taking the preset maximum picture weight value as the weight value of the picture; S3, generating a signature value of the document according to the feature vector and the corresponding weight value; S4. Determine the similarity between the two documents based on their signature values ​​to obtain a duplicate checking result.

5. A document duplication checking terminal according to claim 4, characterized in that: The step S3 specifically comprises the following steps: S31, performing hash calculation on each of the feature vectors to obtain a hash value of each of the feature vectors; S32, adding weight values ​​to each digit value in the hash value of each feature vector to obtain a weighted digital string of each feature vector, if a digit value in the hash value of the feature vector is 1, then the digit value of the weighted digital string of the feature vector is a positive weight value, if a digit value in the hash value of the feature vector is 0, then the digit value of the weighted digital string of the feature vector is a negative weight value; S33, superimposing the weighted digital strings of all the feature vectors to obtain a weighted sequence string; S34. Perform dimensionality reduction processing on the weighted sequence string to obtain the signature value of the document. If a certain digit value in the weighted sequence string is greater than 0, the value of this digit of the signature value of the document is 1. If a certain digit value in the weighted sequence string is less than or equal to 0, the value of this digit of the signature value of the document is 0.

6. A document duplication checking terminal according to claim 5, characterized in that: Each time the processor receives a document, it sequentially executes steps S1 to S3 on the document, and stores the signature value of the document outputted in step S3; The step S4 specifically comprises the following steps: The Hamming distances of the signature values ​​of the two documents are compared. If the Hamming distance is less than a preset value, the two documents are considered to be duplicates, otherwise they are considered not to be duplicates.

Images