A method for judging document similarity based on image, video and text content
By integrating similarity judgment methods of images, videos and text content, the problem of traditional methods being unable to identify duplicate and modified documents is solved, efficient similarity assessment of marketing documents is achieved, and the automated processing capabilities of the knowledge management platform are improved.
Patent Information
- Application Number
- CN202210861048.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-22
- Publication Date
- 2025-09-26
- Estimated Expiration
- 2042-07-22
AI Technical Summary
Existing document similarity judgment methods cannot effectively identify duplicate documents and documents that have been simply modified, especially in marketing documents containing images and videos. Traditional algorithms cannot accurately judge the similarity caused by changes in text position and order, and lack image and video similarity judgment.
A comprehensive judgment method for image, video and text content is adopted, including hash calculation, text segmentation and part-of-speech tagging, TD-IDF value calculation, image hash value comparison and video byte comparison. The document similarity is obtained by combining weighted calculation, and the corpus is used to improve IDF accuracy and reduce the amount of calculation.
It achieves a comprehensive judgment of document similarity, can identify duplicate and simply modified documents, improves the accuracy and efficiency of similarity assessment of marketing documents, and reduces the workload of manual review.
Smart Images
Figure CN115221856B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of platform document management, and in particular relates to a method for judging document similarity based on image, video and text content. Background Art
[0002] Today's global economy is moving toward economic integration and a knowledge-based economy. Networking, virtualization, digitization, and knowledge-based development are becoming key features of modern economic development, making the operating environment facing businesses increasingly complex and volatile. Amidst increasingly fierce market competition, knowledge has become a primary resource for business operations. Competitive advantage increasingly hinges on a company's strong intellectual capital and unique operational capabilities. Consequently, knowledge management is becoming a core management aspect of businesses. Large enterprises and organizations are gradually deploying knowledge management platforms to enable the sharing of both explicit and tacit knowledge. Employees are voluntarily collaborating to share and develop knowledge resources, enabling them to achieve higher goals and generate greater profitability.
[0003] With the launch of a large-scale knowledge management platform and its rapid user growth, some problems have gradually emerged. Among them, a large number of duplicate documents and documents that are simply modified versions of the original files have caused great inconvenience for users to find valuable documents. Each user has to download a large number of documents, browse them one by one, and delete duplicates and documents with high similarity.
[0004] In particular, documents on knowledge management platforms supporting marketing scenarios are mostly customer-facing marketing materials. These documents contain numerous images and videos, as well as extensive headline text. This is especially true for presentation documents, such as PowerPoint. While the text content on a single page remains constant, its placement and size can vary significantly. Traditional text-based document similarity algorithms are not suitable.
[0005] For duplicate documents, hash calculations can be performed, such as using algorithms like MD5, to directly determine similarity based on the hash value. However, the industry has yet to find a perfect technical solution for determining document similarity. One approach, based on string comparison, is relatively accurate for similar documents with minimal text changes, but it is inaccurate for documents with unchanged content but only shifted in position and order. Another approach, based on word frequency statistics, such as using TF-IDF, calculates document similarity, but cannot accurately distinguish between word positions within a document. Furthermore, neither of these algorithms considers the similarity of videos and images within a document.
[0006] Therefore, the platform urgently needs to provide a set of automated methods to assist manual cleaning of duplicate documents and documents that have been simply modified based on the original documents. Summary of the Invention
[0007] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and provide a method for judging the similarity of documents based on images, videos and text content at the same time, automatically identifying duplicate documents from massive documents, and documents that have been simply modified based on the original documents for manual verification and corresponding processing.
[0008] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:
[0009] A method for determining document similarity based on image, video, and text content simultaneously, comprising:
[0010] S1: Select two documents D1 and D2 of the same type and perform hash calculations on them to determine whether D1 and D2 are duplicate documents.
[0011] S2: If D1 and D2 are not duplicate documents, decompress documents D1 and D2 and extract text, images, and videos;
[0012] S3: For documents D1 and D2, perform word segmentation, part-of-speech tagging, and preprocessing on the text to form sets W1 and W2 of nouns and proper nouns respectively;
[0013] S4: Calculate the TD-IDF value for each word in sets W1 and W2, and sort them by TD-IDF value to form lists L1 and L2;
[0014] S5: Select the first N words from L1 and L2 respectively, merge them into set W, calculate the word frequency of each word in set W for each document D1 and D2 respectively, and generate their respective word frequency vectors;
[0015] S6: Calculate the cosine similarity Sim of the word frequency vectors of documents D1 and D2 txt , Sim txt That is the text similarity between documents D1 and D2;
[0016] S7: Perform hash calculations on all images in documents D1 and D2 respectively to obtain hash value lists Hsh1 and Hsh2, and then calculate the image similarity Sim between documents D1 and D2 pic ;
[0017] S8: Calculate the similarity Sim of the video content in documents D1 and D2 using byte comparison video ;
[0018] S9: Set weights w1, w2, and w3 for the text, image, and video similarities of the document respectively, where w1+w2+w3=1. Obtain the document similarity Sim through weighted calculation, compare it with the preset threshold, and draw a conclusion on the document similarity.
[0019] To optimize the above technical solutions, specific measures taken also include:
[0020] The above document types are classified as Word or PowerPoint.
[0021] The above S1 uses an algorithm such as MD5 to perform hash calculation on documents D1 and D2. If the calculated hash values are equal, documents D1 and D2 are duplicate documents.
[0022] The preprocessing described in S3 above includes filtering out all stop words, various adverbs and adjectives, retaining only nouns and proper nouns, and removing duplicates.
[0023] When the above S4 calculates the IDF value, all documents of the same industry or subject in the entire system or platform in the corpus are used.
[0024] The above S7 calculates the image similarity Sim between documents D1 and D2 pic =N hsh / Max(Len1,Len2)
[0025] Among them, N hsh is the number of identical values in Hsh1 and Hsh2, and Len1 and Len2 are the lengths of lists Hsh1 and Hsh2 respectively.
[0026] The above S8 forms the video files in documents D1 and D2 into lists LV1 and LV2, and compares the bytes of each video in LV1 with each video in LV2. If the byte-identical ratio exceeds a preset value T1, the value in list L is incremented by 1, otherwise the value is incremented by 0. Finally, L is a list with values of 1 or 0.
[0027] Calculate the similarity of the video content in documents D1 and D2
[0028] Among them, Len(LV1) and Len(L) are the lengths of lists LV1 and L respectively, and L i The value of the i-th list element in the list L.
[0029] In the above S9, if the document does not contain text, image, or video content, the corresponding similarity is set to 1;
[0030] Calculate the similarity Sim between documents D1 and D2 as Sim txt *w1+Simpic *w2+Sim video *w3;
[0031] Sim is compared with the preset threshold T2. If Sim exceeds T2, documents D1 and D2 are considered similar documents and are transferred to manual review and subsequent processing.
[0032] The present invention has the following beneficial effects:
[0033] The present invention determines the similarity of texts as well as the similarity of images and videos, calculates the similarity of the documents after comprehensive analysis, and prompts manual re-checking for subsequent processing.
[0034] 1. According to the knowledge management platform, the vast majority of documents are marketing documents, which contain a large amount of image information. Therefore, the present invention not only calculates the similarity of text, but also comprehensively considers the similarity of images and videos in the documents, so as to achieve better results in evaluating the similarity of Word and PowerPoint documents.
[0035] 2. The present invention utilizes the characteristics of Word and PowerPoint documents that images can be scaled and cropped without changing the original image files, and determines the similarity of images through a hash function.
[0036] 3. Because only nouns and proper nouns are meaningful in most marketing materials, the present invention performs part-of-speech tagging after word segmentation, filters out all stop words, adverbs, adjectives, etc., retains only nouns and proper nouns, and removes duplicates to form sets W1 and W2 of nouns and proper nouns. TD-IDF calculations are performed only on these words, which significantly reduces the amount of subsequent computations.
[0037] 4. When calculating the TD-IDF value, the present invention uses a corpus of all documents in the same industry or on the same topic on the entire system or platform, rather than all documents on the entire system or platform. This can further improve the accuracy of IDF and reduce the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 This is a flow chart of the method for determining document similarity based on image, video, and text content in the present invention;
[0039] Figure 2 This is a flowchart of text content similarity calculation in the present invention. DETAILED DESCRIPTION
[0040] The embodiments of the present invention are described in further detail below with reference to the accompanying drawings.
[0041] A method for determining document similarity based on image, video, and text content simultaneously, comprising:
[0042] S1: Select two documents D1 and D2 of the same type and perform hash calculations on them to determine whether D1 and D2 are duplicate documents.
[0043] For example, if they are both Word or PowerPoint documents, similarity judgment can be performed.
[0044] Hash the documents D1 and D2 using algorithms such as MD5. If the calculated hash values are equal, then documents D1 and D2 are duplicates.
[0045] S2: If D1 and D2 are not duplicate documents, decompress documents D1 and D2 and extract text, images, and videos;
[0046] If the hash value calculated in S1 is different, the document is decompressed and the text, image, and video are extracted.
[0047] Note: The actual file format of Word and PowerPoint documents in the Microsoft Office suite is zip file format. They can be decompressed using the standard zip decompression algorithm to extract the original text, image and video information.
[0048] S3: Use word segmentation tools to segment, tag, and preprocess documents D1 and D2, respectively, to form sets W1 and W2 of nouns and proper nouns.
[0049] Filter out all stop words, adverbs, and adjectives, retain only nouns and proper nouns, and remove duplicates to form sets W1 and W2 of nouns and proper nouns.
[0050] Because for the vast majority of marketing materials, only nouns and proper nouns have practical meaning, and the amount of subsequent calculations is significantly reduced.
[0051] S4: Calculate the TD-IDF value for each word in sets W1 and W2, and sort them by TD-IDF value to form lists L1 and L2;
[0052] When calculating the IDF value, all documents in the same industry or on the same topic in the entire system or platform in the corpus are used, rather than all documents in the entire system or platform. This can further improve the accuracy of the IDF.
[0053]
[0054] TD-IDF=TF*IDF
[0055] S5: Select the first N words from L1 and L2 respectively, merge them into a set W of length M, calculate the word frequency of each word in set W for each document D1 and D2 respectively, and generate their respective word frequency vectors;
[0056] Note: If there is no repetition of words, then M = 2*N.
[0057] S6: Calculate the cosine similarity Sim of the word frequency vectors of documents D1 and D2 txt , Sim txt That is the text similarity between documents D1 and D2;
[0058] Sim txt The larger the value, the more similar documents D1 and D2 are;
[0059] S7: Perform hash calculations on all images in documents D1 and D2 respectively. The algorithm can be MD5 or other algorithms to obtain hash value lists Hsh1 and Hsh2, with lengths of Len1 and Len2 respectively. Then calculate the image similarity Sim between documents D1 and D2. pic ;
[0060] In Office documents, when scaling or cropping an image, the original image is retained, with only information such as size and position digitally identified. Therefore, in different documents, the same hash value will be calculated when scaling or cropping the same image.
[0061] Determine the number of identical values N in Hsh1 and Hsh2 hsh , calculate the image similarity Sim between documents D1 and D2 pic =N hsh / Max(Len1,Len2)
[0062] S8: Calculate the similarity Sim of the video content in documents D1 and D2 using byte comparison video ;
[0063] Since videos may be edited, a video with mostly the same content but only one frame different will result in a significant change in the hash value. Therefore, for video content, byte comparison is needed to determine similarity.
[0064] Set a threshold T. If the same content of two videos exceeds T1, the two videos are considered similar. The default threshold can be set to 70%.
[0065] The video files in documents D1 and D2 are formed into lists LV1 and LV2. Each video in LV1 is byte-compared with the video in LV2. If the proportion of identical content exceeds T1, the value in list L is increased by 1, otherwise the value is increased by 0. Finally, L is a list with values of 1 or 0, for example [1,0,0,1,1,…0]. The similarity of the video content in documents D1 and D2 is
[0066] S9: Set weights w1, w2, and w3 for the text, image, and video similarities of the document respectively, where w1+w2+w3=1. Obtain the document similarity Sim through weighted calculation, compare it with the preset threshold, and draw a conclusion on the document similarity.
[0067] If there is no corresponding content, the corresponding similarity is set to 1.
[0068] For example, if neither document contains a video, then Sim video The value is 1; if both documents do not contain text, then Sim txt The value is 1; if there is no image in the two documents, Sim pic The value is 1.
[0069] After comparison, the similarity Sim between documents D1 and D2 is Sim txt *w1+Sim pic *w2+Sim video *w3. Set a threshold T2. If the document similarity exceeds T2, the two documents are considered similar and transferred to manual review and subsequent processing.
[0070] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A method for determining document similarity based on image, video, and text content, characterized in that: include: S1: Select two documents D1 and D2 of the same type and perform hash calculations on them to determine whether D1 and D2 are duplicate documents. S2: If D1 and D2 are not duplicate documents, decompress documents D1 and D2 and extract text, images, and videos; S3: For documents D1 and D2, perform word segmentation, part-of-speech tagging, and preprocessing on the text to form sets W1 and W2 of nouns and proper nouns respectively; S4: Calculate the TD-IDF value for each word in sets W1 and W2, and sort them by TD-IDF value to form lists L1 and L2; S5: Select the first N words from L1 and L2 respectively, merge them into set W, calculate the word frequency of each word in set W for each document D1 and D2 respectively, and generate their respective word frequency vectors; S6: Calculate the cosine similarity Sim of the word frequency vectors of documents D1 and D2 txt , Sim txt That is the text similarity between documents D1 and D2; S7: Perform hash calculations on all images in documents D1 and D2 respectively to obtain hash value lists Hsh1 and Hsh2, and then calculate the image similarity Sim between documents D1 and D2 pic ; S8: Calculate the similarity Sim of the video content in documents D1 and D2 using byte comparison video ; S9: Set weights w1, w2, and w3 for the text, image, and video similarities of the document, respectively, where w1+w2+w3=1. Obtain the document similarity Sim through weighted calculation, compare it with the preset threshold, and draw a conclusion on document similarity. Among them, S7 calculates the image similarity Sim between documents D1 and D2 pic =N hsh / Max(Len1,Len2); Among them, N hsh is the number of identical values in Hsh1 and Hsh2, Len1 and Len2 are the lengths of lists Hsh1 and Hsh2 respectively; S8 forms lists LV1 and LV2 for the video files in documents D1 and D2, and compares the bytes of each video in LV1 with those in LV2. If the proportion of identical bytes exceeds a preset value T1, the value in list L is incremented by 1; otherwise, the value is incremented by 0. Finally, L is a list with a value of 1 or 0. Calculate the similarity of the video content in documents D1 and D2 Among them, Len(LV1) and Len(L) are the lengths of lists LV1 and L respectively, and L i The value of the i-th list element in the list L.
2. The method for determining document similarity based on image, video, and text content according to claim 1, characterized in that: The document type is Word or PowerPoint.
3. The method for determining document similarity based on image, video, and text content according to claim 1, wherein: S1 uses the MD5 algorithm to calculate the hash of documents D1 and D2. If the calculated hash values are equal, documents D1 and D2 are duplicate documents.
4. The method for determining document similarity based on image, video, and text content according to claim 1, wherein: The preprocessing described in S3 includes filtering out all stop words, various adverbs and adjectives, retaining only nouns and proper nouns, and removing duplicates.
5. The method for determining document similarity based on image, video, and text content according to claim 1, wherein: When S4 calculates the IDF value, it uses all documents in the corpus that are in the same industry or on the same topic across the entire system or platform.
6. The method for determining document similarity based on image, video, and text content according to claim 1, wherein: In S9, if the document does not contain text, image, or video content, the corresponding similarity is set to 1; Calculate the similarity Sim between documents D1 and D2 as Sim txt *w1+Sim pic *w2+Sim video *w3; Sim is compared with the preset threshold T2. If Sim exceeds T2, documents D1 and D2 are considered similar documents and are transferred to manual review and subsequent processing.
Citation Information
Patent Citations
Feature word extraction method and apparatus, text similarity calculation method and apparatus, and device
WO2021072850A1