A multi-document comparison method combining similarity calculation and exact matching

Abstract

The application discloses a multi-document comparison method combining similarity calculation and accurate matching, and belongs to the technical field of information processing, and comprises the following steps: S1, text preprocessing, S2, block division and positioning, S3, vector embedding, S4, similarity matrix construction, S5, Top-K pair extraction, S6, similarity classification, S7, accurate matching, S8, visual highlighting, and S9, similarity evaluation. Through the sentence-level semantic vector and the Top-K matching mechanism, the influence of document structure changes is effectively overcome, the detection accuracy and robustness of content reconstruction plagiarism are significantly improved, the matching misplacement caused by repeated content in long documents is avoided, the stability of comparison is ensured, the problem of semantic dilution of full-text vectors is solved by fine-grained sentence embedding, the recognition sensitivity to sentence rewriting and local plagiarism is enhanced, meanwhile, through the combination of double-level quantitative indexes and visual highlighting positioning, abstract similarity is converted into traceable evidence, and the result explainability and practical value are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, specifically to a multi-document comparison method that combines similarity calculation and precise matching. Background Technology

[0002] In current technical practices, multi-document comparison methods can be mainly divided into two categories: string matching-based methods and semantic similarity methods based on vector representations. String matching-based methods, such as Levenshtein distance (edit distance), difflib.SequenceMatcher in the Python standard library, and Google's diff_match_patch algorithm, primarily rely on direct surface comparison of characters or words, measuring text differences by calculating operations such as insertion, deletion, and replacement. While these methods have advantages in computational accuracy and simplicity of implementation, they do not understand the semantic context of the text, thus exhibiting significant limitations when processing long documents. Firstly, character-level methods treat documents as a continuous stream of characters, failing to perceive structural levels such as paragraphs or chapters. When document content is identical except for a change in paragraph order, this method may misjudge a significant difference, severely impacting the accuracy of plagiarism detection. Secondly, when there are multiple identical or similar segments in a document, character-level comparison is prone to "matching misalignment," meaning that the algorithm may mistakenly match the paragraph that should correspond to another repeated position. The resulting "accumulated error propagation" will cause all subsequent comparison paths to shift, resulting in a complete distortion of the results.

[0003] To overcome the semantic gaps inherent in character-level methods, vector-based semantic similarity methods have emerged. These methods map words, sentences, or entire texts into numerical vectors using embedding models (such as Doc2Vec and BERT), and then assess semantic closeness using metrics like cosine similarity within the vector space. While this approach has advantages in understanding semantics, it also faces challenges in practical applications. First, encoding an entire long document into a fixed-length vector leads to "semantic dilution and expression averaging." Locally repeated content (such as a plagiarized sentence or paragraph) is diluted by other non-critical content in the overall text, making it difficult to occupy a significant weight in the overall vector and thus failing to be effectively identified. Second, these methods typically only output an overall similarity score (e.g., 0.82), which is "unstable and lacks interpretability." It cannot tell the user where the specific similar content is located in the original text, making it difficult to distinguish whether the overall similarity is due to thematic similarity or local plagiarism. This makes the results impractical in formal plagiarism detection applications that require precise location and source tracing.

[0004] In summary, existing text comparison techniques each have their inherent limitations. Some lack semantic understanding and are sensitive to structural changes and repetitive content, while others fail to meet the demands of refined plagiarism detection due to their coarse granularity and uninterpretable results. Therefore, there is an urgent need in this field for a multi-document comparison method that can balance semantic understanding and accurate matching, while possessing strong robustness and high interpretability. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-document comparison method that combines similarity calculation and precise matching. This method achieves sentence-level semantic vector modeling and a Top-K full-text matching mechanism, effectively overcoming the sensitivity of traditional methods to changes in document structure. Even with paragraph order adjustments, it can accurately identify similar content, significantly improving the accuracy and robustness of plagiarism detection systems against content reconstruction-based plagiarism. Employing a collaborative strategy of "semantic filtering first, then precise location," it effectively avoids matching misalignment issues caused by multiple occurrences of repeated content in long documents, ensuring the stability of the comparison path and the reliability of the results. Through fine-grained sentence-by-sentence embedding and sentence pair analysis, it solves the semantic dilution problem in full-text vector encoding, significantly enhancing the sensitivity to identifying pseudo-original behaviors such as sentence rewriting and partial plagiarism. Simultaneously, the method provides dual-level quantitative indicators (local and overall), and combines precise annotation of similar sentence pairs with visual highlighting of repeated segments, transforming abstract similarity into intuitive and traceable visual evidence, greatly improving the interpretability of the results and their practical value in real-world scenarios such as review and education.

[0006] To achieve the above effects, the present invention provides the following technical solution: a multi-document comparison method combining similarity calculation and precise matching, comprising the following steps: S1. Text Preprocessing: Supports multiple document input formats, including PDF, DOCX, and TXT, and converts documents into plain text. PDF documents are converted to plain text using OCR methods, and DOCX documents are converted into document objects using the python-docx library and plain text is extracted, while simultaneously recording the original sentence's position information within the document.

[0007] S2. Segment the text by sentence and record the original sentence position information: Segment the text by sentence level and record the position information of each sentence in the original text.

[0008] S3. Perform embedding operations on each block: Use an embedding model to convert each sentence block into a vector.

[0009] S4. Calculate the cosine similarity of each block vector in the two comparison documents pairwise, generating an M×N similarity matrix. Document A is divided into M blocks, and document B is divided into N blocks. Each element in the similarity matrix represents the cosine similarity between the i-th block in document A and the j-th block in document B.

[0010] S5. Sort the similarity values ​​of each row from high to low, and retain the top k pairs with sentences in document B. For each row of the similarity matrix, extract all similarity values ​​in that row, sort them from high to low, retain the top k maximum values, and record the position information of the corresponding sentences in document B to form a Top-k pairing list.

[0011] S6. Using a Top-k list, sentences are categorized into two types based on similarity: identical and highly similar. Sentences with a similarity greater than 0.95 and identical are classified as "identical," while sentences with a similarity greater than 0.8 but not identical are classified as "highly similar."

[0012] S7. For "highly similar" pairs, further precise matching is performed: the SequenceMatcher and diff_match_patch algorithms are used to identify identical sub-segments in "highly similar" sentence pairs and record the precise position index of these sub-segments in the original sentences.

[0013] S8. Based on the document input type, visually highlight "identical segments" and generate jump anchor marks. For DOCX documents, use the python-docx library to load the document and highlight identical segments, while generating jump anchor marks. For PDF or plain text, use HTML... The tags highlight identical fragments and generate jump anchors with the data-jump-id attribute.

[0014] S9. Calculate the two-level similarity evaluation index: including local similarity and overall similarity. Local similarity is calculated based on the Top-k maximum similarity value of each sentence in the sentence pair similarity matrix, while overall similarity is calculated based on the ratio of the total number of characters in all identical segments to the total character length of the entire document or the shorter of the two documents.

[0015] Furthermore, the method includes the following steps: According to the operation steps in S1, the recorded position information of the original sentence in the document includes the index of the sentence or the paragraph number in the document.

[0016] Furthermore, the method includes the following steps: according to the operation steps in S3, the embedding model is the Sentence-BERT model.

[0017] Furthermore, the method includes the following steps: according to the operation steps in S4, the cosine similarity is obtained by calculating the cosine value of the two block vectors, with a value range of 0 to 1.

[0018] Furthermore, the process includes the following steps: According to the operation steps in S5, the Top-k pairing list includes sentence numbers in document A, sentence numbers in document B, similarity scores, and original sentence position information.

[0019] Furthermore, the method includes the following steps: According to the operation steps in S6, the similarity classification is achieved through threshold judgment: First, check whether the similarity is greater than 0.95. If it is, further check whether they are completely identical. If they are completely identical, classify them as "completely identical"; otherwise, classify them as "highly similar". If the similarity is not greater than 0.95, check whether it is greater than 0.8. If it is, classify them as "highly similar".

[0020] Furthermore, the process includes the following steps: According to the operation steps in S7, the precise matching includes using the SequenceMatcher and diff_match_patch algorithms to identify completely identical sub-fragments respectively, and merging the search results.

[0021] Furthermore, the method includes the following steps: according to the operation steps in S8, the jump anchor mark has a unique identifier in the format data-jump-id="start_end", where start and end represent the start and end position indices of the sub-segment.

[0022] Furthermore, the method includes the following steps: according to the operation steps in S9, the local similarity is used to evaluate the degree of local alignment of documents at the semantic level, and the overall similarity is used to quantify the actual degree of overlap between documents at the content level.

[0023] Furthermore, the method includes the following steps: It addresses the misjudgment problem caused by paragraph swapping through sentence-level vector semantic modeling and a Top-k full-text sentence pair matching mechanism. It solves the matching misalignment and error propagation problems caused by multiple repetitions by applying character-level algorithms for local comparison in highly similar sentence pairs. It addresses semantic dilution and expression averaging problems through sentence-by-sentence encoding and sentence comparison matrices. It addresses the instability and lack of interpretability of similarity scores through similar sentence pair annotation, highlighting of repeated segments, and jump location.

[0024] This invention provides a multi-document comparison method that combines similarity calculation and precise matching, which has the following beneficial effects: (1) The multi-document comparison method combining similarity calculation and precise matching provided by this invention effectively overcomes the sensitivity of traditional character-level comparison methods to changes in document structure by introducing sentence-level semantic vector modeling and full-text Top-K matching mechanism. Even if there are structural rewritings such as paragraph order adjustment and chapter reorganization between the comparison documents, this method can accurately identify and associate semantically similar content in different positions, significantly reduce the false judgment rate, and greatly improve the robustness and accuracy of the plagiarism detection system when processing plagiarized documents that have undergone content reconstruction.

[0025] (2) The present invention adopts a collaborative strategy of "semantic screening first, then precise positioning". First, semantic vectors are used to screen out candidate sentence pairs with high similarity from a global perspective. Then, a computationally intensive character-level precise matching algorithm is applied only within these candidate ranges. This strategy fundamentally avoids the matching misalignment problem caused by repeated content in long documents in traditional character-level comparison, effectively curbs the spread of incorrect matching, and ensures the stability and reliability of the comparison results. It is particularly suitable for the fine comparison of long texts such as academic papers and technical reports.

[0026] (3) By adopting sentence embedding and fine-grained analysis of sentence pair similarity matrix, this invention successfully solves the semantic dilution and expression averaging problems in traditional full-text vector coding methods. This method can keenly capture semantic overlap and local content duplication behavior at the sentence level. It has excellent detection capabilities for pseudo-original methods such as deliberate sentence rewriting and copying of sentences within paragraphs, thereby improving the sensitivity and accuracy of content similarity recognition at the micro level.

[0027] (4) This invention significantly enhances the interpretability and practical value of plagiarism detection results. It not only provides quantitative similarity indicators in both overall and local dimensions, but also transforms abstract similarity scores into intuitive and traceable visual evidence through functions such as precise annotation of similar sentences, highlighting of repeated fragments in the original text, and interactive jump positioning. This enables users to quickly locate and verify duplicate content, greatly facilitating manual review, responsibility determination, and report generation, and improving the credibility and operability of the system output results in actual scenarios such as review, arbitration, and education. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a multi-document comparison method that combines similarity calculation and precise matching according to the present invention. Detailed Implementation

[0029] This invention provides a technical solution: Example 1, please refer to Figure 1 , This embodiment details the implementation process of a multi-document comparison method that combines similarity calculation and exact matching. System preparation and environment configuration: This method is based on the Python programming language. The system requires the following key libraries to be installed: sentence-transformers (for sentence vector embedding), python-docx (for processing Word documents), PyMuPDF or pdf2image combined with pytesseract (for PDF OCR conversion), difflib (built-in, providing SequenceMatcher), and diff-match-patch (requires the installation of the Python package of the same name). The preferred embedding model is the pre-trained all-MiniLM-L6-v2 model, which has a good balance between efficiency and performance.

[0030] A multi-document comparison method combining similarity calculation and exact matching includes the following steps: Step 1: Text Preprocessing The system receives documents uploaded by users. If the input is in DOCX format, it uses the Document object from the python-docx library to read the file, traverses all paragraphs in the document, and extracts the text. During the extraction process, it records the original position information of each sentence, including its paragraph index, the start position of the character in the paragraph, and the end position of the character.

[0031] If the input is in PDF format, the PyMuPDF library is first used to try to extract the plain text. If the extraction fails or the PDF is a scanned image, pdf2image is called to convert the page into an image, and then pytesseract is used for OCR recognition. Finally, the plain text is output. The position information of each recognized sentence in the original page and the approximate area is also recorded. The TXT file is read directly and initially divided by line or specific delimiter.

[0032] Step 2: Divide the text into blocks by sentence and record the position. The preprocessed plain text is precisely segmented into independent sentences using natural language processing tools (such as NLTK or SpaCy's sentence segmenter). Each sentence is assigned a unique ID and bound to the original position information recorded in step one (such as "segment X, character offset YZ") to form a list of sentences with position labels.

[0033] Step 3: Perform embedding operations on each block. Load the all-MiniLM-L6-v2 model from the sentence-transformers library, input the list of sentences obtained in step two into the model, the model encodes each sentence into a 384-dimensional floating-point vector (the default output dimension of this model), and outputs a set of sentence vectors for two documents: A_embeddings (Mx384) and B_embeddings (Nx384).

[0034] Step 4: Generate a similarity matrix Calculate the cosine similarity between each sentence vector in document A and each sentence vector in document B. The cosine similarity is calculated by dividing the dot product of the two vectors by the product of their magnitudes. Finally, generate a similarity matrix S with M rows (number of sentences in document A) and N columns (number of sentences in document B), where S[i][j] represents the semantic similarity between the i-th sentence in document A and the j-th sentence in document B, with a value range of [0,1].

[0035] Step 5: Extract Top-k pairs Iterate through each row of the similarity matrix S (i.e., each sentence of document A). For the i-th row, sort all N similarity values ​​in the row in descending order. Set k=3 (which can be adjusted as needed). Select the top 3 highest similarity values ​​and record the sentence indices j1, j2, j3 of document B corresponding to these values. Add these 3 pairs (i, j1, sim_score1), (i, j2, sim_score2), and (i, j3, sim_score3) to the Top-k pairing list. This list contains the 3 most likely similar candidates for each sentence of document A in document B.

[0036] Step Six: Similarity Classification For each pair in the Top-k pairing list, first determine if its similarity score is greater than 0.95. If it is, then use string exact comparison (string_a == string_b) to determine if the two sentences are completely identical. If they are completely identical, then mark the pair as "completely identical". If they are not completely identical, but the similarity score is greater than 0.8, then mark it as "highly similar". If the similarity score is not greater than 0.95, but greater than 0.8, then mark it directly as "highly similar". Pairs with a similarity score less than or equal to 0.8 will not be processed further in this embodiment.

[0037] Step 7: Perform exact matching on "highly similar" pairs. For each sentence pair classified as "highly similar", character-level comparisons are performed using difflib.SequenceMatcher and diff_match_patch.diff_match_patch, respectively.

[0038] SequenceMatcher uses the get_matching_blocks() method to retrieve the blocks that are completely matched between two sentences.

[0039] diff_match_patch calculates the difference using diff_main(), and then filters out the segments with the operation type "equal" from the difference results.

[0040] Merge the lists of perfectly matching segments obtained from the two algorithms, remove overlapping parts, and record the exact start and end indices of these perfectly matching substrings in the original sentences of document A and document B.

[0041] Step 8: Visualizing Highlights and Generating Anchor Points For DOCX documents: Reopen the original DOCX file using python-docx. Based on the location information recorded in step two, locate the sentence or sub-segment that needs to be highlighted. For "identical" entire sentences or "identical sub-segments" extracted in step seven, use the Run property of python-docx to set its font background color to yellow (e.g., RGBColor(255,255,0)) to achieve highlighting. At the same time, generate a unique anchor mark for the highlighted segment in a specific area of ​​the document (such as a footnote or comment), in the format <Jump to document X, segment [start-end]>. For PDF / plain text: Render the document content into HTML format, iterate through all the sections that need to be highlighted, and use HTML... Wrap the fragment in a tag and set the CSS style to style="background-color: yellow". Also, in... Add a custom attribute `data-jump-id` to the tag. Its value is a unique identifier for the segment, in the format "A_start_end_B_start_end", for example, `data-jump-id="A_150_160_B_45_55"`. By listening to the click events of these anchors using JavaScript, interactive jump functionality can be achieved.

[0042] Step 9: Calculate the two-level similarity evaluation index Local similarity: From the Top-k pairing list, extract the highest similarity value (i.e., the maximum value per row) between each sentence of document A and a sentence of document B, and then calculate the average of all these highest values ​​as the local semantic similarity of document A relative to document B.

[0043] Overall similarity: Calculate the total number of characters in all "completely identical sub-fragments" obtained in step seven (total_matched_chars), calculate the total number of characters in document A and document B, and take the smaller value (min_total_chars). Overall similarity = total_matched_chars / min_total_chars.

[0044] The final report presents both metrics, quantifying document similarity from the semantic and character content levels, respectively.

[0045] Technical effectiveness verification: Through the above embodiments, this method effectively solves the four major defects mentioned in the background art: When faced with documents with paragraph swapping, sentence-level vector matching successfully identified similar sentences in different positions without misjudging due to structural changes.

[0046] By employing a strategy of semantic filtering followed by precise local matching, mismatches and error propagation caused by repetition in multiple points within long documents are effectively avoided.

[0047] By using sentence-by-sentence encoding instead of full-text encoding, local sentence plagiarism that was ignored by the full-text vector method was successfully detected, thus overcoming the semantic dilution problem.

[0048] The generated visually highlighted report and two-level similarity index make the source of similarity immediately clear, greatly enhancing the interpretability and practicality of the results.

[0049] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-document comparison method that combines similarity calculation and exact matching, characterized in that, Includes the following steps: S1. Text preprocessing: Supports multiple document input formats, including PDF, DOCX and TXT, and converts documents into plain text. PDF documents are converted into plain text using OCR, and DOCX documents are converted into document objects and plain text is extracted using the python-docx library. At the same time, the position information of the original sentences in the document is recorded. S2. Segment the text by sentence and record the original sentence position information: Segment the text by sentence level and record the position information of each sentence in the original text; S3. Perform embedding operations on each block: Use an embedding model to convert each sentence block into a vector; S4. Calculate the cosine similarity of each block vector in the two comparison documents pairwise to generate an M×N similarity matrix, where document A is divided into M blocks and document B is divided into N blocks. Each element in the similarity matrix represents the cosine similarity between the i-th block in document A and the j-th block in document B. S5. Sort the similarity values ​​of each row from high to low, and retain the top k pairing results with sentences in document B. For each row of the similarity matrix, extract all similarity values ​​of that row, sort them from high to low, retain the top k maximum values, and record the position information of the corresponding sentences in document B to form a Top-k pairing list. S6. Using the Top-k list, sentences are divided into two categories based on similarity: completely identical and highly similar. Sentences with a similarity greater than 0.95 and completely identical are classified as "completely identical", while sentences with a similarity greater than 0.8 but not completely identical are classified as "highly similar". S7. For "highly similar" pairings, further precise matching is performed: the SequenceMatcher and diff_match_patch algorithms are used to identify identical sub-segments in "highly similar" sentence pairs and record the precise position index of these sub-segments in the original sentences; S8. Based on the document input type, visually highlight "identical fragments" and generate jump anchor marks. For DOCX documents, use the python-docx library to load the document and highlight identical fragments while generating jump anchor marks. For PDF or plain text, use HTML... Tags highlight identical segments and generate jump anchors with the data-jump-id attribute; S9. Calculate the two-level similarity evaluation index: including local similarity and overall similarity. Local similarity is calculated based on the Top-k maximum similarity value of each sentence in the sentence pair similarity matrix. Overall similarity is calculated based on the ratio of the total number of characters of all identical segments to the total character length of the entire document or the shorter of the two documents.

2. The multi-document comparison method combining similarity calculation and precise matching according to claim 1, characterized in that, The process includes the following steps: According to the operation steps in S1, the recorded position information of the original sentence in the document includes the index of the sentence or the paragraph number in the document.

3. The multi-document comparison method combining similarity calculation and precise matching according to claim 1, characterized in that, Includes the following steps: According to the operation steps in S3, the embedding model is the Sentence-BERT model.

4. The multi-document comparison method combining similarity calculation and precise matching according to claim 1, characterized in that, Includes the following steps: According to the operation steps in S4, the cosine similarity is obtained by calculating the cosine value of the two block vectors, with a value range of 0 to 1.

5. The multi-document comparison method combining similarity calculation and precise matching according to claim 1, characterized in that, The process includes the following steps: According to the operation steps in S5, the Top-k pairing list includes the sentence number in document A, the sentence number in document B, the similarity score, and the original sentence position information.

6. The multi-document comparison method combining similarity calculation and precise matching according to claim 1, characterized in that, Includes the following steps: According to the operation steps in S6, the similarity classification is achieved by threshold judgment: first, check whether the similarity is greater than 0.

95. If it is, further check whether they are completely the same. If they are completely the same, they are classified as "completely the same" class. Otherwise, they are classified as "highly similar" class. If the similarity is not greater than 0.95, check whether it is greater than 0.

8. If it is, they are classified as "highly similar" class.

7. The multi-document comparison method combining similarity calculation and precise matching according to claim 1, characterized in that, Includes the following steps: According to the operation steps in S7, the precise matching includes using the SequenceMatcher and diff_match_patch algorithms to identify completely identical sub-fragments respectively, and merging the search results.

8. The multi-document comparison method combining similarity calculation and precise matching according to claim 1, characterized in that, Includes the following steps: According to the operation steps in S8, the jump anchor mark has a unique identifier in the format data-jump-id="start_end", where start and end represent the start and end position indices of the sub-segment.

9. A multi-document comparison method combining similarity calculation and precise matching according to claim 1, characterized in that, Includes the following steps: According to the operation steps in S9, the local similarity is used to evaluate the degree of local alignment of documents at the semantic level, and the overall similarity is used to quantify the actual degree of overlap between documents at the content level.

10. A multi-document comparison method combining similarity calculation and precise matching according to claim 1, characterized in that, The method includes the following steps: It solves the misjudgment problem caused by paragraph swapping by using sentence-level vector semantic modeling and Top-k full-text sentence pair matching mechanism; it solves the matching misalignment and error propagation problem caused by multiple repetitions by applying character-level algorithms for local comparison in highly similar sentence pairs; it solves the semantic dilution and expression averaging problem by using sentence-by-sentence encoding and sentence comparison matrix; and it solves the problem of unstable similarity scores and lack of interpretability by using similar sentence pair annotation, repeated segment highlighting and jump positioning.

Images