A literature duplicate checking method based on vector retrieval and large model semantic analysis

Abstract

The present application belongs to the technical field of literature duplication checking, and particularly relates to a literature duplication checking method based on vector retrieval and large model semantic analysis, comprising the following steps: step 1, intelligently splitting all the literature in the reference literature library and the user-uploaded literature according to their internal semantic structures, and converting them into vector representations understandable by computers; step 2, performing similarity retrieval on the text fragment vectors of the user-uploaded literature fragment library in the vector index of the reference literature fragment library, calculating the fragment similarity, recalling similar fragments, and obtaining the similar fragment pair set of each text fragment vector in the user-uploaded literature fragment library; and step 3, calculating the single literature duplication rate, literature similarity and other core duplication checking indexes of the user-uploaded literature according to the fragment similarity and the similar fragment pair set, and obtaining similar literature. The method fundamentally solves the problems of semantic fragmentation and shallow matching, and provides more accurate, more in-depth and more operational literature content correlation analysis services for users.

Description

Technical Field

[0001] This invention belongs to the field of document plagiarism detection technology, specifically involving a document plagiarism detection method based on vector retrieval and large-scale model semantic analysis. Background Technology

[0002] In the current field of document management and knowledge services, users generally face a core pain point when conducting document research, policy studies, or writing reports: how to quickly and accurately determine the semantic overlap and relevance between the content of a newly written document or report and the materials in the existing massive document database.

[0003] Current mainstream solutions typically employ "fixed-length fingerprint comparison" technology. Its basic principle is to mechanically segment the document to be analyzed and all documents in the reference database according to a fixed character length (e.g., 13 consecutive Chinese or English characters), generating a series of "text fingerprints." The system then determines the degree of similarity by calculating the amount of overlap between these fingerprints in the two documents. The advantages of this method are high computational efficiency and simplicity of implementation.

[0004] However, this approach has significant limitations: Semantic integrity deficiency: Fixed-length segmentation completely ignores the natural semantic structure and logical paragraphs of the text. A complete sentence or sense group may be cut in the middle, resulting in the lack of semantics in the comparison unit itself, thus affecting the accuracy and interpretability of the comparison results.

[0005] Coarse-grained plagiarism detection: Existing technologies mostly compare the entire document or a fixed number of words in a "sliding window," which cannot accurately locate semantically complete chapters or paragraphs, resulting in inaccurate plagiarism detection results and a tendency to make false or false judgments.

[0006] Weak semantic understanding: Traditional plagiarism detection systems mainly rely on keyword matching or simple text similarity calculations, lacking a deep understanding of the text's semantics. This method is essentially based on literal matching, and its ability to identify content that has been semantically rewritten through synonym replacement, word order adjustment, sentence reconstruction, etc., is very limited, making it unable to effectively identify rewritten and reorganized plagiarized content.

[0007] Poor interpretability of results: The final output is usually only a general similarity percentage, failing to provide users with specific sources, locations, and semantic differences of similar content, making it difficult to support in-depth content review and optimization. Existing systems typically only provide an overall repetition rate percentage, failing to clearly show the specific sources, locations, and degrees of similarity of the repeated content, making it difficult for users to understand and review.

[0008] Lack of in-depth analysis: Existing technologies cannot perform intelligent semantic comparison of highly similar text fragments, point out specific modification points and differences, and provide users with valuable modification suggestions. Summary of the Invention

[0009] The purpose of this invention is to provide a document plagiarism detection method based on vector retrieval and large-scale model semantic analysis. This method abandons the traditional path of "mechanical segmentation + literal matching" and instead adopts a new technical route of "intelligent semantic unit splitting + deep vector semantic retrieval + large-scale model semantic comparison analysis". It aims to fundamentally solve the problems of semantic fragmentation and shallow matching, and provide users with more accurate, in-depth and more operational document content correlation analysis services.

[0010] Technical solution to achieve the purpose of this invention: A document plagiarism detection method based on vector retrieval and large-scale model semantic analysis, the method comprising: Step 1: Intelligently split all documents uploaded by users and in the reference database according to their inherent semantic structure, and convert them into vector representations that can be understood by computers; Step 2: Perform a similarity search on the text fragment vectors of the user-uploaded literature fragment library in the vector index of the reference fragment library, calculate the fragment similarity, recall similar fragments, and obtain a set of similar fragment pairs for each text fragment vector in the uploaded literature fragment library; Step 3: Based on the fragment similarity and the set of similar fragment pairs, calculate the core plagiarism detection indicators such as the duplication rate and document similarity of the single document uploaded by the user, and obtain similar documents.

[0011] Further, step 1 includes: Step 1.1: Perform semantic unit intelligent segmentation on all documents to obtain plain text fragments of each document's segmentation. Step 1.2: Convert the plain text fragment into a text vector to obtain the text fragment vector.

[0012] Further, step 2 includes: using cosine similarity to calculate the similarity between each text fragment vector in the user-uploaded document fragment library and each text fragment vector in each document in the reference fragment library; setting a similarity threshold to recall text fragment vectors in the reference fragment library with similarity higher than the similarity threshold as reference fragments; and selecting the top 5 reference fragments with the highest similarity for each text fragment vector in the user-uploaded document fragment library to form a preliminary set of similar fragment pairs and construct a similarity record.

[0013] Furthermore, the similarity record includes: uploaded fragment text, uploaded fragment word count, uploaded document ID, reference fragment text, reference ID, and similarity score.

[0014] Further, step 3 includes: Step 3.1: Calculate the duplication rate of the single document uploaded by the user based on the fragment similarity; Step 3.2: Calculate the document similarity of the set based on similar segments, and sort the references to obtain similar documents.

[0015] Furthermore, the formula for calculating the duplication rate of a single document in step 3.1 is as follows: CurCore = / ; In the formula, CurCore represents the duplication rate of a single document; MaxScore(i) represents the highest similarity score of the i-th text segment vector in a single document uploaded by a user; WordNum(i) represents the number of characters in the i-th text segment vector of a single document uploaded by the user; i represents each segment in a single document uploaded by the user; M represents the total number of segments that a single document uploaded by a user is split into.

[0016] Furthermore, in step 3.1, if the i-th text segment vector of a single document uploaded by the user does not recall any reference segments, then MaxScore(i) is assigned the value 0.

[0017] Furthermore, step 3.2, calculating document similarity, specifically involves: Based on the set of all similar fragment pairs obtained in step 2, group them by reference ID; for each unique reference ID, extract the set of all similar fragment pairs with all fragments of the user-uploaded single document, as the similarity record for that specific reference; according to the similarity record for that specific reference, for each fragment of the user-uploaded single document, find the reference fragment with the highest similarity to that fragment in that specific reference, and record the highest similarity score and its text word count; aggregate the highest similarity scores and their text word counts corresponding to all fragments of the user-uploaded single document using the same calculation formula as in step 3.1, and calculate the similarity RefCore between the user-uploaded single document and that specific reference.

[0018] Furthermore, step 3.2, obtaining similar documents, specifically involves: Sort all RefCores corresponding to all reference IDs in descending order, take the top 5, obtain the 5 references most similar to the user-uploaded single document and their respective similarity scores, and take the 5 references most similar to the user-uploaded single document as similar documents.

[0019] Furthermore, the method also includes: Step 4, performing semantic comparison analysis using a large language model for similar documents: For the similar documents found in Step 3.2, for any segment in a single document uploaded by the user, select the segment most similar to it from the similar documents, call the large language model, take these two highly similar segments as input, instruct the model to analyze the similarities and differences between the two in terms of semantics, structure, word choice, etc., and generate a "modification suggestion" report, pointing out where the original text may have been modified, providing users with in-depth and actionable feedback.

[0020] The beneficial technical effects of this invention are as follows: 1. This invention provides a document plagiarism detection method based on vector retrieval and large-scale model semantic analysis. This method, based on intelligent document structure segmentation, differs from fixed-word segmentation. It utilizes the heading hierarchy and numbering system of Word documents to divide the document into semantically complete chapters and paragraphs as the basic units for plagiarism detection. By intelligently segmenting the document into chapters, the semantic integrity of the plagiarism detection units is ensured, thereby improving the accuracy and reliability of plagiarism detection, significantly increasing the precision and recall rate of document plagiarism detection, and reducing false positives and false negatives. Through semantic unit segmentation, it avoids the false positives caused by semantic fragmentation in traditional methods, resulting in more accurate plagiarism detection results.

[0021] 2. The present invention provides a document plagiarism detection method based on vector retrieval and large-scale model semantic analysis. It introduces deep semantic understanding capabilities, uses advanced vector retrieval technology to capture the semantic similarity of texts, and combines a large-scale model to conduct deep semantic comparison analysis on highly similar segments. It can not only identify literal repetitions, but also effectively identify semantic plagiarism and rewriting, and achieve effective identification of semantically rewritten content, thereby improving the intelligence level of the plagiarism detection system.

[0022] 3. This invention provides a document plagiarism detection method based on vector retrieval and large-scale model semantic analysis. Building upon plagiarism detection, it introduces a large-scale model to perform in-depth semantic analysis on highly similar segments, generating specific modification suggestions. This upgrades the plagiarism detection system from a "detection tool" to an "auxiliary modification tool," enhancing the interpretability and practicality of the results. It not only calculates the overall duplication rate but also accurately locates specific similar documents and segments, generating specific modification suggestions through the large-scale model to assist users in content optimization. It provides a clear and detailed plagiarism report, including duplication rate, a list of similar documents, specific similar segment locations, and modification suggestions, greatly improving user experience, the value of the plagiarism report, and the operability of the results. 4. This invention provides a document plagiarism detection method based on vector retrieval and large-scale model semantic analysis. It employs a word-weighted similarity rate calculation model, which allows the similarity of longer passages to have a greater impact on the final result. This makes the results more reflective of the true plagiarism situation, resulting in more scientific and realistic calculations, and avoiding the excessive influence of high similarity in short texts on the overall results. Through a refined calculation model, objective and quantitative similarity rate indicators are generated for each document.

[0023] 5. The present invention provides a document plagiarism detection method based on vector retrieval and large model semantic analysis, which adopts a two-level similarity calculation mechanism of "document-fragment": firstly, the fragment-level similarity is calculated, and then the document-level similarity is aggregated and calculated, and the references are sorted accordingly, so as to accurately locate the most relevant references. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to the embodiments.

[0025] This invention provides a document plagiarism detection method based on vector retrieval and large-scale model semantic analysis, specifically including the following steps: Step 1, Data Preprocessing: Intelligently split all documents uploaded by users and in the reference database according to their inherent semantic structure, and convert them into vector representations that can be understood by computers.

[0026] Step 1.1: Perform semantic unit intelligent segmentation on all documents to obtain plain text fragments of each document's segmentation. The system intelligently splits all Word documents from user-uploaded literature and the reference database into multiple semantically complete segments, as follows: This method utilizes the XML format characteristics of Word documents with the docx file type to parse the document structure, identify all heading paragraphs (such as "Chapter 1", "1.1", "Section 1", etc.) and their corresponding outline levels in the Word document; and divides the identified heading paragraphs into hierarchical levels according to the outline levels to obtain the heading hierarchy.

[0027] Based on the heading hierarchy, the content of the Word document is recursively divided into chapters in a tree structure.

[0028] For each leaf chapter of the tree structure, if its text length exceeds a preset threshold (such as 1024 characters), it is further segmented according to the natural paragraph boundaries to obtain split fragments, ensuring that each split fragment is semantically complete and has a controllable length.

[0029] Extract the plain text content of each segment to obtain the plain text segment, and record its original document ID and the number of words in the segment.

[0030] For user-uploaded documents, all plain text fragments are obtained, and a user-uploaded document fragment library consisting of semantically complete fragments is constructed. For all documents in the reference database, and all plain text fragments obtained, construct a reference fragment database consisting of semantically complete fragments.

[0031] In one specific implementation, extracting the plain text content of each segment to obtain a plain text segment includes: removing non-plain text elements from each segment (removing structured tags, metadata, and citation tags); compressing consecutive whitespace characters (spaces, tabs, and newlines) in the remaining text into a single space; and retaining paragraph ending tags (e.g., treating two or more consecutive newlines as paragraph separators and replacing them with a single newline character). Non-plain text elements include: HTML / XML tags, Markdown tags, LaTeX commands, header / footer tags, footnote / endnote tags, image / table placeholders, paragraph numbering, and special control characters.

[0032] When splitting documents from user-uploaded documents and all documents in the reference database, if the ability to parse the structure of Word documents is not available, sentence segmentation and topic segmentation models based on natural language processing (NLP) can be used to approximate the division of semantic units, but their accuracy and efficiency may not be as good as directly parsing the document structure.

[0033] Step 1.2: Convert the plain text fragment into text vectors to obtain text fragment vectors. Using a pre-trained text embedding model, all plain text fragments obtained in step 1.1 (including fragments from user-uploaded bibliographic fragment libraries and reference fragment libraries) are converted into high-dimensional semantic text fragment vectors. The text embedding model can be the BGE-Large model, the text-embedding-ada-002 model, the m3e model, etc.

[0034] The text fragment vectors include text fragment vectors from the user-uploaded bibliographic fragment library and text fragment vectors from the reference fragment library.

[0035] A vector retrieval library is used to construct an efficient vector index for all text fragment vectors in the reference fragment library (plagiarism comparison library) to enable fast similarity retrieval.

[0036] In one specific implementation, the FAISS library is used as a vector retrieval library to construct a vector index for all text fragment vectors in the reference fragment library.

[0037] Step 2: Perform a similarity search on the text fragment vectors uploaded by the user in the vector index of the reference fragment library, calculate the fragment similarity, recall similar fragments, and obtain a set of similar fragment pairs for each text fragment vector in the uploaded reference fragment library. For each text fragment vector uploaded by the user to the bibliographic fragment library, a similarity search is performed in the vector index of the bibliographic fragment library.

[0038] Cosine similarity is used to calculate the similarity between each text fragment vector in the user-uploaded document fragment library and each text fragment vector in each document in the reference fragment library; a similarity threshold is set, and only text fragment vectors in the reference fragment library with a similarity higher than the similarity threshold are recalled as reference fragments.

[0039] In one specific implementation, the similarity threshold is set to 0.5. The similarity threshold can be adjusted according to the actual application scenario and the required level of stringency for plagiarism detection.

[0040] For each text fragment vector uploaded by a user to the literature fragment library, the top 5 reference fragments with the highest similarity are selected to form a preliminary set of similar fragment pairs, and a similarity record is constructed. The similarity record for each set of similar fragment pairs includes: uploaded fragment text, uploaded fragment word count, uploaded literature ID, reference fragment text, reference ID, and similarity score.

[0041] Step 3: Based on the fragment similarity and the set of similar fragment pairs, calculate the core plagiarism detection indicators such as the duplication rate and document similarity of the user-uploaded document, and obtain similar documents. Step 3.1: Calculate the duplication rate of the user-uploaded single document based on fragment similarity. The duplication rate of a single document is calculated based on the highest similarity score of each text segment vector and the number of characters in the uploaded document. The specific calculation formula is as follows: CurCore = / ; In the formula, CurCore represents the duplication rate of a single document; MaxScore(i) represents the highest similarity score of the i-th text segment vector in a single document uploaded by a user; WordNum(i) represents the number of characters in the i-th text segment vector of a single document uploaded by the user; i represents each segment in a single document uploaded by the user; M represents the total number of segments that a single document uploaded by a user is split into.

[0042] If the i-th text fragment vector of a single document uploaded by a user does not recall any reference fragments, then MaxScore(i) is assigned the value 0.

[0043] This formula uses word count weighting to ensure that the similarity of longer text segments has a greater impact on the final document duplication rate, resulting in a more objective outcome.

[0044] Step 3.2: Calculate the document similarity score based on similar segments in the set, and sort the references to obtain similar documents. Based on the set of all similar fragment pairs obtained in step 2 (e.g., 50 uploaded fragments * 5 = 250 records), group them by reference ID.

[0045] For each unique reference ID, extract the set of all similar fragment pairs that are similar to all fragments of the single document uploaded by the user, and use this as the similarity record for that specific reference.

[0046] Based on the similarity record of the specific reference, for each segment of the user-uploaded single document, find the reference segment with the highest similarity to the segment in the specific reference, and record the highest similarity score and its text word count; aggregate the highest similarity scores and text word counts corresponding to all segments of the user-uploaded single document using the same calculation formula as in step 3.1, and calculate the similarity RefCore between the user-uploaded single document and the specific reference.

[0047] Sort all RefCores corresponding to all reference IDs in descending order, take the top 5, obtain the 5 references most similar to the user-uploaded single document and their respective similarity scores, and take the 5 references most similar to the user-uploaded single document as similar documents.

[0048] Step 4: For similar documents, perform semantic comparative analysis using a large language model. For the five most similar references found in step 3.2, a Large Language Model (LLM) is used to interactively compare any fragments in a single document uploaded by the user.

[0049] Users can select any segment from a user-uploaded document, and the system will automatically select and highlight the most similar segment from the references on the right.

[0050] The system calls the Large Language Model (LLM), taking the two highly similar segments as input. The instruction model analyzes the similarities and differences between the two in terms of semantics, structure, and word choice, and generates a "suggested modification" report, pointing out where the original text may have been modified, providing users with in-depth and actionable feedback.

[0051] Large Language Models (LLMs) can be selected from different commercial or open-source models, such as GPT-4, Claude, GLM, etc.

[0052] The present invention has been described in detail above with reference to the embodiments. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention. All contents not described in detail in the present invention can be derived from existing technologies.

Claims

1. A document plagiarism detection method based on vector retrieval and large-scale model semantic analysis, characterized in that, The method includes: Step 1: Intelligently split all documents uploaded by users and in the reference database according to their inherent semantic structure, and convert them into vector representations that can be understood by computers; Step 2: Perform a similarity search on the text fragment vectors of the user-uploaded literature fragment library in the vector index of the reference fragment library, calculate the fragment similarity, recall similar fragments, and obtain a set of similar fragment pairs for each text fragment vector in the uploaded literature fragment library; Step 3: Based on the fragment similarity and the set of similar fragment pairs, calculate the core plagiarism detection indicators such as the duplication rate and document similarity of the single document uploaded by the user, and obtain similar documents.

2. The document plagiarism detection method based on vector retrieval and large-scale model semantic analysis according to claim 1, characterized in that, Step 1 includes: Step 1.1: Perform semantic unit intelligent segmentation on all documents to obtain plain text fragments of each document's segmentation. Step 1.2: Convert the plain text fragment into a text vector to obtain the text fragment vector.

3. The document plagiarism detection method based on vector retrieval and large-scale model semantic analysis according to claim 1, characterized in that, Step 2 includes: using cosine similarity to calculate the similarity between each text fragment vector in the user-uploaded document fragment library and each text fragment vector in each document in the reference fragment library; setting a similarity threshold to recall text fragment vectors in the reference fragment library with similarity higher than the similarity threshold as reference fragments; and selecting the top 5 reference fragments with the highest similarity for each text fragment vector in the user-uploaded document fragment library to form a preliminary set of similar fragment pairs and construct a similarity record.

4. The document plagiarism detection method based on vector retrieval and large-scale model semantic analysis according to claim 3, characterized in that, The similarity record includes: uploaded fragment text, uploaded fragment word count, uploaded document ID, reference fragment text, reference ID, and similarity score.

5. The document plagiarism detection method based on vector retrieval and large-scale model semantic analysis according to claim 1, characterized in that, Step 3 includes: Step 3.1: Calculate the duplication rate of the single document uploaded by the user based on the fragment similarity; Step 3.2: Calculate the document similarity of the set based on similar segments, and sort the references to obtain similar documents.

6. The document plagiarism detection method based on vector retrieval and large-scale model semantic analysis according to claim 5, characterized in that, The formula for calculating the duplication rate of a single document in step 3.1 is as follows: CurCore = / ; In the formula, CurCore represents the duplication rate of a single document; MaxScore(i) represents the highest similarity score of the i-th text segment vector in a single document uploaded by a user; WordNum(i) represents the number of characters in the i-th text segment vector of a single document uploaded by the user; i represents each segment in a single document uploaded by the user; M represents the total number of segments that a single document uploaded by a user is split into.

7. The document plagiarism detection method based on vector retrieval and large-scale model semantic analysis according to claim 6, characterized in that, In step 3.1, if the i-th text segment vector of a single document uploaded by the user does not recall any reference segments, then MaxScore(i) is assigned the value 0.

8. The document plagiarism detection method based on vector retrieval and large-scale model semantic analysis according to claim 6, characterized in that, Step 3.2, calculating document similarity, specifically involves: Based on the set of all similar fragment pairs obtained in step 2, group them by reference ID; for each unique reference ID, extract the set of all similar fragment pairs with all fragments of the user-uploaded single document, as the similarity record for that specific reference; according to the similarity record for that specific reference, for each fragment of the user-uploaded single document, find the reference fragment with the highest similarity to that fragment in that specific reference, and record the highest similarity score and its text word count; aggregate the highest similarity scores and their text word counts corresponding to all fragments of the user-uploaded single document using the same calculation formula as in step 3.1, and calculate the similarity RefCore between the user-uploaded single document and that specific reference.

9. A document plagiarism detection method based on vector retrieval and large-scale model semantic analysis according to claim 8, characterized in that, Step 3.2, obtaining similar documents, specifically involves: Sort all RefCores corresponding to all reference IDs in descending order, take the top 5, obtain the 5 references most similar to the user-uploaded single document and their respective similarity scores, and take the 5 references most similar to the user-uploaded single document as similar documents.

10. The document plagiarism detection method based on vector retrieval and large-scale model semantic analysis according to claim 1, characterized in that, The method further includes: Step 4, performing semantic comparison analysis using a large language model for similar documents: For the similar documents found in Step 3.2, for any segment in a single document uploaded by the user, select the segment most similar to it from the similar documents, call the large language model, take these two highly similar segments as input, instruct the model to analyze the similarities and differences between the two in terms of semantics, structure, word choice, etc., and generate a "modification suggestion" report, pointing out where the original text may have been modified, providing users with in-depth and actionable feedback.

Images