A test question duplicate detection method based on semantic focus of a large language model

Abstract

The application discloses a kind of based on the semantic focus of test question duplicate checking method of large language model, the method is based on corpus construction and text normalization, complete the construction of question corpus and metadata annotation;Using semantic vectorization representation strategy, the question in corpus is mapped to dense vector, and offline semantic vector library is constructed;Propose semantic vector recall→SimHash denoising→Reranker model rearrangement screening mechanism, solve the problem that traditional literal comparison cannot identify synonymous rewriting;Propose multi-level screening-large model deep judgment-online threshold self-adaptive cooperation architecture, through real-time feedback continuous iteration, realize semantic level accurate duplicate checking;By starting mechanism, according to the data in artificial review library Dynamic adjustment vector recall threshold and large model deep judgment threshold, to maximize the accuracy and efficiency of duplicate checking.The application solves the problem of synonymous rewriting, short text representation failure, static threshold false alarm / miss.

Description

Technical Field

[0001] This invention relates to a test question plagiarism detection method, specifically a test question plagiarism detection method based on semantic focusing of a large language model. Background Technology

[0002] With the explosive growth of digital content, the importance of text plagiarism detection technology has become increasingly prominent. Existing plagiarism detection methods are mainly divided into two categories: literal matching-based methods and semantic analysis-based methods.

[0003] Literal matching-based plagiarism detection methods include N-gram and edit distance. N-gram is a classic text similarity detection method. Its basic idea is to divide the text into continuous substrings of length N (called N-grams), and then calculate the similarity by comparing the N-gram sets of two texts. The advantages of this method are its simplicity, high computational efficiency, and ability to quickly detect literal repetitions between texts. However, its limitations are also obvious. First, it can only detect completely identical substrings; for texts that have undergone synonym replacement, word order adjustment, or sentence transformation, the N-gram method will completely fail. Second, the N-gram method is very sensitive to the length and segmentation granularity of the text; different values ​​of N will lead to different plagiarism detection results, making it difficult to find a universally optimal solution. Edit distance (also known as Levenshtein distance) is a method for measuring the similarity between two strings. It measures their similarity by calculating the minimum number of editing operations (including insertion, deletion, and replacement of characters) required to transform one string into another. The edit distance method can detect subtle differences between texts and is sensitive to some small-scale text modifications. The disadvantages are that it has high computational complexity, low efficiency in deduplication for long texts or large-scale text sets, and it also relies on literal matching and cannot identify semantic similarity.

[0004] Deduplication methods based on semantic analysis include traditional and modern semantic analysis methods. Traditional methods mainly include TF-IDF (Term Frequency-Inverse Document Frequency) and word vector models. TF-IDF is a statistical text representation method that measures the importance of words by calculating their term frequency (TF) and inverse document frequency (IDF). TF-IDF can capture keyword information in text, but its representation ability drops sharply for short or sparse texts. Furthermore, TF-IDF cannot capture semantic relationships between words and cannot effectively identify synonyms or near-synonyms. Word vector models (such as Word2Vec and GloVe) can capture semantic similarity between words by embedding them into a high-dimensional vector space. However, word vector models have limitations when processing short texts. First, word vector models are based on static word embeddings, and cannot accurately represent the semantics of polysemous words or context-dependent words. Secondly, when processing short texts, word vector models are prone to sparsity issues due to the short text length and limited contextual information, thus affecting the accuracy of semantic representation. With the development of deep learning technology, deep learning-based semantic analysis methods have gradually become a research hotspot. These methods mainly include recurrent neural networks (RNNs), long short-term memory networks (LSTMs), gated recurrent units (GRUs), and the Transformer architecture. These models have the advantage of capturing long-distance dependencies and contextual information in text, thus representing the semantics of the text more accurately. However, they also have some problems in practical applications. First, training these models requires a large amount of labeled data, and obtaining labeled data is costly. Second, these models have high computational complexity and low efficiency in deduplication for large-scale text sets. Third, these models still suffer from inaccurate semantic representation when processing short texts. For example, in short text scenarios such as multiple-choice questions, due to the short text length and limited contextual information, these models may experience semantic drift or semantic ambiguity, thus affecting the accuracy of deduplication.

[0005] Therefore, existing plagiarism detection methods mainly have the following problems:

[0006] (1) Rewriting missed: Existing plagiarism detection systems rely on N-gram or edit distance, which can only perform literal comparison and cannot identify hidden repetitions after synonym replacement, word order adjustment and sentence transformation, resulting in a large number of semantic-level missed detections.

[0007] (2) Short text representation failure: In the scenario of short questions of 200 to 300 words, the TF-IDF or static word vectors have a sharp decline in representation ability due to the sparse vocabulary, resulting in insufficient robustness of the system to the semantic parsing of short texts and frequent misjudgments.

[0008] (3) Static threshold false alarms / false alarms: The existing system uses a fixed threshold, which cannot dynamically adapt to the distribution differences brought about by different subjects, difficulties or question bank evolution, making it difficult to balance recall and precision, and the cost of secondary manual review is high. Summary of the Invention

[0009] To address the issues of missing paraphrased responses, ineffective short text representations, and false positives / false negatives caused by static thresholds, this invention provides a semantically focused test question deduplication method based on a large language model. This method focuses on identifying similarities between questions through semantic analysis, without relying on traditional literal matching techniques. Through a three-level progressive process of "multi-level screening → deep judgment by a large model → threshold adaptation," it can identify hidden repetitions that are "different in expression but semantically consistent" without comparing the text itself, achieving true semantic-level deduplication. Targeted semantic representations are used at each level to form complementary multi-dimensional features, significantly improving the robustness of short text semantic parsing. Online logistic regression is introduced at the comprehensive judgment layer, and manual feedback is used to fine-tune the recall and judgment thresholds, achieving a dynamic balance between recall and precision, and reducing the cost of secondary manual review.

[0010] The objective of this invention is achieved through the following technical solution:

[0011] A semantic-focused question plagiarism detection method based on a large language model is disclosed. This method utilizes a deep learning model for semantic analysis to achieve text plagiarism detection. Semantic features of the text are extracted using a pre-trained language model (such as BERT, RoBERTa, etc.), and the similarity between feature vectors is calculated to determine text duplication. Specifically, the input text is first pre-processed, including removing useless symbols and filtering stop words to reduce noise. Then, the text is encoded using a pre-trained language model to obtain semantic embedding vectors, and the similarity between vectors is calculated using methods such as cosine similarity. Finally, a preset similarity threshold is used to determine whether the text is duplicated. The method includes the following steps:

[0012] Step 1: Corpus Construction

[0013] Based on existing data crawling and processing methods, a dedicated corpus for plagiarism detection was constructed. The specific steps are as follows:

[0014] Step 1-1: By integrating the question resources accumulated by users, as well as the content of publicly available textbooks, teaching aids, authoritative question bank websites, and education news portals, collect multiple-choice questions related to the subject in batches;

[0015] Steps 1-2: Text cleaning: Use hash algorithms to remove duplicates; use regular expressions to remove HTML tags, advertisements, and special characters; unify UTF-8 encoding and full-width / half-width format.

[0016] Steps 1-3: Mark the time and source of the questions to form metadata;

[0017] Step 2, semantic vector pre-generation:

[0018] A semantic vectorization representation strategy is adopted to map the questions in the corpus into dense vectors, and an offline semantic vector library is constructed. The specific steps are as follows:

[0019] Step 2-1: The system calls the embedding model to map all the questions in the corpus into dense vectors of fixed dimensions at once;

[0020] Step 2-2: Persistently store the above vectors using local vector indexes to form an offline, closed, and hot-loadable semantic vector library;

[0021] Step 3: Multi-stage semantic filtering:

[0022] Based on standard vector retrieval and considering the characteristics of short multiple-choice texts, a "multi-stage semantic filtering" mechanism is proposed to complete the filtering process. The specific steps are as follows:

[0023] Step 3-1, Vector Index Construction:

[0024] Step 3-1-1: The vector index adopts a structure suitable for fast retrieval in high-dimensional space to optimize the location and distance calculation of candidate questions;

[0025] Step 3-1-2: Control the size of a single cluster by configuring the construction parameters of the vector index to ensure retrieval efficiency;

[0026] Step 3-1-3: Before importing the semantic vector of the question into the vector index, normalize it so that the cosine similarity can be approximated by using Euclidean distance.

[0027] Step 3-2, User Input Vectorization:

[0028] Step 3-2-1: When a user submits a question to be searched in the local system, the system will perform standardized text preprocessing on it, including cleaning and stop word filtering, to ensure consistency with the vector space of the questions in the question bank.

[0029] Step 3-2-2: Using an embedding method with the same dimension as the index vector, the user input is converted into a normalized vector, which serves as the query key for retrieval.

[0030] Step 3-3, Approximate Nearest Neighbor Search:

[0031] Based on the locally built vector index, the system performs an approximate nearest neighbor search. Using the user-input vector as a reference, it quickly finds a certain number of questions in the question bank that are most similar to it. The search results include the question ID and its distance from the query vector.

[0032] Steps 3-4: Cosine threshold filtering:

[0033] Step 3-4-1: Calculate the cosine similarity of the search results and convert the distance into a similarity score;

[0034] Step 3-4-2: Set an initial threshold and periodically update the threshold based on the performance of the validation set using a local optimization algorithm to maximize precision.

[0035] Step 3-4-3: The system retains only questions with similarity scores higher than the threshold, forming a concise candidate set. The output information includes question ID, similarity score, and vector distance.

[0036] Steps 3-5: SimHash noise reduction:

[0037] A further round of locality-sensitive hashing is applied to the simplified candidate set after cosine thresholding. The specific steps are as follows:

[0038] Step 3-5-1: Calculate a fixed-length binary fingerprint for each candidate topic;

[0039] Step 3-5-2: Compare the fingerprint of the question entered by the user with the Hamming distance. Candidates whose distance exceeds the set threshold are immediately eliminated, while those within the distance are retained.

[0040] Steps 3-6: Reranker reordering:

[0041] Step 3-6-1: Load the pre-trained Reranker model, construct the candidate questions and the questions to be searched as input pairs with the candidate questions after SimHash denoising, in the format "{question to be searched}[SEP]{candidate questions}", and send them to the Reranker model in batches. The Reranker model calculates the semantic similarity score of each input pair and re-sorts the candidate questions according to the score.

[0042] Step 3-6-2: Select the top k ranked questions from the sorted candidate questions as the final candidate results;

[0043] Step 4, Deep Decision Making for Large Models:

[0044] The final semantic determination is completed using a large language model. The specific steps are as follows:

[0045] Step 4-1: Construct prompt words by combining the question to be searched with each candidate question according to a fixed template;

[0046] Step 4-2: The prompt words are fed into the local large language model in sequence, and the large language model outputs the repetition probability between 0 and 1;

[0047] Step 4-3, the construction of prompt words and the model reasoning are encapsulated as a loop process until all candidate questions are judged;

[0048] Step 4-4: After the questions are judged, the questions judged as "non-duplicate" will be added to the temporary database. When the number of questions reaches 1000, all the questions in the temporary database will be added to the database and the index will be updated on the next startup.

[0049] Step 5, Threshold Adaptation:

[0050] By employing a "start-up review" mechanism, the vector recall threshold and the large model depth judgment threshold are dynamically adjusted based on the data in the manual review database to maximize the accuracy and efficiency of plagiarism detection. The specific steps are as follows:

[0051] Step 5-1: The system operates on a "start-up review" cycle: whenever the system restarts, it immediately backtracks the local manual review database and brings all manual tags accumulated since the last shutdown into view at once.

[0052] Step 5-2: If the review record is empty, the existing two-level threshold will be silently applied to ensure the continuity of the deduplication logic; if the review record is not empty, the vector recall threshold θ will be calculated based solely on the samples in this batch. vec With large model depth threshold θ llm The corresponding Precision, Recall and F1 Score are evaluated iteratively within a preset interval, with the goal always locked on maximizing the single objective function of F1.

[0053] Step 5-3: The new threshold obtained through optimization is then written to local storage. If the sample sparsity causes the F1 score to drop beyond the preset warning line, the system will automatically roll back to the previous effective version and record the rollback time, triggering reason, and difference between the old and new thresholds in the local log. This ensures that no matter how long the system is down, the two thresholds will still be accurately aligned with the current distribution of the question bank when it is restarted, and the recall and precision will always be at the best balance point.

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] 1. Improved semantic-level plagiarism detection accuracy: Adopting a three-level progressive process of "multi-batch screening → deep judgment of large model → threshold adaptation", it accurately identifies hidden repetitions such as synonym rewriting and word order adjustment, and solves the problem of synonym rewriting slipping through the net.

[0056] 2. Enhanced robustness of short text parsing: For short text scenarios, a multi-dimensional semantic representation strategy is adopted to significantly improve the robustness of short text semantic parsing, avoid misjudgment caused by lexical sparsity, and solve the problem of short text representation failure.

[0057] 3. Dynamically adapt to question bank distribution: Introduce online logistic regression to fine-tune the recall and decision thresholds in real time, dynamically balance recall and precision, reduce the cost of secondary manual review, and solve the problem of false alarms / missed alarms caused by static thresholds.

[0058] 4. Significantly improved plagiarism detection efficiency: Through efficient vector indexing and a two-stage coarse screening mechanism, the reasoning burden of large language models is reduced, significantly improving plagiarism detection efficiency and enabling efficient processing of massive question banks. Attached Figure Description

[0059] Figure 1 This is a flowchart of a semantic-focused test question plagiarism detection process based on a large language model.

[0060] Figure 2 Multi-stage semantic filtering;

[0061] Figure 3 This is a question for the input system;

[0062] Figure 4 Standardized text preprocessing;

[0063] Figure 5 The 500 questions were obtained using an approximate nearest neighbor search.

[0064] Figure 6 The 300 questions obtained by denoising the hash;

[0065] Figure 7 The 10 problems obtained by reordering using BGE-Reranker;

[0066] Figure 8 To score the repetition probability of 10 candidate questions for DeepSeek. Detailed Implementation

[0067] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0068] This invention provides a semantic-focused question plagiarism detection method based on a large language model, such as... Figure 1As shown, the first step involves building a corpus and standardizing text to construct a question corpus and annotate its metadata. The second step employs a semantic vectorization strategy to map the questions in the corpus into dense vectors and construct an offline semantic vector library. The third step, based on standard vector retrieval and considering the short text characteristics of multiple-choice questions, proposes a three-stage screening mechanism: "semantic vector recall → SimHash denoising → Reranker model re-ranking," solving the problem that traditional literal comparison cannot identify paraphrasing. The fourth step utilizes a large language model to creatively propose a three-stage collaborative architecture: "multi-level screening - large model deep judgment - online threshold adaptation," achieving semantic-level accurate plagiarism detection through continuous iteration and real-time feedback. Finally, through a "start-up review" mechanism, the vector recall threshold and large model deep judgment threshold are dynamically adjusted based on data in the manual review library to maximize the accuracy and efficiency of plagiarism detection. The specific steps are as follows:

[0069] Step 1: Corpus Construction

[0070] To ensure the authenticity and scalability of the question data, we first constructed a dedicated corpus for plagiarism detection based on existing data crawling and processing methods. The specific steps are as follows:

[0071] Step 1-1: By integrating the question resources accumulated by users, as well as the content of publicly available textbooks, teaching aids, authoritative question bank websites, and education news portals, collect multiple-choice questions related to the subject in batches.

[0072] Steps 1-2: Text cleaning: Use hash algorithms to remove duplicates; use regular expressions to remove HTML tags, advertisements, special symbols, etc.; unify UTF-8 encoding and full-width / half-width format.

[0073] Steps 1-3: Mark the time and source of the questions to form metadata.

[0074] Step 2, semantic vector pre-generation:

[0075] After the corpus is built, to prevent traditional literal comparison from failing to identify hidden repetitions such as synonym rewriting and word order adjustments, this stage adopts a semantic vectorization representation strategy. This maps the questions in the corpus into dense vectors and constructs an offline semantic vector library. The specific steps are as follows:

[0076] Step 2-1: The system calls the embedding model to map all the questions in the corpus into a dense vector of fixed dimensions at once.

[0077] Step 2-2: Persistently store the above vectors using a local vector index to form an offline, closed, and hot-loadable semantic vector library. This semantic vector library serves as the benchmark for subsequent recall and comparison stages, capturing subtle semantic differences in the questions and effectively avoiding the risk of missed detections caused by literal comparison.

[0078] Step 3: Multi-stage semantic filtering:

[0079] After the semantic vectors are pre-generated, the system faces a massive question bank. Directly performing deep semantic comparisons on all historical questions using a large language model would incur enormous computational overhead. To avoid this bottleneck, this stage introduces a "multi-stage semantic filtering" mechanism, completing the filtering process in six steps, such as... Figure 2 As shown, the specific steps are as follows:

[0080] Step 3-1, Vector Index Construction: This stage aims to build an efficient vector index for the question bank. The index adopts a structure suitable for fast retrieval in high-dimensional space to optimize the location and distance calculation of candidate questions. By configuring the index construction parameters, the size of a single cluster is controlled to ensure retrieval efficiency. Before importing the semantic vectors of the questions into the index, they are normalized to approximate the cosine similarity using Euclidean distance.

[0081] Step 3-2, User Input Vectorization: When a user submits a question to be searched in the local system, the system performs standardized text preprocessing, including cleaning and stop word filtering, to ensure consistency with the vector space of questions in the question bank. Next, using an embedding method with the same dimension as the index vector, the user input is converted into a normalized vector, which serves as the query key for retrieval.

[0082] Step 3-3, Approximate Nearest Neighbor Search: Based on the locally built vector index, the system performs an approximate nearest neighbor search. Using the user-input vector as a reference, it quickly finds a certain number of questions in the question bank that are most similar to it. This number needs to be set after targeted testing for different questions. The search results include the question ID and its distance from the query vector, to facilitate subsequent manual review or automatic processing.

[0083] Steps 3-4: Cosine Threshold Filtering: Cosine similarity is calculated on the search results, converting the distance into a similarity score. An initial threshold is set, and this threshold is periodically updated using a local optimization algorithm based on the performance of the validation set to maximize precision. Finally, the system retains only questions with similarity scores higher than the threshold, forming a streamlined candidate set. In typical operation, the size of the candidate set should be much smaller than the original question bank, significantly improving the efficiency of subsequent processing. The output information includes question ID, similarity score, and vector distance, providing necessary data support for subsequent model depth assessment and manual review.

[0084] Steps 3-5, SimHash Denoising: To further compress the candidate set and eliminate approximation errors, the system adds a round of locality-sensitive hashing (LSH) filtering to the streamlined candidate set after cosine thresholding. Specifically, the system calculates a fixed-length binary fingerprint for each candidate question and then compares it with the Hamming distance of the fingerprint of the user-input question; candidates with a distance exceeding a set threshold are immediately eliminated, while those within the threshold are retained. This step further reduces the size of the candidate set, significantly lowering the inference burden of subsequent large language models.

[0085] Steps 3-6: Reranker Re-ranking: After SimHash denoising, the system introduces the Reranker module to perform deep semantic re-ranking of candidate questions to select the Top-10 candidate questions most semantically relevant to the question to be searched. The system loads a pre-trained Reranker model, which is based on a cross-encoder architecture and can deeply analyze the semantic interaction information between the query and candidate questions. The candidate questions denoised by SimHash and the question to be searched are constructed as input pairs in the format "{question to be searched}[SEP]{candidate question}", and are batched into the Reranker model. The model calculates the semantic similarity score for each input pair and re-ranks the candidate questions based on the score. Finally, the system selects the top k ranked questions from the ranked candidate questions as the final candidate results. These candidate questions are highly semantically relevant to the question to be searched, providing more accurate input for subsequent deep semantic judgment and significantly improving the accuracy and reliability of deduplication.

[0086] Step 4, Deep Decision Making for Large Models:

[0087] After multiple stages of screening, the system retains only a few candidate questions. To avoid the risk of misjudgment caused by traditional similarity thresholds, this stage uses a large language model to complete the final semantic determination. The specific steps are as follows:

[0088] Step 4-1: The system first constructs prompt words for the question to be searched and each candidate question according to a fixed template. The prompt word template can be "Please judge whether the following two questions are semantically equivalent and give the probability\n{question to be searched}\n{candidate question}";

[0089] Step 4-2: The prompt words are fed into the local large model in sequence, and the large model outputs the repetition probability between 0 and 1.

[0090] Step 4-3: Since the number of candidate questions is greater than one, the prompt word construction and model reasoning actions are encapsulated into a loop process until all candidate questions are judged.

[0091] Step 4-4: After the questions are judged, the questions that are judged as "non-repeating" will be added to the temporary database. When the number of questions reaches 1000, the system will add all the questions in the temporary database to the database and update the index on the next startup.

[0092] Step 5, Threshold Adaptation:

[0093] Step 5-1: The system operates on a "start-up verification" cycle: whenever the system restarts, it immediately backtracks to the local manual verification database, bringing all manual labels accumulated since the last shutdown into view at once; these manual labels are generated through secondary verification of the system's judgment results by professional reviewers, and have high accuracy and authority.

[0094] Step 5-2: If the review record is empty, the existing two-level threshold will be silently applied to ensure the continuity of the deduplication logic; if the review record is not empty, the vector recall threshold θ will be calculated based solely on the samples in this batch. vec With large model depth threshold θ llm The corresponding Precision, Recall, and F1 Score are calculated and iterated within a preset range, with the goal of always maximizing the single objective function of F1.

[0095]

[0096]

[0097]

[0098] Wherein, TP (True Positive) represents the number of true positives, that is, the positives correctly identified by the model; FP (False Positive) represents the number of false positives, that is, the negatives incorrectly identified as positives by the model; and FN (False Negative) represents the number of false negatives, that is, the positives incorrectly identified as negatives by the model.

[0099] Step 5-3: The optimized new threshold is immediately written to local storage and takes effect without restarting any modules. If the sample sparsity causes the F1 score to drop beyond the preset warning line, the system automatically rolls back to the previous effective version and records the rollback time, triggering reason, and difference between the old and new thresholds in the local log. This ensures that no matter how long the system is down, the two thresholds will still be accurately aligned with the current distribution of the question bank upon restart, and the recall and precision will always be at the optimal balance point.

[0100] Example:

[0101] The system development platform is a Linux operating system, with eight NVIDIA 4090D GPUs. The program is written in Python 3.11, using the PyTorch 2.3.0 framework and transformer version 4.43.4. The model is deployed locally via Ollam.

[0102] The overall implementation and system construction process of the method is as follows:

[0103] First, 200,000 multiple-choice questions were obtained from user-accumulated question banks, open educational platforms (National Education Resource Public Service Platform), and authoritative publications ("Five-Year College Entrance Examination, Three-Year Simulation"). The SimHash algorithm was used for text deduplication, with a similarity threshold of 0.85, to remove duplicate questions. Regular expressions were used to remove HTML tags, redundant spaces, and special characters, and the text encoding was standardized to UTF-8. Timestamps (e.g., "2023") and source information (e.g., "National College Entrance Examination Paper I") were extracted using regular expressions to generate metadata. Finally, a corpus containing 200,000 annotated text entries was constructed.

[0104] Next, the BERT-base model (768-dimensional) is used to encode all the questions in the corpus, mapping each question to a dense vector of fixed dimensions (768-dimensional). To achieve efficient vector retrieval, the FAISS (FacebookAI Similarity Search) library is used to build a vector index, with the IVFADC (Inverted File + Product Quantization) structure selected and the cluster size controlled at 1000 to optimize retrieval efficiency. The generated dense vectors are persistently stored in the local vector index.

[0105] Next, when a user submits a question to be searched, the system first performs standardized text preprocessing, including cleaning and stop word filtering, to ensure consistency with questions in the question bank. Then, it calls the BERT-base model to convert the user input into a normalized vector, which serves as the query key. Based on the locally built vector index, the system performs an approximate nearest neighbor search, quickly finding the 500 most similar questions in the question bank, using the user-input vector as a reference. The search results include the question ID and its distance from the query vector. Subsequently, cosine similarity is calculated on the search results, converting the distance into a similarity score, with an initial threshold of 0.8.

[0106] To further reduce the candidate size and eliminate approximation errors, the system adds a round of locality-sensitive hashing (LSH) filtering to the streamlined candidate set after cosine thresholding. Specifically, the system calculates a 64-bit binary fingerprint for each candidate question using the SimHash algorithm. Then, it compares the fingerprint of each candidate question with the fingerprint of the user-input question using Hamming distance, setting a Hamming distance threshold of 5. Candidates with a distance exceeding the threshold are immediately eliminated, while those within the threshold are retained.

[0107] After hash filtering, the system combines the question to be searched and each candidate question into question pairs, and inputs them in batches into the BGE-Reranker-v2-M3 model. The pre-trained BGE-Reranker-v2-M3 model performs deep semantic analysis on each question pair, calculating a semantic similarity score. Subsequently, the system re-ranks all candidate questions based on these scores, ensuring that the semantically most relevant questions are ranked first. Finally, the system selects the top 10 ranked questions as the final candidate results.

[0108] After screening, the system retains only 10 candidate questions. This stage uses the deepseek-r1:70B large language model to complete the final semantic determination. The system first constructs prompt words for the question to be searched and each candidate question according to a fixed template: "Please determine whether the following two questions are semantically equivalent and give the probability\n{question to be searched}\n{candidate question}". The prompt words are sequentially sent to the locally deployed deepseek-r1:70B model, which outputs the repetition probability between 0 and 1. Since the number of candidate questions is greater than one, the prompt word construction and model inference are encapsulated as a loop process until all candidate questions are judged.

[0109] For a complete example of plagiarism checking for a single question, please see [link / reference]. Figures 3-8 . Figure 3 The image shows the user-inputted question for the plagiarism check, a multiple-choice question about the concept of "digital immortality," containing a stem and four options: A, B, C, and D. This is the initial input for the plagiarism check process. Figure 4 After standardized text preprocessing, redundant spaces and special characters in the questions are removed, and the text format is unified to UTF-8 encoding, laying the foundation for subsequent vector conversion. The preprocessed questions are then vectorized using an embedding model, and approximate nearest neighbor retrieval is used to obtain... Figure 5 The 500 candidate questions shown represent a set of questions in the question bank that are initially semantically similar to the question to be searched. They then proceed to a hash denoising stage, where the Hamming distance is calculated using the SimHash algorithm and filtered to obtain the final result. Figure 6The 300 questions shown have had irrelevant candidates eliminated due to vector approximation errors, further reducing the candidate size. Next, the BGE-Reranker-v2-M3 model performs deep semantic re-ranking on the 300 candidate questions, ultimately yielding... Figure 7 The Top-10 candidate questions shown represent the best performers in terms of semantic relevance. Finally, the deepseek-r1:70B large language model performs semantic equivalence determination on each of these 10 candidate questions, such as... Figure 8 As shown, the specific repetition probability is output (the repetition probability of the candidate question shown in the example is 0.85), completing the entire plagiarism detection process from input to final judgment.

Claims

1. A semantic-focused test question plagiarism detection method based on a large language model, characterized in that... The method includes the following steps: Step 1: Corpus Construction Based on existing data crawling and processing methods, a dedicated corpus for plagiarism detection was constructed. Step 2, semantic vector pre-generation: A semantic vectorization representation strategy is adopted to map the questions in the corpus into dense vectors and to build an offline semantic vector library. Step 3: Multi-stage semantic filtering: Based on standard vector retrieval and considering the characteristics of short multiple-choice texts, a "multi-stage semantic filtering" mechanism is proposed to complete the filtering process. The specific steps are as follows: Step 3-1, Vector Index Construction: Step 3-1-1: The vector index adopts a structure suitable for fast retrieval in high-dimensional space to optimize the location and distance calculation of candidate questions; Step 3-1-2: Control the size of a single cluster by configuring the construction parameters of the vector index to ensure retrieval efficiency; Step 3-1-3: Before importing the semantic vector of the question into the vector index, normalize it so that the cosine similarity can be approximated by using Euclidean distance. Step 3-2, User Input Vectorization: Step 3-2-1: When a user submits a question to be searched in the local system, the system will perform standardized text preprocessing on it, including cleaning and stop word filtering, to ensure consistency with the vector space of the questions in the question bank. Step 3-2-2: Using an embedding method with the same dimension as the index vector, the user input is converted into a normalized vector, which serves as the query key for retrieval. Step 3-3, Approximate Nearest Neighbor Search: Based on the locally built vector index, the system performs an approximate nearest neighbor search. Using the user-input vector as a reference, it quickly finds a certain number of questions in the question bank that are most similar to it. The search results include the question ID and its distance from the query vector. Steps 3-4: Cosine threshold filtering: Step 3-4-1: Calculate the cosine similarity of the search results and convert the distance into a similarity score; Step 3-4-2: Set an initial threshold and periodically update the threshold based on the performance of the validation set using a local optimization algorithm to maximize precision. Step 3-4-3: The system retains only questions with similarity scores higher than the threshold, forming a concise candidate set. The output information includes question ID, similarity score, and vector distance. Steps 3-5: SimHash noise reduction: Add a round of locality-sensitive hashing to the simplified candidate set after cosine threshold filtering; Steps 3-6: Reranker reordering: Step 3-6-1: Load the pre-trained Reranker model, construct the candidate questions and the questions to be searched as input pairs with the candidate questions after SimHash denoising, in the format "{question to be searched}[SEP]{candidate questions}", and send them to the Reranker model in batches. The Reranker model calculates the semantic similarity score of each input pair and re-sorts the candidate questions according to the score. Step 3-6-2: Select the top k ranked questions from the sorted candidate questions as the final candidate results; Step 4, Deep Decision Making for Large Models: The final semantic determination is completed using a large language model. Step 5, Threshold Adaptation: By employing a "start-up review" mechanism, the vector recall threshold and the large model depth judgment threshold are dynamically adjusted based on the data in the manual review database to maximize the accuracy and efficiency of plagiarism detection. The specific steps are as follows: Step 5-1: The system operates on a "start-up review" cycle: whenever the system restarts, it immediately backtracks the local manual review database and brings all manual labels accumulated since the last shutdown into view at once; Step 5-2: If the review record is empty, the existing two-level threshold will be used silently to ensure the continuity of the deduplication logic. If the review record is not empty, then the vector recall threshold θ is calculated based solely on the samples from this batch. vec With large model depth threshold θ llm The corresponding Precision, Recall and F1 Score are evaluated iteratively within a preset interval, with the goal always locked on maximizing the single objective function of F1. Step 5-3: The new threshold obtained through optimization is then written to local storage. If the sample sparsity causes the F1 score to drop beyond the preset warning line, the system will automatically roll back to the previous effective version and record the rollback time, triggering reason, and difference between the old and new thresholds in the local log. This ensures that no matter how long the system is down, the two thresholds will still be accurately aligned with the current distribution of the question bank when it is restarted, and the recall and precision will always be at the best balance point.

2. The test question plagiarism detection method based on semantic focus of a large language model according to claim 1, characterized in that... The specific steps of step 1 are as follows: Step 1-1: By integrating the question resources accumulated by users, as well as the content of publicly available textbooks, teaching aids, authoritative question bank websites, and education news portals, collect multiple-choice questions related to the subject in batches; Steps 1-2: Text cleaning: Use hash algorithms to remove duplicates; use regular expressions to remove HTML tags, advertisements, and special characters; unify UTF-8 encoding and full-width / half-width format; Steps 1-3: Mark the time and source of the questions to form metadata.

3. The test question plagiarism detection method based on semantic focus of a large language model according to claim 1, characterized in that... The specific steps of step 2 are as follows: Step 2-1: The system calls the embedding model to map all the questions in the corpus into dense vectors of fixed dimensions at once; Step 2-2: Persistently store the above vectors using local vector indexes to form an offline, closed, and hot-loadable semantic vector library.

4. The test question plagiarism detection method based on semantic focus of a large language model according to claim 1, characterized in that... The specific steps of steps 3-5 are as follows: Step 3-5-1: Calculate a fixed-length binary fingerprint for each candidate topic; Step 3-5-2: Compare the fingerprint of the question entered by the user with the Hamming distance. Candidates whose distance exceeds the set threshold are immediately eliminated, while those within the distance are retained.

5. The test question plagiarism detection method based on semantic focus of a large language model according to claim 1, characterized in that... The specific steps of step 4 are as follows: Step 4-1: Construct prompt words by combining the question to be searched with each candidate question according to a fixed template; Step 4-2: The prompt words are fed into the local large language model in sequence, and the large language model outputs the repetition probability between 0 and 1; Step 4-3, the construction of prompt words and the model reasoning are encapsulated as a loop process until all candidate questions are judged; Step 4-4: After the questions are judged, the questions that are judged as "non-duplicate" will be added to the temporary database. When the number of questions reaches 1000, all the questions in the temporary database will be added to the database and the index will be updated on the next startup.

Images