A test paper de-duplication and data fusion method, a storage medium and an equipment

Abstract

The application discloses a test paper de-duplication identification and data fusion method, a storage medium and equipment, and through construction of a multi-granularity semantic comparison model and a dynamic threshold determination mechanism, accurate identification of deep-level similarity of test paper content is realized, and misjudgment caused by factors such as naming difference and layout change is effectively overcome; a unified virtual test paper identification (VTID) mechanism is introduced, test papers with consistent content are merged into the same logical entity, and the consistency of data views is ensured; through metadata fusion, the data integrity can be ensured. By using the application, the redundancy rate of a question bank can be significantly reduced, resource utilization rate and teaching data analysis accuracy are improved, and a high-quality data basis is provided for intelligent test paper setting, personalized practice and teaching quality evaluation in a K12 education tablet scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent education technology, specifically to a method, storage medium, and device for test paper deduplication and data fusion. Background Technology

[0002] In the process of digitalizing K-12 education, various schools and educational research institutions have continuously uploaded a large number of test paper resources to the question bank platform, resulting in a large number of duplicate test papers in the question bank with different titles but highly similar or even completely identical content.

[0003] Traditional question bank management relies primarily on manual review or simple text matching for duplicate checks, making it difficult to handle complex situations such as naming differences across institutions, format changes, and partial modifications. Furthermore, the lack of a unified test paper identification mechanism often results in identical test papers being stored and analyzed multiple times, leading to fragmented and scattered metadata (such as citation counts, usage duration, and source information), which negatively impacts the accuracy of teaching data analysis and the level of intelligence in resource recommendation.

[0004] Existing technologies mostly focus on deduplication of individual questions or hash comparison of the entire file, which cannot effectively identify test paper variants with similar structures but minor adjustments in expression, nor have they established a closed-loop management system that supports automatic fusion of multi-source data. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention aims to provide a method, storage medium, and device for test paper deduplication and data fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A method for test paper deduplication and data fusion includes the following steps: S1. Obtain the original content data of the test papers to be included in the question bank; S2. Perform structured parsing on the original content data of the test paper to be processed, extract the question sequence list, the question stem text of each question, the option set, the correct answer identifier, the knowledge point tags and the layout features, and form the test paper to be processed into a structured test paper vector. S3. Input the structured test paper vector obtained in step S2 into the multi-granularity semantic comparison model, and calculate its question-by-question similarity score and whole-paper similarity score with each existing test paper in the question bank. S4. Based on the question-by-question similarity score and the whole-paper similarity score calculated in step S3, a dynamic threshold determination mechanism is used to determine whether the test paper to be processed is a duplicate test paper. S5. If the test paper is determined to be a duplicate, the metadata fusion process is initiated to merge and calibrate the source institution, upload time, and number of references of the test paper to be processed with the metadata of the corresponding existing test papers, and to update the global statistical data. S6. After metadata fusion, assign a unified virtual test paper identifier (VTI) to the test papers to be processed and the corresponding existing books, so as to realize the logical normalization management of test papers with cross titles but consistent content.

[0007] Furthermore, in step S2, the specific process by which the multi-granularity semantic comparison model calculates the question-by-question similarity score and the whole-paper similarity score between the test paper to be processed and a certain existing test paper in the question bank is as follows: (1) Calculation of question-by-question similarity score: (1.1) Using the BERT model pre-trained in the field of education, the question stem text of each question in the test paper to be processed and the existing test paper is converted into a high-dimensional vector; (1.2) Calculate the semantic similarity between each question in the test paper to be processed and all questions in the existing test paper, construct a similarity matrix, and find the question pairing combination that makes the overall matching score the highest by using the optimal matching strategy; then calculate the cosine similarity between the high-dimensional vectors of the two paired questions to obtain a semantic similarity value in the range of [0,1]. (1.3) The question-by-question similarity score between the two questions is calculated using a weighted fusion method: Question-by-question similarity score = w1 × semantic similarity + w2 × option set similarity + w3 × consistency of correct answer label; Among them, the weights w1 > w2 ≈ w3; (1.4) Calculate the average of the question-by-question similarity scores for the test paper to be processed and the existing test papers; (2) Calculation of the whole-volume similarity score: (2.1) Construction of the relational graph of test questions and analysis of topological similarity: Each test paper is modeled as a question relationship graph, with nodes representing questions and edges representing logical or sequential relationships between questions; then the similarity between the test paper to be processed and the question relationship graphs of existing test papers is quantified, and a structural similarity score S_struct is output; (2.2) Generating content digest fingerprints based on the locality-sensitive hashing algorithm: Based on the Locality Sensitive Hash (LSH) algorithm, a compact content summary fingerprint is generated for the entire test paper. Then, the test paper to be processed is compared with the content summary fingerprints of all existing test papers in the question bank, and the fingerprint similarity Slsh between the test paper to be processed and all existing test papers in the question bank is calculated. (2.3) Fusion calculation of whole-volume similarity: Overall similarity score = α × average question-by-question similarity score + β × S_struct + γ × Slsh; Where α, β, γ are adjustable weights; The overall similarity score is used to determine whether two test papers are duplicates with identical content.

[0008] Furthermore, in step S4, the dynamic threshold determination mechanism specifically adjusts the determination threshold automatically based on the subject, grade level, and question type of the test paper. The dynamic threshold is calculated based on the basic similarity threshold T0, the complexity factor g of the test paper to be processed, and the noise tolerance factor f. T = T0 × (1 + g - f); Among them, the complexity factor g of the test paper to be processed is determined based on the number of questions, the proportion of subjective questions, and the density of mixed text and graphics; the noise tolerance factor f is dynamically adjusted based on the OCR recognition confidence and the historical accuracy records of the source institution.

[0009] Furthermore, in step S5, the metadata fusion process is as follows: The system prioritizes duplicate exam papers between those awaiting processing and those already stored, determining which version is the master version and which is the non-master version based on the source institution's level, teacher feedback, and usage frequency. It then synchronizes valid metadata increments from the non-master version to the master version, including new citation records, supplementary knowledge point tags, and user error correction suggestions. For conflicting metadata, it employs a timestamp-first or voting mechanism to determine the appropriate value, ensuring the consistency and integrity of the merged metadata.

[0010] Furthermore, in step S6, the unified virtual test paper identifier VTID is generated using an irreversible hash function combined with the education region code. The generation formula is as follows: VTID = H(Sorted(QuestionID_List)) || Region_Code || Subject_Grade_Code; Where H() is the SHA-256 hash function, Sorted() is the sorting function, QuestionID_List is the list of unique IDs for all questions, and Sorted(QuestionID_List) means sorting the list of unique IDs for all questions according to the numerical value of the unique IDs; Region_Code represents the code of the main region for which the test paper is applicable, and Subject_Grade_Code represents the combination code of subject and grade; the || symbol represents string concatenation, which is directly connected without adding a separator.

[0011] Furthermore, in step S4, when it is determined that the test paper to be processed is a duplicate of an existing test paper, a warning is issued to the uploader, the corresponding existing test paper is displayed, and the uploader is provided with three processing paths: forced upload, merging into an existing test paper, or creating a revision branch. When the uploader chooses forced upload, the test paper to be processed is uploaded to the question bank as an independent test paper. When the uploader chooses to merge into an existing test paper, steps S5 and S6 are executed. When the uploader chooses to create a revision branch, the test paper to be processed is uploaded to the question bank and a version number is generated for it.

[0012] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0013] The present invention also provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the above-described method.

[0014] The beneficial effects of this invention are as follows: 1. This invention achieves accurate identification of deep similarity of test paper content by constructing a multi-granularity semantic comparison model and a dynamic threshold judgment mechanism, effectively overcoming misjudgment caused by factors such as naming differences and layout changes; 2. This invention introduces a unified virtual test paper identifier (VTID) mechanism to group test papers with consistent content into the same logical entity, thus ensuring the consistency of the data view; 3. This invention ensures data integrity through metadata fusion; 4. This invention can significantly reduce the redundancy rate of the question bank, improve resource utilization and the accuracy of teaching data analysis, and provide a high-quality data foundation for intelligent test paper generation, personalized practice and teaching quality assessment in the K12 education tablet scenario. Detailed Implementation

[0015] The present invention will be further described below. It should be noted that this embodiment is based on the present technical solution and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to this embodiment.

[0016] This embodiment provides a method for test paper deduplication and data fusion, including the following steps: S1. Obtain the original content data of the test papers to be included in the question bank; S2. Perform structured parsing on the original content data of the test paper to be processed, extract the question sequence list, the question stem text of each question, the option set, the correct answer identifier, the knowledge point tags and the layout features, and form the test paper to be processed into a structured test paper vector. S3. Input the structured test paper vector obtained in step S2 into the multi-granularity semantic comparison model, and calculate its question-by-question similarity score and whole-paper similarity score with each existing test paper in the question bank. S4. Based on the question-by-question similarity score and the whole-paper similarity score calculated in step S3, a dynamic threshold determination mechanism is used to determine whether the test paper to be processed is a duplicate test paper. S5. If the test paper is determined to be a duplicate, the metadata fusion process is initiated to merge and calibrate the source institution, upload time, and number of references of the test paper to be processed with the metadata of the corresponding existing test papers, and to update the global statistical data. S6. After metadata fusion, assign a unified Virtual Test ID (VTID) to the test papers to be processed and the corresponding existing books, so as to realize the logical normalization management of test papers with consistent content but different titles.

[0017] In this embodiment, the specific process by which the multi-granularity semantic comparison model calculates the question-by-question similarity score and the whole-paper similarity score between the test paper to be processed and a certain existing test paper in the question bank is as follows: (1) Calculation of question-by-question similarity score: (1.1) Using the BERT model pre-trained in the field of education, the question stem text of each question in the test paper to be processed and the existing test paper is converted into a high-dimensional vector (i.e. "digital password"). (1.2) Calculate the semantic similarity between each question in the test paper to be processed and all questions in the existing test paper, construct a similarity matrix, and find the question pairing combination that results in the highest overall matching score by using the optimal matching strategy (such as the Hungarian algorithm); then calculate the cosine similarity between the high-dimensional vectors of the two paired questions to obtain a semantic similarity value in the range [0,1]. For example, "Xiaoming bought 3 apples" and "Xiaohong bought three fruits" have different wording, but because their meanings are similar, their cosine similarity is high (e.g., 0.87). This step is the core of question-by-question comparison and solves the problem of missed judgment caused by differences in expression in traditional text comparison.

[0018] (1.3) The question-by-question similarity score between the two questions is calculated using a weighted fusion method: Question-by-question similarity score = w1 × semantic similarity + w2 × option set similarity + w3 × consistency of correct answer label Among them, the weights w1>w2≈w3 highlight the dominant semantic position.

[0019] (1.4) Calculate the average of the question-by-question similarity scores for the test paper to be processed and the existing test papers.

[0020] (2) Calculation of the whole-volume similarity score: The overall similarity score is a comprehensive assessment of whether two test papers are duplicates. It considers not only the similarity of individual questions but also the consistency of the test papers' organizational structure and topological features. The specific process is as follows: (2.1) Construction of the relational graph of test questions and analysis of topological similarity: Each test paper is modeled as a question relationship graph, with nodes representing questions and edges representing logical or sequential relationships between questions. Then, graph edit distance, sequence alignment algorithms (such as DTW), or graph neural networks (GNN) are used to quantify the similarity between the question relationship graph of the test paper to be processed and the existing test papers, and a structural similarity score S_struct is output.

[0021] (2.2) Generating content digest fingerprints based on the Locality Sensitive Hash (LSH) algorithm: Based on the LSH algorithm, a compact content summary fingerprint is generated for the entire exam paper, which can quickly reflect its main content structure. The exam paper to be processed is compared with the content summary fingerprints of all existing exam papers in the question bank, and the fingerprint similarity (Slsh) between the exam paper to be processed and all existing exam papers in the question bank is calculated. If the hash collision rate is high, there is a high probability of content overlap. The content summary fingerprint can be used for rapid initial screening to exclude obviously irrelevant exam papers and improve efficiency.

[0022] (2.3) Fusion calculation of whole-volume similarity Combining semantic, structural, and summary information, a weighted fusion strategy is used to calculate the final volume-level similarity score: Overall similarity score = α × average question-by-question similarity score + β × S_struct + γ × Slsh Among them, α, β, and γ are adjustable weights that can be automatically adjusted according to the subject and grade (e.g., humanities subjects emphasize semantics, while science subjects emphasize structure).

[0023] The overall similarity score is used to determine whether two test papers are duplicates with identical content.

[0024] In this embodiment, in step S4, the dynamic threshold determination mechanism specifically adjusts the determination threshold automatically according to the subject, grade level, and question type of the test paper. The dynamic threshold is calculated based on the basic similarity threshold T0, the complexity factor g of the test paper to be processed, and the noise tolerance factor f. T = T0 × (1 + g - f); Among them, the complexity factor g of the test paper to be processed is determined based on the number of questions, the proportion of subjective questions, and the density of mixed text and graphics; the noise tolerance factor f is dynamically adjusted based on the OCR recognition confidence and the historical accuracy records of the source institution.

[0025] The dynamic threshold mechanism automatically adjusts the tightness of the plagiarism detection standard based on the characteristics of different exam papers. The basic standard (T0) is equivalent to an initial similarity score line (e.g., 85 points), which is a common starting score for all exam papers. The exam paper complexity factor g represents the difficulty bonus (+g). If the exam paper has many questions, many subjective questions, or complex images and formulas (e.g., high school physics exam), the standard is automatically increased (g increases) because such exam papers are prone to "question overlap". The noise tolerance factor f represents the error reduction (-f). If the exam paper has poor scanning quality (blurred OCR recognition) or the question-setting organization has a history of errors, the standard needs to be appropriately relaxed (f increases). Therefore, the dynamic threshold = basic score × (1 + difficulty bonus - error reduction).

[0026] In step S5, the metadata fusion process is as follows: The system prioritizes duplicate exam papers between those awaiting processing and those already stored, determining which version is the master version and which is the non-master version based on the source institution's level, teacher feedback, and usage frequency. It then synchronizes valid metadata increments from the non-master version to the master version, including new citation records, supplementary knowledge point tags, and user error correction suggestions. For conflicting metadata, it employs a timestamp-first or voting mechanism to determine the appropriate value, ensuring the consistency and integrity of the merged metadata.

[0027] The aforementioned metadata fusion process addresses the issue of which version to use as the baseline when merging metadata from duplicate exam papers. Based on the source institution's level, teacher feedback, and usage frequency, a more authoritative version can be determined and used as the master version. Specifically, the source institution's level, teacher feedback, and usage frequency can be weighted to obtain a quantified value for authority priority. If different exam paper versions have contradictory annotations for the same question (e.g., exam paper A annotates a question as "trigonometric functions," while exam paper B annotates the same question as "plane geometry"), the annotation closer to the current time can be adopted if timestamp priority is used, or the majority opinion should be adopted if a voting mechanism is used; for example, if 80% of schools annotate the question as "trigonometric functions," then the annotation of "trigonometric functions" will be adopted.

[0028] In this embodiment, in step S6, the Unified Virtual Test Identifier (VTID) is generated using an irreversible hash function combined with the educational region code. The generation formula is as follows: VTID = H(Sorted(QuestionID_List)) || Region_Code || Subject_Grade_Code; Where H() is the SHA-256 hash function, Sorted() is the sorting function, QuestionID_List is the list of unique IDs for all questions, and Sorted(QuestionID_List) means sorting the list of unique IDs for all questions according to the numerical value of the unique IDs; Region_Code represents the code of the main region for which the test paper is applicable, and Subject_Grade_Code represents the combination code of subject and grade; the || symbol represents string concatenation, which is directly connected without adding a separator.

[0029] All test papers with the same VTID are treated as the same logical entity in the question bank, sharing a unified data dashboard and analysis reports.

[0030] VTID is like issuing a unified "digital ID card" to duplicate test papers with the same content. No matter how the test paper title or format changes, as long as the question combination is consistent, they will be automatically classified into the same category, which facilitates unified management and analysis.

[0031] In the above VTID generation formula, H() represents the SHA-256 hash function, which turns the list of unique IDs for all questions into a fixed-length garbled string (such as "a1b2c3...") to ensure that it cannot be reverse-engineered; Sorted(QuestionID_List) means that all unique IDs for all questions are sorted in ascending order to prevent different VTIDs from being generated for test papers with the same content due to different question orders. Region_Code indicates the region to which the test paper is applicable. For example, "BJ" represents Beijing, "SH-A" represents Shanghai Region A, and the code for Chaoyang District in Beijing might be "BJ-CY". Subject_Grade_Code is a combination code for the subject and grade, such as "MA3" for third-grade mathematics, "EN6" for sixth-grade English, and fifth-grade Chinese might be coded as "CN5".

[0032] In this embodiment, in step S4, when it is determined that the test paper to be processed is a duplicate of an existing test paper, a warning is issued to the uploader, the corresponding existing test paper is displayed, and the uploader is provided with three processing paths: forced upload, merging into an existing test paper, or creating a revision branch. When the uploader chooses forced upload, the test paper to be processed is uploaded to the question bank as an independent test paper, suitable for situations where there are indeed significant changes. When the uploader chooses to merge into an existing test paper, steps S5 and S6 are executed. When the uploader chooses to create a revision branch, the test paper to be processed is uploaded to the question bank and a version number is generated for it. Through the above process, the human-machine collaborative decision-making capability can be preserved.

[0033] For those skilled in the art, various corresponding changes and modifications can be made based on the above technical solutions and concepts, and all such changes and modifications should be included within the protection scope of the claims of this invention.

Claims

1. A method for test paper deduplication and data fusion, characterized in that, Includes the following steps: S1. Obtain the original content data of the test papers to be included in the question bank; S2. Perform structured parsing on the original content data of the test paper to be processed, extract the question sequence list, the question stem text of each question, the option set, the correct answer identifier, the knowledge point tags and the layout features, and form the test paper to be processed into a structured test paper vector. S3. Input the structured test paper vector obtained in step S2 into the multi-granularity semantic comparison model, and calculate its question-by-question similarity score and whole-paper similarity score with each existing test paper in the question bank. S4. Based on the question-by-question similarity score and the whole-paper similarity score calculated in step S3, a dynamic threshold determination mechanism is used to determine whether the test paper to be processed is a duplicate test paper. S5. If the test paper is determined to be a duplicate, the metadata fusion process is initiated to merge and calibrate the source institution, upload time, and number of references of the test paper to be processed with the metadata of the corresponding existing test papers, and to update the global statistical data. S6. After metadata fusion, a unified virtual exam paper identifier (VTID) is assigned to the exam papers to be processed and the corresponding existing books, so as to realize the logical normalization management of exam papers with consistent content but different titles.

2. The method according to claim 1, characterized in that, In step S2, the specific process by which the multi-granularity semantic comparison model calculates the question-by-question similarity score and the whole-paper similarity score between the test paper to be processed and a certain existing test paper in the question bank is as follows: (1) Calculation of question-by-question similarity score: (1.1) Using the BERT model pre-trained in the field of education, the question stem text of each question in the test paper to be processed and the existing test paper is converted into a high-dimensional vector; (1.2) Calculate the semantic similarity between each question in the test paper to be processed and all questions in the existing test paper, construct a similarity matrix, and find the question pairing combination that makes the overall matching score the highest by using the optimal matching strategy; then calculate the cosine similarity between the high-dimensional vectors of the two paired questions to obtain a semantic similarity value in the range of [0,1]. (1.3) The question-by-question similarity score between the two questions is calculated using a weighted fusion method: Question-by-question similarity score = w1 × semantic similarity + w2 × option set similarity + w3 × consistency of correct answer label; Among them, the weights w1 > w2 ≈ w3; (1.4) Calculate the average of the question-by-question similarity scores for the test paper to be processed and the existing test papers; (2) Calculation of the whole-volume similarity score: (2.1) Construction of the relational graph of test questions and analysis of topological similarity: Each test paper is modeled as a question relationship graph, with nodes representing questions and edges representing logical or sequential relationships between questions; then the similarity between the test paper to be processed and the question relationship graphs of existing test papers is quantified, and a structural similarity score S_struct is output; (2.2) Generating content digest fingerprints based on the locality-sensitive hashing algorithm: Based on the Locality Sensitive Hash (LSH) algorithm, a compact content summary fingerprint is generated for the entire test paper. Then, the test paper to be processed is compared with the content summary fingerprints of all existing test papers in the question bank, and the fingerprint similarity Slsh between the test paper to be processed and all existing test papers in the question bank is calculated. (2.3) Fusion calculation of whole-volume similarity: Overall similarity score = α × average question-by-question similarity score + β × S_struct + γ × Slsh; Where α, β, γ are adjustable weights; The overall similarity score is used to determine whether two test papers are duplicates with identical content.

3. The method according to claim 1, characterized in that, In step S4, the dynamic threshold determination mechanism specifically adjusts the determination threshold automatically based on the subject, grade level, and question type of the test paper. The dynamic threshold is calculated based on the basic similarity threshold T0, the complexity factor g of the test paper to be processed, and the noise tolerance factor f. T = T0 × (1 + g - f); Among them, the complexity factor g of the test paper to be processed is determined based on the number of questions, the proportion of subjective questions, and the density of mixed text and graphics; the noise tolerance factor f is dynamically adjusted based on the OCR recognition confidence and the historical accuracy records of the source institution.

4. The method according to claim 1, characterized in that, In step S5, the metadata fusion process is as follows: The system prioritizes duplicate exam papers between those awaiting processing and those already stored, determining which version is the master version and which is the non-master version based on the source institution's level, teacher feedback, and usage frequency. It then synchronizes valid metadata increments from the non-master version to the master version, including new citation records, supplementary knowledge point tags, and user error correction suggestions. For conflicting metadata, it employs a timestamp-first or voting mechanism to determine the appropriate value, ensuring the consistency and integrity of the merged metadata.

5. The method according to claim 1, characterized in that, In step S6, the unified virtual test paper identifier VTID is generated using an irreversible hash function combined with the education region code. The generation formula is as follows: VTID = H(Sorted(QuestionID_List)) || Region_Code || Subject_Grade_Code; Where H() is the SHA-256 hash function, Sorted() is the sorting function, QuestionID_List is the list of unique IDs for all questions, and Sorted(QuestionID_List) means sorting the list of unique IDs for all questions according to the numerical value of the unique IDs; Region_Code represents the code of the main region for which the test paper is applicable, and Subject_Grade_Code represents the combination code of subject and grade; the || symbol represents string concatenation, which is directly connected without adding a separator.

6. The method according to claim 1, characterized in that, In step S4, when it is determined that the test paper to be processed is a duplicate of an existing test paper, a warning is issued to the uploader, the corresponding existing test paper is displayed, and the uploader is provided with three processing paths: forced upload, merging into an existing test paper, or creating a revision branch. When the uploader chooses forced upload, the test paper to be processed is uploaded to the question bank as an independent test paper. When the uploader chooses to merge into an existing test paper, steps S5 and S6 are executed. When the uploader chooses to create a revision branch, the test paper to be processed is uploaded to the question bank and a version number is generated for it.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

8. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method of any one of claims 1-6.