A simplified question generation method and system based on a vector model and vector similarity retrieval

Abstract

The application discloses a kind of based on vector model and vector similarity retrieval's concise question generation method and system, first to the multi-modal test data of user upload is extracted and obtains query test text set, then the vector value of each query test text in query test text set is calculated using vector model generation method and obtains query test vector, then based on query test vector adopts vector similarity retrieval method in the test database of query test text set each query test text in query test text set is multi-modal retrieval and obtains query test comprehensive similarity score;The application effectively realizes the function that has using accurate semantic representation and similarity retrieval to repeat and test point similar test question is comprehensively identified and concise question set, not only greatly improves the identification accuracy of concise question, also reduces the labor cost of artificial screening redundant test question, simultaneously also improves user practice efficiency, reduces test question bank resource consumption.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for generating concise questions based on vector models and vector similarity retrieval. Background Technology

[0002] Against the backdrop of rapid development in educational informatization and online education, test questions, as the core carrier of knowledge assessment and skills training, have seen a continuous expansion in their reserves. Currently, test question banks built by educational institutions, enterprises, and individuals mainly come from two channels: first, user-uploaded test question resources, covering various types of questions accumulated in daily teaching and internal assessment scenarios; and second, publicly available test question resources obtained from public educational platforms and examination websites through web scraping technology. These two channels have jointly driven the rapid expansion of test question banks, but have also brought significant issues related to test question quality and management.

[0003] Currently, existing question banks generally suffer from high redundancy, leading to a large number of duplicate questions and questions with similar test points mixed together. This not only makes it easy for duplicate questions to exist in different forms within the same question bank due to subtle differences in question stem wording, option order adjustments, or different sources, but also makes it easy for similar questions with different question stem scenarios and question formats to test highly consistent core knowledge points and abilities. This redundancy problem causes learners to repeatedly encounter duplicate or similar questions during practice, resulting in wasted training time and failure to achieve comprehensive coverage and efficient consolidation of knowledge points, seriously affecting learners' learning outcomes. At the same time, redundant questions also consume a lot of server storage resources, requiring more manpower and time to manually screen questions during question bank updates, classifications, and maintenance, reducing the overall efficiency of question bank management. Therefore, it is necessary to design a simplified question generation method and system based on vector models and vector similarity retrieval. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and to better and more effectively address the problem of high redundancy in existing test question banks. This redundancy leads to a large number of duplicate questions and questions with similar test points mixed together. Not only do duplicate questions easily exist in different forms within the same test bank due to subtle differences in question stem wording, option order adjustments, or different sources, but similar questions with different question scenarios and question formats but highly consistent core knowledge points and ability requirements are also common. This redundancy problem causes learners to repeatedly encounter duplicate or similar questions during practice, resulting in wasted training time and hindering comprehensive coverage and efficient consolidation of knowledge points. This seriously affects learners' learning outcomes, and redundant questions also consume a large amount of server storage resources. This requires more manpower and time to manually screen questions during the updating, classification, and maintenance of the question bank, reducing the overall efficiency of question bank management. To address this issue, a simplified question generation method and system based on vector models and vector similarity retrieval is provided. This method achieves the function of comprehensively identifying and simplifying the question set by using accurate semantic representation and similarity retrieval. It not only significantly improves the accuracy of simplified question identification but also reduces the manpower cost of manually screening redundant questions, while also improving user practice efficiency and reducing the consumption of question bank resources.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A simplified question generation method based on vector models and vector similarity retrieval includes the following steps:

[0007] Step A: Extract the multimodal test question data uploaded by the user and obtain the set of query test question texts;

[0008] Step B: Calculate the vector value of each query question text in the query question text set using the vector model generation method and obtain the query question vector;

[0009] Step C: Based on the query question vector, the vector similarity retrieval method is used to perform multimodal retrieval on each query question text in the query question text set in the database to be retrieved and obtain the comprehensive similarity score of the query questions. Then, the query question texts are sorted in descending order according to the comprehensive similarity score of the query questions, and the query question texts with the highest ranking are selected to form a query question retrieval result list.

[0010] Step D: Based on the database of questions to be retrieved, set a standard threshold for the similarity of simplified questions, and then compare the standard threshold for the similarity of simplified questions with the comprehensive similarity score of the query question text in the query question retrieval results list to obtain the query question judgment result;

[0011] Step E involves storing the query question text set into and updating the question database to be retrieved, and then assembling the simplified questions from the question judgment results into a simplified question set and outputting it.

[0012] The aforementioned simplified question generation method based on vector model and vector similarity retrieval, in step A, extracts multimodal test question data uploaded by users and obtains a set of query question texts, wherein the multimodal test question data uploaded by users includes Word, Excel, PDF, TXT, WPS, WPT, PPT, JPEG, PNG, CSV, ET and JSON formats.

[0013] The aforementioned simplified question generation method based on vector models and vector similarity retrieval, step B, involves calculating the vector value of each query question text in the query question text set using the vector model generation method to obtain the query question vector. The specific steps are as follows.

[0014] Step B1: Use the BERT tokenizer to segment the query text and generate a token sequence containing CLS and SEP tags;

[0015] Step B2 involves truncating or padding the token sequence using a set maximum sequence length (maxlen) to generate a fixed-length token ID sequence. The maximum sequence length (maxlen) can be adjusted as needed for different lengths of multiple-choice questions, fill-in-the-blank questions, short-answer questions, and comprehensive questions.

[0016] Step B3: Input the token ID sequence into the BERT segmenter, wherein the BERT segmenter specifically maps the token ID sequence into an initial vector, and then obtains the hidden state matrix through a self-attention mechanism and a feedforward neural network;

[0017] Step B4: Select a feature extraction strategy based on the text length to extract the core vector of the hidden state matrix. The feature extraction strategy uses word embedding model and bag-of-words model to dynamically adapt to test content of different lengths.

[0018] Step B5: Perform L2 normalization on the core vector of the hidden state matrix to obtain the query question vector with a modulus of 1, as shown in formula (1).

[0019] (1)

[0020] in, To query the question vector, This is the core vector of the hidden state matrix.

[0021] The aforementioned method for generating concise questions based on vector models and vector similarity retrieval, specifically the truncation or padding process in step B2, is as follows:

[0022] Step B21: If the length of the query question text is greater than the maximum sequence length maxlen, then truncate the query question text from the end of the query question text.

[0023] Step B22: If the length of the query question text is less than the maximum sequence length maxlen, then use PAD padding markers to pad the end of the query question text to maxlen and form a token ID sequence.

[0024] The aforementioned method for generating concise questions based on vector models and vector similarity retrieval, specifically the text length selection feature extraction strategy in step B4, involves the following steps:

[0025] Step B41: If the length of the query question text is no more than 512 tokens, then extract the row vector corresponding to the CLS token;

[0026] Step B42: If the length of the query question text is greater than 512 tokens, then extract the average value of the row vectors corresponding to all CLS tokens.

[0027] The aforementioned simplified question generation method based on vector models and vector similarity retrieval, in step C, involves using vector similarity retrieval to perform multimodal retrieval on each query question text in the query question text set within the database of questions to be retrieved, based on the query question vector, and obtaining a comprehensive similarity score for each query question. Then, the query question texts are sorted in descending order according to the comprehensive similarity score, and the top-ranked query question texts are selected to form a query question retrieval result list. The specific steps are as follows.

[0028] Step C1: For each question text to be retrieved in the question database... Using the BGE-M3 model Encode and obtain the first encoded vector ;

[0029] Step C2, query the test question text. Using the BGE-M3 model Encode and obtain the second encoded vector ;

[0030] Step C3, based on the first encoding vector Second encoding vector Perform multimodal retrieval and similarity score calculation to obtain the overall similarity score of the query questions;

[0031] Step C4: Sort the query question texts in descending order based on the comprehensive similarity score of the query questions, and select the top-ranked query question texts to form a query question retrieval result list. The proportion of the top-ranked texts can be adjusted as needed.

[0032] The aforementioned simplified question generation method based on vector models and vector similarity retrieval, specifically the multimodal retrieval and similarity score calculation steps in step C3, are as follows:

[0033] Step C31: Perform dense retrieval and obtain an initial candidate text set, wherein the dense retrieval specifically involves calculating dense vector similarity, as shown in formula (2).

[0034] (2)

[0035] in, For dense vector similarity, To retrieve the test question text Dense vectors, For the text of the test questions to be retrieved Dense vectors;

[0036] Step C32 involves performing sparse retrieval based on the initial candidate text set. Specifically, sparse retrieval involves calculating sparse vector similarity, as shown in formula (3).

[0037] (3)

[0038] in, For sparse vector similarity, To retrieve the test question text sparse vectors, For the text of the test questions to be retrieved sparse vectors, Let be the value in the i-th dimension;

[0039] Step C33: The initial candidate text set is reordered to obtain a rearranged candidate text set, and then multi-vector retrieval is performed based on the rearranged candidate text set. Specifically, the multi-vector retrieval involves calculating multi-vector similarity, as shown in formula (4).

[0040] (4)

[0041] in, For multi-vector similarity, To retrieve the test question text Multivectors, For the text of the test questions to be retrieved Multivectors, In the first Values ​​in each dimension;

[0042] Step C34 involves weighted fusion of dense vector similarity, sparse vector similarity, and multi-vector similarity to obtain the comprehensive similarity score of the query questions, as shown in formula (5).

[0043] (5)

[0044] in, To query the overall similarity score of test questions, For intensive retrieval weights, For sparse retrieval weights, Weights for multi-vector retrieval.

[0045] The aforementioned method for generating concise questions based on vector models and vector similarity retrieval includes step D, which involves setting a standard threshold for the similarity of concise questions based on the database of questions to be retrieved. Then, the standard threshold is compared sequentially with the comprehensive similarity score of the query question text in the query question retrieval results list to obtain the query question determination result. This query question determination result includes both concise and non-concise questions. The specific steps are as follows.

[0046] Step D1: If the similarity standard threshold of the simplified question is greater than the comprehensive similarity score of the query question text, then the query question text is determined to be a simplified question.

[0047] Step D2: If the similarity standard threshold of the simplified question is less than the overall similarity score of the query question text, then the query question text is determined to be a non-simplified question.

[0048] A simplified question generation system based on vector models and vector similarity retrieval includes a question extraction module, a question vector calculation module, a question similarity calculation module, a question judgment module, and a question output module. The question extraction module extracts multimodal question data uploaded by the user and obtains a set of query question texts. The question vector calculation module calculates the vector value of each query question text in the query question text set using a vector model generation method to obtain a query question vector. The question similarity calculation module uses vector similarity retrieval to perform a similarity retrieval on each query question text in the query question text set within the question database. The system performs multimodal retrieval and obtains the comprehensive similarity score of the query questions. Then, it sorts the query question texts in descending order based on the comprehensive similarity score and selects the top-ranked query question texts to form a query question retrieval result list. The question determination module sets a simplified question similarity standard threshold based on the database of questions to be retrieved, and then compares the simplified question similarity standard threshold with the comprehensive similarity score of the query question texts in the query question retrieval result list to obtain the query question determination result. The question output module stores the query question text set into and updates the database of questions to be retrieved, and then assembles the simplified questions from the question determination result into a simplified question set and outputs it.

[0049] The beneficial effects of this invention are as follows: This invention provides a simplified question generation method and system based on vector models and vector similarity retrieval. First, it extracts multimodal test question data uploaded by the user to obtain a set of query question texts. Then, it uses a vector model generation method to calculate the vector value of each query question text in the query question text set to obtain a query question vector. Next, based on the query question vector, it uses a vector similarity retrieval method to perform multimodal retrieval on each query question text in the query question text set in the database to be retrieved, obtaining a comprehensive similarity score for the query questions. Then, it sorts the query question texts in descending order according to the comprehensive similarity score and selects the top-ranked query question texts to form a query question retrieval result list. Subsequently, it sets a simplified question similarity standard threshold based on the database to be retrieved, and then compares the simplified question similarity standard threshold with the comprehensive similarity score of the query question texts in the query question retrieval result list to obtain the query question determination result. Finally, it retrieves the query question text... This set stores and updates the database of questions to be retrieved, and then assembles the simplified questions from the question judgment results into a simplified question set and outputs it. This effectively realizes that the simplified question generation method and system can comprehensively identify and simplify the question set by using accurate semantic representation and similarity retrieval. Moreover, by combining the BERT word segmenter with the BGE-M3 model, it can perform deep semantic representation of the questions and break through the limitations of traditional keyword matching, thereby accurately identifying redundant questions with different stems but the same test points. At the same time, by combining vector database indexing and multimodal retrieval, the similarity retrieval time of the million-level question database can be shortened to the second level. By setting similarity thresholds, the number of returned results and retrieval weights, it can be dynamically adjusted according to different subjects and different exam types to meet the customized simplification needs of different users such as educational institutions, enterprises and individuals. This not only greatly improves the recognition accuracy of simplified questions, but also reduces the labor cost of manually screening redundant questions, while also improving user practice efficiency and reducing the consumption of question database resources. Attached Figure Description

[0050] Figure 1 This is an overall flowchart of a simplified question generation method based on vector model and vector similarity retrieval according to the present invention;

[0051] Figure 2 This is a schematic diagram illustrating the principle of a simplified question generation system based on vector models and vector similarity retrieval, according to the present invention.

[0052] Figure 3 This is a flowchart of the vector model generation method of the present invention.

[0053] Figure 4 This is a flowchart of the multimodal retrieval process of the present invention. Detailed Implementation

[0054] The present invention will now be further described with reference to the accompanying drawings.

[0055] like Figure 1 As shown, the present invention provides a method for generating concise questions based on vector models and vector similarity retrieval, comprising the following steps:

[0056] Step A: Extract the multimodal test data uploaded by the user and obtain a set of query test texts. The multimodal test data uploaded by the user can be in the formats of Word, Excel, PDF, TXT, WPS, WPT, PPT, JPEG, PNG, CSV, ET and JSON.

[0057] The file size should not exceed 100MB and the number of pages should not exceed 200.

[0058] like Figure 3 As shown, step B involves using a vector model generation method to calculate the vector value of each query question text in the query question text set and obtaining the query question vector. The specific steps are as follows:

[0059] Step B1: Use the BERT tokenizer to segment the query text and generate a token sequence containing CLS and SEP tags;

[0060] Step B2 involves truncating or padding the token sequence using a set maximum sequence length (maxlen) to generate a fixed-length token ID sequence. The maximum sequence length (maxlen) can be adjusted as needed for different question types, such as multiple-choice, fill-in-the-blank, short-answer, and comprehensive questions. The specific steps are as follows.

[0061] Step B21: If the length of the query question text is greater than the maximum sequence length maxlen, then truncate the query question text from the end of the query question text.

[0062] Step B22: If the length of the query question text is less than the maximum sequence length maxlen, then use PAD padding markers to pad the end of the query question text to maxlen and form a token ID sequence.

[0063] The maxlen setting is 8192 tokens.

[0064] Step B3: Input the token ID sequence into the BERT segmenter, wherein the BERT segmenter specifically maps the token ID sequence into an initial vector, and then obtains the hidden state matrix through a self-attention mechanism and a feedforward neural network;

[0065] Step B4: Based on the text length, select a feature extraction strategy to extract the core vector of the hidden state matrix. The feature extraction strategy uses word embedding model and bag-of-words model to dynamically adapt to test content of different lengths. The specific steps are as follows:

[0066] Step B41: If the length of the query question text is no more than 512 tokens, then extract the row vector corresponding to the CLS token;

[0067] Step B42: If the length of the query question text is greater than 512 tokens, then extract the average value of the row vectors corresponding to all CLS tokens.

[0068] Step B5: Perform L2 normalization on the core vector of the hidden state matrix to obtain the query question vector with a modulus of 1, as shown in formula (1).

[0069] (1)

[0070] in, To query the question vector, This is the core vector of the hidden state matrix.

[0071] like Figure 4 As shown, in step C, based on the query question vector, the vector similarity retrieval method is used to perform multimodal retrieval on each query question text in the query question text set in the database to be retrieved, and obtain the comprehensive similarity score of the query questions. Then, the query question texts are sorted in descending order according to the comprehensive similarity score of the query questions, and the query question texts with the highest ranking are selected to form the query question retrieval result list. The specific steps are as follows.

[0072] Step C1: For each question text to be retrieved in the question database... Using the BGE-M3 model Encode and obtain the first encoded vector ;

[0073] Step C2, query the test question text. Using the BGE-M3 model Encode and obtain the second encoded vector ;

[0074] Step C3, based on the first encoding vector Second encoding vector The specific steps for performing multimodal retrieval, calculating similarity scores, and obtaining the overall similarity score of the query questions are as follows.

[0075] Step C31: Perform dense retrieval and obtain an initial candidate text set, wherein the dense retrieval specifically involves calculating dense vector similarity, as shown in formula (2).

[0076] (2)

[0077] in, For dense vector similarity, To retrieve the test question text Dense vectors, For the text of the test questions to be retrieved Dense vectors;

[0078] Step C32 involves performing sparse retrieval based on the initial candidate text set. Specifically, sparse retrieval involves calculating sparse vector similarity, as shown in formula (3).

[0079] (3)

[0080] in, For sparse vector similarity, To retrieve the test question text sparse vectors, For the text of the test questions to be retrieved sparse vectors, Let be the value in the i-th dimension;

[0081] Step C33: The initial candidate text set is reordered to obtain a rearranged candidate text set, and then multi-vector retrieval is performed based on the rearranged candidate text set. Specifically, the multi-vector retrieval involves calculating multi-vector similarity, as shown in formula (4).

[0082] (4)

[0083] in, For multi-vector similarity, To retrieve the test question text Multivectors, For the text of the test questions to be retrieved Multivectors, In the first Values ​​in each dimension;

[0084] Step C34 involves weighted fusion of dense vector similarity, sparse vector similarity, and multi-vector similarity to obtain the comprehensive similarity score of the query questions, as shown in formula (5).

[0085] (5)

[0086] in, To query the overall similarity score of test questions, For intensive retrieval weights, For sparse retrieval weights, Weights for multi-vector retrieval.

[0087] The range of values ​​for the overall similarity score of the query questions is [0,1].

[0088] Step C4: Sort the query question texts in descending order based on the comprehensive similarity score of the query questions, and select the top-ranked query question texts to form a query question retrieval result list. The proportion of the top-ranked texts can be adjusted as needed.

[0089] Step D involves setting a simplified question similarity standard threshold based on the database of questions to be retrieved. Then, the simplified question similarity standard threshold is compared sequentially with the overall similarity score of the query question text in the query question retrieval results list to obtain the query question determination result. This result includes both simplified and non-simplified questions. The specific steps are as follows.

[0090] The similarity threshold for simplified questions is 0.9.

[0091] Step D1: If the similarity standard threshold of the simplified question is greater than the comprehensive similarity score of the query question text, then the query question text is determined to be a simplified question.

[0092] Step D2: If the similarity standard threshold of the simplified question is less than the overall similarity score of the query question text, then the query question text is determined to be a non-simplified question.

[0093] Step E involves storing the query question text set into and updating the question database to be retrieved, and then assembling the simplified questions from the question judgment results into a simplified question set and outputting it.

[0094] like Figure 2As shown, a simplified question generation system based on vector models and vector similarity retrieval includes a question extraction module, a question vector calculation module, a question similarity calculation module, a question judgment module, and a question output module. The question extraction module extracts multimodal question data uploaded by the user and obtains a set of query question texts. The question vector calculation module calculates the vector value of each query question text in the query question text set using a vector model generation method to obtain a query question vector. The question similarity calculation module uses vector similarity retrieval to search the database for each query question text in the query question text set based on the query question vector. The system performs multimodal retrieval and obtains the comprehensive similarity score of the query questions. Then, it sorts the query question texts in descending order based on the comprehensive similarity score and selects the top-ranked query question texts to form a query question retrieval result list. The question determination module sets a simplified question similarity standard threshold based on the database of questions to be retrieved, and then compares the simplified question similarity standard threshold with the comprehensive similarity score of the query question texts in the query question retrieval result list to obtain the query question determination result. The question output module stores the query question text set into and updates the database of questions to be retrieved, and then assembles the simplified questions from the question determination result into a simplified question set and outputs it.

[0095] To better illustrate the effects of the present invention, a comparative embodiment of generating simplified questions using the method of the present invention and existing methods is presented below, as shown in Table 1.

[0096] Table 1. Comparison of the simplification principles of using the method of the present invention and existing methods for generating simplified questions;

[0097]

[0098] This embodiment's experimental record: Python was used to process the test questions, and the test question bank contained 1353 items. A final threshold was set to be less than 0.6, or the number of simplified questions was less than 35% of the total original question bank data. 1. After text deduplication and removal of interfering characters using regularization, 1164 texts remain. 2. After deduplication, the 1164 test questions have an initial vector similarity retrieval threshold set to 0.96. The test question set is traversed to generate test question vector values, which are then inserted into the vector database. After subsequent test questions generate vector values, they need to be compared with the previous test question set for vector similarity. If the similarity is higher than the specified threshold, the test questions are discarded and not inserted into the vector database. 3. Start the loop to calculate vector similarity retrieval. After each round of question set review, the threshold needs to be reduced by 0.02 to simplify the questions.

[0099] 4. Ultimately, because the number of streamlined questions was less than 35% of the original number, the streamlined question bank contained 381 questions. The test question bank log is as follows, with database ID: test_1, indicating the start of question bank stream simplification.

[0100] Database ID: test_1, similarity: 0.96, simplification time: 40.682995319366455;

[0101] The database ID is test_1, the similarity is 0.96, and the number of remaining questions is 1072.

[0102] Database ID: test_1, similarity: 0.94, simplification time: 37.839807987213135;

[0103] The database ID is test_1, the similarity is 0.94, and the number of remaining questions is 1000.

[0104] Database ID: test_1, similarity: 0.92, simplification time: 33.78770685195923;

[0105] The database ID is test_1, the similarity is 0.92, and the number of remaining questions is 929.

[0106] Database ID: test_1, similarity: 0.90, simplification time: 31.109954833984375;

[0107] The database ID is test_1, the similarity is 0.90, and the number of remaining questions is 865.

[0108] Database ID: test_1, Similarity: 0.88, Simplification time: 29.686406135559082

[0109] The database ID is test_1, the similarity is 0.88, and the number of remaining questions is 820.

[0110] Database ID: test_1, Similarity: 0.86, Simplification time: 27.643317222595215

[0111] The database ID is test_1, the similarity is 0.86, and the number of remaining questions is 757.

[0112] Database ID: test_1, Similarity: 0.84, Simplification time: 25.35301423072815

[0113] The database ID is test_1, the similarity is 0.84, and the number of remaining questions is 700.

[0114] Database ID: test_1, Similarity: 0.82, Simplification time: 23.299750089645386

[0115] The database ID is test_1, the similarity is 0.82, and the number of remaining questions is 608.

[0116] Database ID: test_1, Similarity: 0.80, Simplification time: 19.963558673858643 seconds

[0117] The database ID is test_1, the similarity is 0.80, and the number of remaining questions is 514.

[0118] Database ID: test_1, Similarity: 0.78, Simplification time: 16.942593812942505

[0119] The database ID is test_1, the similarity is 0.78, and the number of remaining questions is 441.

[0120] Database ID: test_1, Similarity: 0.76, Simplification time: 15.808372020721436

[0121] The database ID is test_1, the similarity is 0.76, and the number of remaining questions is 381.

[0122] paper_id:test_1 Simplified test questions successfully.

[0123] In summary, the present invention provides a simplified question generation method and system based on vector models and vector similarity retrieval. First, it extracts multimodal test question data uploaded by the user to obtain a set of query question texts. Then, it uses a vector model generation method to calculate the vector value of each query question text in the set and obtains a query question vector. Next, based on the query question vector, it uses a vector similarity retrieval method to perform multimodal retrieval on each query question text in the set of query question texts in the database to be retrieved and obtains a comprehensive similarity score for the query questions. Then, it sorts the query question texts in descending order according to the comprehensive similarity score and selects the top-ranked query question texts to form a query question retrieval result list. Subsequently, it sets a simplified question similarity standard threshold based on the database to be retrieved, and then compares the simplified question similarity standard threshold with the comprehensive similarity scores of the query question texts in the query question retrieval result list to obtain a query question determination result. Finally, it returns the query question text set to the database. The system combines and updates the database of questions to be retrieved, then assembles the simplified questions from the question judgment results into a simplified question set and outputs it. This effectively realizes that the simplified question generation method and system can comprehensively identify and simplify the question set by using accurate semantic representation and similarity retrieval. Furthermore, by combining the BERT word segmenter with the BGE-M3 model, it can perform deep semantic representation of the questions and break through the limitations of traditional keyword matching, thereby accurately identifying redundant questions with different stems but the same test points. At the same time, by combining vector database indexing and multimodal retrieval, the similarity retrieval time of the million-level question database can be shortened to the second level. By setting similarity thresholds, the number of returned results, and retrieval weights, it can be dynamically adjusted according to different subjects and different exam types to meet the customized simplification needs of different users such as educational institutions, enterprises, and individuals. This not only greatly improves the recognition accuracy of simplified questions, but also reduces the labor cost of manually screening redundant questions, while also improving user practice efficiency and reducing the consumption of question database resources.

[0124] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating a concise question based on a vector model and vector similarity search, characterized in that: Includes the following steps, Step A: Extract the multimodal test question data uploaded by the user and obtain a set of query test question texts. The multimodal test question data uploaded by the user includes Word, Excel, PDF, TXT, WPS, WPT, PPT, JPEG, PNG, CSV, ET and JSON formats. Step B involves using a vector model generation method to calculate the vector value of each query question text in the query question text set and obtaining the query question vector. The specific steps are as follows: Step B1: Use the BERT tokenizer to segment the query text and generate a token sequence containing CLS and SEP tags; Step B2 involves truncating or padding the token sequence using a set maximum sequence length (maxlen) to generate a fixed-length token ID sequence. The maximum sequence length (maxlen) can be adjusted as needed for different lengths of multiple-choice questions, fill-in-the-blank questions, short-answer questions, and comprehensive questions. Step B3: Input the token ID sequence into the BERT tokenizer, wherein the BERT tokenizer specifically maps the token ID sequence into an initial vector, and then obtains the hidden state matrix through a self-attention mechanism and a feedforward neural network; Step B4: Select a feature extraction strategy based on the text length to extract the core vector of the hidden state matrix. The feature extraction strategy uses word embedding model and bag-of-words model to dynamically adapt to test content of different lengths. Step B5: Perform L2 normalization on the core vector of the hidden state matrix to obtain the query question vector with a modulus of 1, as shown in formula (1). （1） wherein, is a query question vector, is a hidden state matrix core vector; Step C involves using vector similarity retrieval based on the query question vectors to perform multimodal retrieval on each query question text in the query question text set within the database of questions to be retrieved, obtaining a comprehensive similarity score for each query question. Then, the query question texts are sorted in descending order according to the comprehensive similarity score, and the top-ranked query question texts are selected to form a query question retrieval result list. The specific steps are as follows. Step C1: For each question text to be retrieved in the question database... Using the BGE-M3 model Encode and obtain the first encoded vector ; Step C2, query the test text Using the BGE-M3 model Encoding and obtaining a second encoding vector ; Step C3, performing multi-modal search and similarity score calculation according to the first encoding vector and the second encoding vector and obtaining a comprehensive similarity score of the query test paper, wherein the multi-modal search and similarity score calculation specifically comprises the following steps, Step C31: Perform dense retrieval and obtain an initial candidate text set, wherein the dense retrieval specifically involves calculating dense vector similarity, as shown in formula (2). （2） wherein, is a dense vector similarity, is a dense vector of the query test text , is a dense vector of the test text to be retrieved . Step C32 involves performing sparse retrieval based on the initial candidate text set. Specifically, sparse retrieval involves calculating sparse vector similarity, as shown in formula (3). （3） in, For sparse vector similarity, To retrieve the test question text sparse vectors, For the text of the test questions to be retrieved sparse vectors, Let be the value in the i-th dimension; Step C33: The initial candidate text set is reordered to obtain a rearranged candidate text set, and then multi-vector retrieval is performed based on the rearranged candidate text set. Specifically, the multi-vector retrieval involves calculating multi-vector similarity, as shown in formula (4). （4） in, For multi-vector similarity, To retrieve the test question text Multivectors, For the text of the test questions to be retrieved Multivectors, In the first Values ​​in each dimension; Step C34 involves weighted fusion of dense vector similarity, sparse vector similarity, and multi-vector similarity to obtain the comprehensive similarity score of the query questions, as shown in formula (5). （5） in, To query the overall similarity score of test questions, For intensive retrieval weights, For sparse retrieval weights, Weights for multi-vector retrieval; Step C4: Sort the query question texts in descending order according to the comprehensive similarity score of the query questions, and select the query question texts with the highest ranking to form a query question retrieval result list. The proportion of the top-ranked texts can be adjusted as needed. Step D: Based on the database of questions to be retrieved, set a standard threshold for the similarity of simplified questions, and then compare the standard threshold for the similarity of simplified questions with the comprehensive similarity score of the query question text in the query question retrieval results list to obtain the query question judgment result; Step E involves storing the query question text set into and updating the question database to be retrieved, and then assembling the simplified questions from the question judgment results into a simplified question set and outputting it. 2.The simplified question generation method based on vector model and vector similarity search according to claim 1, characterized in that: The specific steps for truncation or padding in step B2 are as follows: Step B21: If the length of the query question text is greater than the maximum sequence length maxlen, then truncate the query question text from the end of the query question text. Step B22: If the length of the query question text is less than the maximum sequence length maxlen, then use PAD padding markers to pad the end of the query question text to maxlen and form a token ID sequence. 3.The simplified question generation method based on vector model and vector similarity search according to claim 2, characterized in that: The specific extraction steps for the text length selection feature extraction strategy in step B4 are as follows: Step B41: If the length of the query question text is no more than 512 tokens, then extract the row vector corresponding to the CLS token; Step B42: If the length of the query question text is greater than 512 tokens, then extract the average value of the row vectors corresponding to all CLS tokens.

4. The method of claim 3, wherein the method is characterized by: Step D involves setting a simplified question similarity standard threshold based on the database of questions to be retrieved. Then, the simplified question similarity standard threshold is compared sequentially with the overall similarity score of the query question text in the query question retrieval results list to obtain the query question determination result. This result includes both simplified and non-simplified questions. The specific steps are as follows. Step D1: If the similarity standard threshold of the simplified question is greater than the comprehensive similarity score of the query question text, then the query question text is determined to be a simplified question. Step D2: If the similarity standard threshold of the simplified question is less than the overall similarity score of the query question text, then the query question text is determined to be a non-simplified question.

5. A reduced question generation system based on vector model and vector similarity search, the reduced question generation system based on the reduced question generation method according to any one of claims 1-4, characterized in that: It includes a question extraction module, a question vector calculation module, a question similarity calculation module, a question judgment module, and a question output module. The question extraction module is used to extract multimodal question data uploaded by users and obtain a set of query question texts. The question vector calculation module is used to calculate the vector value of each question text in the question text set using a vector model generation method and obtain the question vector. The question similarity calculation module is used to perform multimodal retrieval on each query question text in the query question text set in the database of questions to be retrieved based on the query question vector using the vector similarity retrieval method, and obtain the comprehensive similarity score of the query questions. Then, the query question texts are sorted in descending order according to the comprehensive similarity score of the query questions, and the query question texts with the highest ranking are selected to form a query question retrieval result list. The test question determination module is used to set a simplified question similarity standard threshold based on the test question database to be retrieved, and then compare the simplified question similarity standard threshold with the comprehensive similarity score of the query question text in the query question retrieval result list to obtain the query question determination result; The test output module is used for storing and updating the query test text set into the test database to be searched, and outputting the simplified test set composed of the test determination results.

Images