A method for generating reading comprehension test questions based on a large language model and multi-agent system

Abstract

This application discloses a method for generating reading comprehension questions based on a large language model and multi-agent system. The process of generating a reading comprehension question can be divided into several stages. First, the RAG method is used to generate a question stem including the reading material and the question. After the question stem is generated, an option generation agent generates multiple sets of options based on the question stem. Two evaluation models are used to select the optimal option through a debate. Then, the optimal option is revised by revising the debate records obtained by the revision agent. Finally, the question-answering agent generates the answer to the question composed of the aforementioned question stem and the revised options. If the question-answering agent determines that the question is unreasonable based on the answer generated by the question-answering agent, the question is further revised, thereby improving the quality of reading comprehension question generation.

Description

Technical Field

[0001] This application relates to the field of text generation technology, and in particular to a method for generating reading comprehension test questions based on a large language model multi-agent system. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning technologies, increasingly intelligent solutions are emerging for generating reading comprehension test questions (multiple-choice questions).

[0003] In existing technologies, question generation techniques can be based on Retrieval-Augmented Generation (RAG). This involves retrieving relevant materials from a vector database and then combining this with a large language model to output complete reading comprehension questions. Specifically, an embedding model can be used to vectorize the text, retrieve relevant paragraphs as context, and then a pre-trained large language model can be used to generate a complete reading comprehension question.

[0004] While this method increases the diversity of questions, it still has certain problems:

[0005] The above approach suffers from several drawbacks during the RAG process. Firstly, the granularity of the retrieval is too coarse, leading to interference between the retrieved results. Secondly, after retrieving relevant materials for generating reading comprehension questions, directly generating complete reading comprehension questions using a single model may result in logical flaws, thus leading to low-quality generated reading comprehension questions.

[0006] There is currently no effective solution to the problem that the generated reading comprehension questions in the existing technologies may contain logical flaws, resulting in low or even poor quality reading comprehension questions. Summary of the Invention

[0007] The embodiments of this disclosure provide a method, apparatus, and storage medium for generating reading comprehension test questions based on a large language model multi-agent system, in order to at least solve the technical problem that the generated reading comprehension test questions in the prior art may have logical loopholes, thus resulting in low or even poor quality of the generated reading comprehension test questions.

[0008] According to one aspect of the present disclosure, a method for generating reading comprehension test questions based on a large language model and multiple agents is provided, comprising: obtaining keywords input by a user corresponding to a reading comprehension test question to be generated; generating a stem content corresponding to the reading comprehension test question based on the keywords using a RAG-based stem generation model, the stem content including reading material and questions; generating multiple sets of options corresponding to the questions based on the stem content using a multiple option generation agent, wherein the multiple sets of options are different; debating which set of options is optimal using two preset evaluation models, obtaining a debate record and the optimal option selected by the two evaluation models; revising the optimal option based on the debate record using a revision agent, obtaining a revised option; answering the question stem content and the revised option using a question-answering agent, obtaining an answer result; and generating the reading comprehension test question based on the stem content, the revised option, and the answer result.

[0009] According to another aspect of the present disclosure, a storage medium is also provided, the storage medium including a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0010] According to another aspect of the present disclosure, a reading comprehension test question generation device based on a large language model multi-agent is also provided, comprising: an acquisition module for acquiring keywords input by a user corresponding to the reading comprehension test question to be generated; a stem generation module for generating stem content corresponding to the reading comprehension test question based on the keywords using a RAG-based stem generation model, wherein the stem content includes reading material and questions; an option generation module for generating multiple sets of options based on the stem content using multiple option generation agents, wherein the multiple sets of options are different; a debate module for debating which set of options is optimal among the multiple sets of options using two preset evaluation models, obtaining a debate record and the optimal option selected by the two evaluation models; a revision module for revising the optimal option based on the debate record using a revision agent, obtaining a revised option; an answering module for answering the stem content and the revised option using a question-answering agent, obtaining an answering result; and a test question generation module for generating the reading comprehension test question based on the stem content, the revised option, and the answering result.

[0011] According to another aspect of the present disclosure, a reading comprehension test question generation device based on a large language model multi-agent is also provided, comprising: a processor; and a memory connected to the processor, configured to provide the processor with instructions to process the following steps: acquiring user-input keywords corresponding to a reading comprehension test question to be generated; generating a stem content corresponding to the reading comprehension test question based on the keywords using a RAG-based stem generation model, the stem content including reading material and questions; generating multiple sets of options corresponding to the questions based on the stem content using a multiple option generation agent, wherein the multiple sets of options are all different; debating which set of options is optimal using two preset evaluation models, obtaining a debate record and the optimal option selected by the two evaluation models; revising the optimal option based on the debate record using a revision agent, obtaining a revised option; answering the question based on the stem content and the revised option using a question-answering agent, obtaining an answer result; and generating the reading comprehension test question based on the stem content, the revised option, and the answer result.

[0012] In this embodiment, after generating the question stem using the RAG method, the generation of options and answers is divided into multiple stages (divergence stage, convergence stage, revision stage, practice stage, and backtracking stage), which are completed collaboratively by multiple agents. The generation of options and answers is based on the question stem, with multiple agents with different functions collaboratively evaluating and revising the options and answers to ensure the rationality of the final generated reading comprehension questions. Therefore, this method, through a multi-agent collaborative mechanism, combines the collaborative optimization of options and answers by the option generation agent, the evaluation model (also known as the "evaluation agent"), the revision agent, and the question-solving agent, improving the diversity and logical rigor of generated reading comprehension questions from multiple perspectives. This solves the problem described in the background art that "existing technologies may have logical loopholes in the generated reading comprehension questions, resulting in low or even poor quality of the generated reading comprehension questions." Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this application, illustrate exemplary embodiments of this disclosure and are used to explain this disclosure, but do not constitute an undue limitation of this disclosure. In the drawings:

[0014] Figure 1 This is a hardware structure block diagram of a computing device for implementing the method described in Embodiment 1 of this disclosure;

[0015] Figure 2This is a flowchart illustrating the reading comprehension test question generation method based on a large language model multi-agent according to the first aspect of Embodiment 1 of this disclosure;

[0016] Figure 3 This is a schematic diagram of a multi-stage, multi-agent collaborative process for generating reading comprehension test questions, provided in Embodiment 1 of this disclosure.

[0017] Figure 4 This is a flowchart illustrating a method for generating question stem content using the RAG method, as provided in Embodiment 1 of this disclosure.

[0018] Figure 5 This is a schematic diagram illustrating an example of a generated reading comprehension test question provided in Embodiment 1 of this disclosure;

[0019] Figure 6 This is a schematic diagram of a reading comprehension test question generation device based on a large language model and multi-agent according to the first aspect of Embodiment 2 of this disclosure; and

[0020] Figure 7 This is a schematic diagram of a reading comprehension test question generation device based on a large language model multi-agent according to the first aspect of Embodiment 3 of this disclosure. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Example 1

[0024] According to this embodiment, a method for generating reading comprehension test questions based on a large language model multi-agent is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] The method embodiments provided in this example can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Figure 1 A hardware block diagram of a computing device for implementing a reading comprehension test question generation method based on a large language model and multiple agents is shown. Figure 1 As shown, a computing device may include one or more processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, transmission device, and input / output interface are connected to the processor via a bus. In addition, it may also include a display, keyboard, and cursor control device connected to the input / output interface. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, a computing device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element in a computing device. As involved in the embodiments of this disclosure, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0027] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the reading comprehension test question generation method based on a large language model multi-agent in this embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the above-mentioned application method for generating reading comprehension test questions based on a large language model multi-agent. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the computing device via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the computing device's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0029] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of the computing device.

[0030] It should be noted here that, in some optional embodiments, the above... Figure 1 The computing device shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computing devices.

[0031] Under the aforementioned operating environment, according to the first aspect of this embodiment, a method for generating reading comprehension test questions based on a large language model multi-agent system is provided. This method can be used by... Figure 1 The computing device implementation is shown. Figure 2 A flowchart illustrating the method is shown below. (Refer to...) Figure 2 As shown, the method includes:

[0032] S202: Obtain the keywords input by the user that correspond to the reading comprehension questions to be generated;

[0033] S204: Based on keywords, generate question stems corresponding to the reading comprehension questions using a RAG-based question stem generation model. The question stems include the reading material and the questions.

[0034] S206: Generate an intelligent agent from multiple options. Based on the question stem, generate multiple sets of options that correspond to the question. Each set of options is different.

[0035] S208: Using two preset evaluation models, debate which set of options is the best among multiple sets of options, and obtain the debate record and the best option selected by the two evaluation models;

[0036] S210: The agent revises the optimal option based on the debate record to obtain the revised option;

[0037] S212: The intelligent agent responds to the question stem and the revised options to obtain the answer result; and

[0038] S214: Generate reading comprehension questions based on the question stem, the revised options, and the answers.

[0039] Figure 3 This is a schematic diagram of a multi-stage, multi-agent collaborative process for generating reading comprehension test questions, provided in Embodiment 1 of this disclosure.

[0040] Specifically, see Figure 3 As shown, when generating reading comprehension questions, the user inputs keywords corresponding to the questions into the computing device. The user can input keywords such as "history of space development." The computing device then obtains the keywords input by the user that correspond to the reading comprehension questions to be generated (S202).

[0041] Then, based on the keyword "history of aerospace development," the computing device generates the corresponding question stem content for the reading comprehension question using a RAG-based question stem generation model (S204). Specifically, the user can input keywords corresponding to the reading comprehension question to be generated through their terminal. The terminal sends the keywords to the computing device; the specific form of the terminal is not limited here. After receiving the keywords, the computing device will generate the corresponding question stem content for the reading comprehension question based on the keywords using RAG. The question stem content includes the reading material and the question (question stem generation stage).

[0042] Continue to refer to Figure 3As shown, after the computing device generates the question stem, it can generate an agent with multiple options. Based on the question stem, it generates multiple sets of options corresponding to the question, where each set of options is different (S206). Then, using two preset evaluation models, it debates which set of options is optimal, obtaining a debate record and the optimal option selected by the two evaluation models (S208). Then, the revision agent revises the optimal option based on the debate record, obtaining the revised option (S210). The question-answering agent answers the question stem and the revised option, obtaining the answer result (S212); and based on the question stem, the revised option, and the answer result, it generates reading comprehension questions (S214).

[0043] As described in the background section, existing methods for generating reading comprehension questions have the following technical problems: the retrieval granularity in the RAG process is too coarse, leading to mutual interference between the retrieval results; after retrieving relevant materials for generating reading comprehension questions, directly generating complete reading comprehension questions through a single model may result in logical flaws in the generated reading comprehension questions, thus leading to low quality of the generated reading comprehension questions.

[0044] In view of this, the technical solution of this application first generates the question stem (i.e., reading material and questions) of reading comprehension questions only through RAG, and then generates options and answers based on the question stem. Since this invention generates only the question stem content through RAG rather than the complete question (including question stem content, options, and answers), it solves the coarse-grained retrieval problem of existing technologies to a certain extent. Furthermore, in the process of generating options and answers, the generation of options and answers is divided into multiple stages (divergence stage, convergence stage, revision stage, practice stage, and backtracking stage), and is completed collaboratively by multiple intelligent agents. The generation of options and answers, based on the question stem content, involves multiple intelligent agents with different functions collaboratively evaluating and revising the options and answers, ensuring the rationality of the final generated reading comprehension questions. Therefore, this method utilizes a multi-agent collaboration mechanism, combining the option generation agent, the evaluation model (also known as the "evaluation agent"), the revision agent, and the question-answering agent to collaboratively optimize the options and test answers. This enhances the diversity and logical rigor of generated reading comprehension questions from multiple perspectives, thereby solving the problem described in the background art that "the generated reading comprehension questions in the prior art may contain logical loopholes, thus leading to low or even poor quality of the generated reading comprehension questions."

[0045] Optionally, the operation of generating the question stem content corresponding to the reading comprehension test question using a RAG-based question stem generation model based on the keyword includes: searching a preset vector database based on the keyword vector corresponding to the keyword to obtain the corresponding search results; filling the search results into a preset prompt word template to obtain reconstructed prompt words, wherein the prompt word template contains real reading comprehension test questions as demonstrations; and inputting the reconstructed prompt words into the question stem generation model to generate the aforementioned question stem content.

[0046] The computing device can first construct a corpus specifically for generating reading comprehension test questions, then construct the aforementioned vector database using the corpus, and subsequently use the RAG method to generate the question stems using the constructed vector database.

[0047] To ensure the reading materials in the question stem are grounded in reality rather than arbitrarily fabricated by the model, and to enhance the scalability and timeliness of the reading materials, a corpus is first constructed based on existing data crawling and processing methods. The specific steps are as follows: A large amount of relevant data is acquired through existing paper materials and web crawling; text cleaning: duplicate text data is removed using hash algorithms; irrelevant characters are removed; text formatting is standardized; long texts are segmented into several semantically continuous paragraph blocks based on paragraph characteristics, ensuring that each paragraph block is not too lengthy; each paragraph block is labeled with its topic, time, and source, forming the metadata in the corpus. Then, following the standard RAG method, vectorization is performed using an embedding model, and persistence is achieved using the FAISS vector library. Through this corpus, a vector database is constructed.

[0048] Then, when a user needs to generate reading comprehension questions (i.e., when the user inputs keywords), the computing device can use the vector database and the RAG method to generate the question stem.

[0049] in, Figure 4 This is a flowchart illustrating a method for generating question stems using the RAG method, as provided in Embodiment 1 of this disclosure. Figure 4 As can be seen, after obtaining the keywords input by the user, the computing device can determine the embedding vector corresponding to the keyword through an embedding model. Then, based on the determined embedding vector, the computing device queries a vector database to retrieve related paragraph blocks as retrieval material (retrieval results). Next, the computing device fills this retrieval material into a pre-set prompt word template to obtain reconstructed prompt words. Finally, the computing device inputs the prompt words into a question stem generation model to generate a question stem containing the reading material and the question.

[0050] For example, a user can input keywords such as "history of aerospace development," and the computing device can determine the embedding vector corresponding to the keyword "history of aerospace development" through an embedding model. The computing device then performs a search according to the RAG method, obtaining a search result such as "China's aerospace development originated from...". During this search process, the metadata of the text annotations also plays a role in fine-grained retrieval, ensuring that text blocks categorized as "aerospace" are accurately retrieved. Next, this method reconstructs the prompt words. The prompt word template can be: "Based on this material {search material}, integrate and generate a reading comprehension question, and provide a suitable question based on the material. The question material requirements are: ..., several demonstrations." The demonstrations can be selected from real reading comprehension questions; this few-shot prompting method helps the question stem generation model better understand the requirements. The reconstructed prompt words are then fed into the question stem generation model to generate the final question stem containing the reading material and the question.

[0051] It should be noted that the existing technologies described in the background section still suffer from coarse-grained retrieval problems. That is, traditional RAG question generation methods do not design retrieval strategies specifically for the characteristics of reading comprehension questions. They often simply use a whole retrieved document as a reference, which may contain a lot of irrelevant or distracting information. Furthermore, this method does not consider the influence of the retrieved documents on the reading material and answer choices in the reading comprehension questions; it merely generates complete reading comprehension questions based on these documents at a single time step, leading to a higher probability of abnormal model output. Additionally, the documents lack annotations for features such as topic and time.

[0052] Furthermore, in existing vector libraries, a single document is often a complete set of test papers, reading comprehension questions, etc., which are generated simultaneously from the question stem and options. This leads to coarse-grained retrieval problems: much of the retrieved content is actually not helpful for question generation, and may even interfere with or hinder question generation.

[0053] In this specification, the question stems (i.e., reading materials and questions) for reading comprehension tests are generated solely using RAG (Reference Aid Group). Since the metadata in the corpus used to generate the question stems is finely annotated, the vector database constructed based on this corpus also possesses the same fine granularity. This allows for fine-grained retrieval when searching the vector database based on embedded vectors, avoiding interfering results. Consequently, inputting the reconstructed prompt words from the retrieval results into the material generation model enables the generation of high-quality question stems, thus addressing, to some extent, the coarse-grained retrieval problem present in existing technologies.

[0054] The aforementioned corpus can be constructed as follows: Obtain 100,000 text entries related to reading comprehension from public educational platforms (e.g., public service platforms for educational resources) and authoritative publications (e.g., "Five-Year College Entrance Examination, Three-Year Simulation"). Text deduplication can be performed using the SimHash algorithm with a similarity threshold of 0.85, removing duplicate paragraphs. Regular expressions are used to remove HTML tags, redundant spaces, and special characters, and the text encoding is standardized to UTF-8. The spaCy tool is used to segment long texts by paragraph, ensuring each paragraph block is no longer than 500 characters. The DeepSeek-R1-671B model's API is called to categorize paragraph blocks by topic (e.g., "technology," "history," and "literature"), and timestamps (e.g., "2023") and source information are extracted using regular expressions to generate metadata. The final corpus contains nearly 500,000 annotated text entries, achieving 95% topic coverage and supporting dynamic updates.

[0055] Next, the computing device can encode paragraph blocks into 768-dimensional vectors using the Sentence-BERT embedding model, store them in the FAISS vector database, and associate them with metadata (including topic, time, and source). When a user enters a query, it is first vectorized using Sentence-BERT, and then the top-1 similar paragraph blocks are retrieved. The retrieved paragraph blocks are then input into a local question generation model (here, the Deepseek-R1-32B model deployed locally on ollama is used) to generate integrated reading materials and questions.

[0056] It should be noted that the models used to instantiate the option generation agent and the question-answering agent can be obtained by fine-tuning the pre-trained base model (i.e., the pre-trained large language model).

[0057] Optionally, before generating multiple sets of options corresponding to the question based on the question stem, the method further includes: instantiating an option generation model based on a pre-trained base model to obtain multiple option generation agents. The option generation model is obtained through the following operations: acquiring standard questions from a standard question set, which are obtained by converting real reading comprehension questions into structured data containing reading material fields, question fields, option fields, and question answer fields; generating corresponding option generation logic information based on the question stem content of the standard questions using a pre-defined large language model; the option generation logic information represents the thought process of generating corresponding options according to the question stem content in the standard questions; constructing an option generation training set based on the question stem content of the standard questions in the standard question set, the corresponding options, and the corresponding option generation logic information; and fine-tuning the pre-trained base model based on the option generation training set to obtain the option generation model.

[0058] Optionally, before obtaining the answer result by having the question-solving agent respond to the question stem and the revised options, the method further includes: instantiating the question-solving model based on the pre-trained basic model to obtain the question-solving agent, wherein the question-solving model is obtained through the following operations: obtaining standard questions from a standard question set; generating question-solving strategy information and test answers selected from the options corresponding to the standard questions based on the question stem content of the standard questions through a pre-set large language model; if the selected test answers match the test answers corresponding to the standard questions, constructing a question-solving training set based on the reading materials, questions, options, test answers, and question-solving strategy information corresponding to the standard questions; and fine-tuning the pre-trained basic model based on the question-solving training set to obtain the question-solving model.

[0059] The methods of obtaining the option generation model by fine-tuning the pre-trained basic model and instantiating the option generation agent, and obtaining the question-solving model by fine-tuning the pre-trained basic model and instantiating the question-solving agent are similar and will be explained together below.

[0060] This stage aims to build the training data needed for fine-tuning the models (i.e., the option generation model and the question-answering model mentioned above). First, computing devices can acquire a large number of reading comprehension questions through various channels and format them into files containing fields for reading materials, questions, options, and correct options (i.e., the test answer fields mentioned above). Then, a standard question set containing thousands of high-quality, representative standard questions can be constructed (e.g., through manual selection).

[0061] For the option generation training set, the input consists of the reading material and questions of standard questions in the standard question set. The annotations needed are the four generated options and the thought process behind their generation (i.e., the option generation logic information mentioned above). Therefore, for a training sample in the aforementioned option generation training set, the sample data includes the stem of a standard question, and the annotation information includes the options for that standard question and the option generation logic information.

[0062] For the practice set, the input is a complete standard question (without the answer key) from the standard question set. The annotation should include the correct answer to this standard question and the thought process involved in solving it. Therefore, for the options mentioned above, a training sample is generated in the training set. The sample data includes a standard question without the answer key, and the annotation information includes the answer key and the thought process behind solving the question.

[0063] Since manual annotation would be costly and of poor quality, automated annotation is used to generate training sets for both options and questions. This is accomplished using model APIs with analytical capabilities, such as Deepseek-R1 and GPT4-O1.

[0064] For the training set of option generation, the options that need to be labeled are actually the options already existing in the standard question set (which can be 4 options). However, the generation logic and process of the four options need to be labeled with the help of the model. Therefore, this method proposes a reverse thinking approach, that is, given the reading material, the question and the options, let the large language model work backward to think about how these four options are generated, thereby obtaining the option generation logic information. The labeling is completed before the options are even set up (that is, the options use the options from the standard question itself).

[0065] Similarly, for the question-solving training set, the part that needs to be annotated with the help of a large language model is the thought process for solving the questions. Here, a free-response-filtering scheme is proposed. Without any additional hints, the large language model first freely generates the thought process and selected answer for the standard question. Next, it determines whether the answer selected by the model is the actual answer in the standard question set. If so, the complete thought process of this free-response (i.e., the aforementioned thought process information) is retained as an annotation; if not, the model answers again without memory until it answers the question correctly, or reaches the maximum number of attempts, at which point the question is considered invalid.

[0066] This involves extracting 10,000 reading comprehension questions from a past exam question bank and formatting them into a JSON file containing "reading material, question, options, and correct option (answer)". 5,000 high-quality questions are manually selected and retained as the base dataset. The DeepSeek-R1-671B model's API is used to construct the thought process data for option generation and question answering (option generation training set and question answering training set) according to the designed question template. The training set is in ALPA format, a JSON file containing three fields: system, instruction, and output. The output field is annotated by DeepSeek-R1-671B.

[0067] After obtaining the MindChain dataset (the training set and problem-solving training set generated by the above options), a pre-trained base model can be selected (e.g., using Qwen2.5-32B-Instruct as the pre-trained base model). Supervised fine-tuning of the model can then be performed based on the existing Low-Rank Adaptation (LoRA) method. The purpose of supervised fine-tuning is to gradually make the model's output logic approximate the output of the dataset, given the input portion of the dataset.

[0068] Model weight updates use the standard LoRA method, with the forward propagation formula as follows. The original weights are frozen during model parameter updates. Only update the low-rank matrix with fewer parameters. and This will reduce training costs.

[0069]

[0070] The objective function for training can use the standard cross-entropy loss function. The training process follows the standard next-token-predict method, predicting only the token at position i at a time and providing the prediction probability p for the ground truth at that position. The ground truth value of the token at position i is then added to the context, and the model continues to predict the token at position i+1. This process is repeated until the prediction is complete, and the total loss is obtained for gradient descent.

[0071]

[0072] The LoRA adapter (rank r=8) can be loaded. Using the Transformers framework, the learning rate is set to 1e-4, the batch size to 4, and training is performed for 2 epochs. The loss function is cross-entropy loss, and the optimizer is AdamW. Training takes approximately 4 hours (8×4090D GPU). The LoRA weights (matrices B and A) are merged with the pre-trained base model to obtain the fine-tuned option generation model and question-solving model.

[0073] The following section provides a detailed explanation of the five stages mentioned above for collaboratively generating options and test answers. Please refer to [link / reference needed]. Figure 3 ,from Figure 3 As can be seen, the process of generating a reading comprehension question can be divided into several stages. First, the computing device generates the question stem, which includes the reading material and the question itself. After generating the question stem, the computing device uses an intelligent agent to generate multiple sets of corresponding options based on the question stem. Figure 3 In the divergent processing stage, the generated options are evaluated through a debate process using a judging model to find the optimal option. Figure 3 The middle stage is the convergence phase. Then, the optimal option is revised by revising the debate records obtained by the agent in the debate (in...). Figure 3 This is referred to as the revision phase), and then the question-solving agent determines the answer to the question consisting of the aforementioned question stem and the revised options (in the context of the revision phase). Figure 4 This is referred to as the practice stage. If there are unreasonable aspects to the question to be answered, the question to be answered will be further revised (in...). Figure 3This is referred to as the backtracking phase in Chinese. Below, we will introduce each phase in detail, starting with the divergence phase.

[0074] Optionally, an intelligent agent is generated through multiple options. Based on the question stem, the agent generates multiple sets of options corresponding to the question. Specifically, this includes: inputting the question stem into the intelligent agent that generates multiple options, and each intelligent agent that generates multiple options outputs a corresponding set of options and the corresponding question answer.

[0075] Specifically, refer to Figure 3 As shown, in the divergence phase: First, the fine-tuned option generation model is used as the option generation agent. To enhance the diversity of inference paths, multiple option generation agents are instantiated. Each option generation agent uses a sampling inference strategy instead of a greedy inference strategy, and sets high inference parameters (temperature, top-p, and top-k, etc.) to ensure that the outputs of each option generation agent are not identical.

[0076] After generating the question stem, the option generation agent receives the question stem and reconstructs it into the input paradigm for fine-tuning the option generation model. After deep thinking, the option generation agent outputs its thought process and final options for the problem. In this process, the inputs and actions of multiple option generation agents are completely consistent, but they produce different reasoning paths and diverse options.

[0077] Then, enter Figure 3 The convergence phase in this method aims to converge the multiple divergent options into the optimal set of options. In this method, a debate is used to determine the optimal set of options.

[0078] Optionally, the process involves debating which set of options is optimal among multiple options using two preset evaluation models, obtaining the debate record and the optimal option selected by the two evaluation models. Specifically, this includes: an initialization phase, in which the two evaluation models are divided into a positive evaluation model and a negative evaluation model, and the viewpoint information of the positive and negative evaluation models is determined. The viewpoint information of the positive evaluation model includes the optimal option selected by the positive evaluation model from multiple options and the corresponding reasons, while the viewpoint information of the negative evaluation model includes the optimal option selected by the negative evaluation model from multiple options and the corresponding reasons. The optimal option selected by the negative evaluation model is inconsistent with the optimal option selected by the positive evaluation model. During the debate phase, in each round of debate, the affirmative evaluation model, based on the statements made by the negative evaluation model in the previous round, determines the statements and corresponding confidence levels of both sides. The statements indicate whether the respective evaluation model agrees with the opposing viewpoint and maintains its own. After each round of debate, the statements from both evaluation models are stored to obtain a debate record. Furthermore, after each round, the confidence levels of the affirmative and negative evaluation models are determined. If the confidence levels differ, the debate ends, and the optimal option selected by the evaluation model with the higher confidence level is adopted as the optimal option selected by both evaluation models. If the confidence levels are the same, the next round of debate begins.

[0079] Specifically, we first instantiate two judging models (judging agents), which serve as the affirmative side (i.e., the affirmative judging model mentioned above) and the negative side (i.e., the negative judging model mentioned above) in this debate. The task for both sides is to select the optimal set of options from the multiple sets mentioned above (i.e., the optimal option mentioned above) and to debate the opposing viewpoints. For ease of description, we will use "affirmative" to represent the affirmative judging model and "negative" to represent the negative judging model below.

[0080] First, the viewpoints are initialized (initialization phase). The affirmative side, without any role restrictions, chooses the set of options it considers better and gives corresponding reasons. The negative side, on the other hand, is guided by prompts to become the role opposing the affirmative side's viewpoint, thus giving a different viewpoint, namely a different set of optimal options and corresponding reasons. As a result, the computing device will obtain two completely different viewpoints.

[0081] Next, the debate officially begins. The affirmative side will receive the previous round's statements from the negative side, and will be asked whether they agree with the negative side's viewpoint and whether they maintain their own. Similarly, the negative side will receive the previous round's statements from the affirmative side, and will be asked whether they agree with the affirmative side's viewpoint and whether they maintain their own, thus obtaining separate statements from both sides. If the two sides fail to reach an agreement in this round, the debate will proceed to the next round. After each round of debate, the statements from both sides will be stored according to their roles, for example, in the format of "Affirmative side's statement record: First round statement, Second round statement..., Negative side's statement record:...".

[0082] To conclude the debate, the concept of confidence analysis is introduced. Each evaluation model conducts a self-confidence assessment after each round of debate, thus obtaining the confidence levels of both sides. For example, confidence levels can be divided into four categories: (very confident, somewhat confident, uncertain, and incorrect). After a round of debate, if one side's confidence level is higher than the other side's, it indicates that the other side has been persuaded, the debate ends, and the set of options chosen by the more confident side (the side with the higher confidence level) is output as the optimal option. Otherwise, if both sides have the same confidence level, it indicates that there is still a disagreement, and the debate continues. To prevent the debate from going on indefinitely, a value k can be set as the maximum number of debate rounds. If the two sides still disagree after reaching the maximum number of debate rounds, the optimal option chosen by the affirmative side is defaulted to being the optimal option selected by both evaluation models.

[0083] Then, enter Figure 3 In the revision phase, the computing device revises the optimal option based on the debate record obtained in the above debate.

[0084] Optionally, the operation of revising the optimal option by the revising agent based on the debate record to obtain the revised option specifically includes: revising the optimal option by the revising agent based on the opposing opinions against the optimal option in the debate record to obtain the revised option.

[0085] In other words, the revision phase involves incorporating opposing viewpoints from the debate process to adjust the optimal option. The revision agent can absorb reasonable suggestions from these opposing viewpoints and revise the optimal options derived from the debate. For example, even if the affirmative side ultimately wins the debate, their viewpoint isn't necessarily "completely correct," and the negative side's statements aren't necessarily "worthless." Therefore, the revision agent can extract useful criticisms from the negative side's statements and revise the optimal option chosen by the affirmative side, resulting in a revised option. This revision aligns with human intuition.

[0086] After the computing device determines the revised options, it will proceed to... Figure 4 The process involves two phases: the implementation phase and the backtracking phase. Since the conclusions drawn in the implementation phase will be used to determine whether to proceed to the backtracking phase, both phases will be described together below.

[0087] Optionally, the operation of obtaining the answer result by having the question-solving agent answer the question stem and the revised options specifically includes: forming a question to be answered by the question stem and the revised options, and having the question-solving agent answer the question to be answered n times to obtain the corresponding n answer results. The process of the question-solving agent answering the question n times is independent and does not share context.

[0088] Optionally, based on the question stem, the revised options, and the answer results, a reading comprehension test is generated. Specifically, this includes: counting the most frequent answer among n answer results, determining whether the most frequent answer matches the answer corresponding to the revised option; if they match, generating a reading comprehension test based on the question to be answered and the most frequent answer; and if they do not match, revising the question to be answered through a revision agent to obtain a revised question, answering the revised question again through a question-answering agent to obtain a second answer result, and generating a reading comprehension test based on the second answer result and the revised question.

[0089] Implementation Phase: In this phase, the fine-tuned question-solving model is instantiated into a question-solving agent. Since this agent is trained on a large amount of real reading comprehension test data, it can be considered a highly capable test-taker in a real-world scenario. The question stem (reading material and options) and the revised options from the previous phase are treated as the problem (i.e., the question to be answered mentioned above), and reconstructed as the input paradigm for fine-tuning the question-solving model. After deep thinking, the agent provides what it considers the correct option, thus obtaining its answer. To prevent randomness, the agent performs the question (i.e., the question to be answered mentioned above) n times, each time independently without sharing context. Then, the most frequently occurring option (i.e., the most frequently occurring answer mentioned above) is counted among the n attempts. For example, if the agent answers 5 times, choosing option A 3 times, option B once, and option D once, then option A is chosen as its answer. At this point, it is determined whether the answers provided by the question-answering agent (i.e., the most frequently occurring answers mentioned above) are consistent with the correct options identified when the question was generated (i.e., the answers corresponding to the revised options mentioned above). If they are consistent, it means that high-level test takers can answer this question correctly most of the time, and this question can be used in practice. The question-generating task of each agent is completed, and the questions to be answered and the most frequently occurring answers mentioned above can be used as the generated reading comprehension questions.

[0090] If there is a discrepancy, it means that even high-achieving test-takers would likely be unable to answer this question correctly most of the time. This question might be obscure, unusual, or even logically flawed, making it an unreasonable question. Therefore, a backtracking phase is needed to revise and correct this question.

[0091] The correct option identified when the question was generated (i.e., the answer to the question corresponding to the revised option mentioned above) may refer to the answer to the question given by the option generating agent when generating the optimal option corresponding to the revised option, or it may refer to the answer to the question given by the revising agent when generating the revised option.

[0092] Backtracking Phase: To correct unreasonable questions that appeared during the practice phase, the computing device backtracks to the revision phase. While the revision agent remains unchanged, the input can change to the output of the question-solving agent, the complete question to be answered (which may include reading material, questions, and options), and the question-generating record (which may include the revision record of the revision agent and the option-generating record of the option-generating agent). The revision agent now faces a scenario similar to a confrontation between a high-level test-taker and a mid-level question-generating expert. Both sides provide their respective question-solving and question-generating records. The revision agent first synthesizes the information and then further revises the question to be answered. During this phase, the revision agent can not only modify the options but is also allowed to modify the reading material and questions to address the problem of unreasonable questions arising from the question stem rather than the options, thus obtaining the revised question.

[0093] Then, the AI ​​can continue to answer the revised questions. If the most frequent answer in the n answers given by the AI ​​is the same as the answer to the revised question, the revised question and its corresponding answer can be used as the generated reading comprehension question.

[0094] It should be noted that while the fine-tuned models (i.e., both the option generation model and the question-answering model mentioned above are fine-tuned models) maintain excellent performance in downstream tasks (i.e., tasks related to reading comprehension question generation), their performance declines in general tasks and instruction following ability. Therefore, in these five stages, only the option generation agent and the question-answering agent are instantiated using fine-tuned models. The other agents are instantiated using pre-trained large language models, which perform moderately well in downstream tasks but are stronger in general tasks. This design comprehensively considers the advantages of both the fine-tuned models and the pre-trained large language models while avoiding their respective disadvantages, achieving a complementary effect among multiple agents.

[0095] Furthermore, to ensure that each agent in a multi-agent system understands its responsibilities, separate system prompts need to be designed for the option-generating agent, the judging agent, the revising agent, and the question-solving agent. For example, the system prompt for the option-generating agent could be: "You are a reading comprehension question-generating expert, {several question-generating demonstrations}"; the system prompt for the question-solving agent could be: "You are a reading comprehension question-solving expert, {several question-solving demonstrations}". These demonstrations can be carefully designed manually, and the few-shot method has proven effective in this stage.

[0096] The following section introduces the specific implementation method of the above-mentioned five-stage multi-agent collaboration, in which the specific behavior of each agent can be guided by prompt words.

[0097] The agents are instantiated: the option generation agent and the question-answering agent are instantiated by fine-tuning the model. Since the fine-tuned model is in safetensor format under the transformers training framework, it needs to be converted to the gguf format required by OLLAMA using the llama.cpp tool. Then, 4-bit quantization is performed to complete the creation of the OLLAMA model. Afterwards, the ChatOpenAI and create_react_agent methods of Langgraph are used to instantiate the local OLLAMA fine-tuned model into agents. Due to resource constraints, only two option generation agents can be instantiated, with their inference parameters set to temperature=0.8, top-p=0.9, top-k=40; and temperature=0.7, top-p=0.8, top-k=30, respectively. For the evaluation agent and the revision agent, the Deepseek-R1-32B pre-trained model deployed locally in OLLAMA can be used, and it is also instantiated into agents using Langgraph. Finally, the multiple agents can be arranged according to a five-stage collaborative structure using the graph structure of Langgraph.

[0098] Debate records can be obtained by maintaining message queues. These message queues during the debate phase can be divided into two types: role message queues and debate message queues. Each judging model corresponds to one role message queue and one debate message queue. The role message queue only maintains all the statements made by either side during the debate. For example, the affirmative side's role message queue records all their statements from the first round, second round, etc. The debate message queue maintains the debate message records for a particular side, constructed by interleaving the role message queues. Specifically, a side's debate record includes not only their own statements but also the opposing side's rebuttals and statements in each round. For example, the structure of the affirmative side's debate message queue could be: task description instruction - affirmative side's initial viewpoint - negative side's statement - affirmative side's statement - negative side's statement... Similarly, the structure of the negative side's debate message queue could be: task description instruction - negative side's initial viewpoint - affirmative side's statement - negative side's statement - affirmative side's statement... Existing debate messages are treated as context, serving as the agent's memory of the debate. Thus, a complete debate record can be obtained through the messages in the role message queue and the debate message queue.

[0099] Since the debate against the optimal option is needed in the subsequent revision stage, it is possible to determine which side selected the optimal option in the initialization stage, identify the evaluation model corresponding to that side, determine the role message queue of the evaluation model that opposes that evaluation model, and identify the objections against the optimal option from that role message queue.

[0100] In other words, post-debate revision can begin by determining the winning viewpoint, taking the affirmative side as an example. Next, the entire message queue of the opposing side can be obtained, and the revision agent can extract point-by-point criticisms of the affirmative side's viewpoint (i.e., the aforementioned objections to the optimal option), without retaining other irrelevant information, such as the option group chosen by the opposing side. Finally, the revision agent, combining these criticisms, revises the "optimal" option group chosen by the affirmative side into a "better" option group, resulting in the revised option.

[0101] The input for the revision phase during backtracking is quite complex. To avoid the revision agent struggling to understand all the input, it can be mainly divided into three parts: question generation record, complete question, and answering record. The complete question is passed in as a JSON dictionary, with keys including the question stem, options (which can include four options: A, B, C, and D), and the answer. The answering record can contain the thought process information of the answering agent and the final selected option, also passed in a JSON format that is easy for the model to understand. The question generation record needs to be simplified and compressed. The revision agent first extracts a summary of the original question generation record (which may include the output of the option generation agent) and the revised record, and then reconstructs the latest question generation record based on the question generation logic of the revised summary. Finally, these three parts of input are embedded into a prompt word template, supplemented by some standard instructions, to obtain the revised question.

[0102] Figure 5 This is a schematic diagram illustrating an example of a generated reading comprehension test question provided in Embodiment 1 of this disclosure.

[0103] For a complete example of generating a reading comprehension question, please see [link / example]. Figure 5 Two agent-generating agents, based on the retrieved question stem (including reading material and questions), generated two distinct sets of options. The first set of options considered the core meaning of the material to be "resistance to traditional ethics," while the second set considered it to be "practicing natural instincts." The evaluation model then entered a debate phase, where both sides selected different options as the "optimal option" and offered rebuttals to each other's viewpoints. Because the opposing side had a higher confidence level than the affirmative side in the second round, the "optimal" option group converged to the second option group. In the revision phase, the revision agent fully utilized the affirmative side's criticisms (oppositional opinions) from the debate, incorporating "resistance" into option A and modifying the original option D. However, since the question-solving agent tended to choose option A (choosing A 3 out of 5 attempts), which contradicted the initial question-generating strategy, a backtracking revision phase was initiated. The revision agent directly selected the key sentence from the original material as the correct option, while slightly modifying previous options as distractors. The final version of the problem was given to the problem-solving AI, which solved it correctly in all five attempts, proving the problem's effectiveness. Furthermore, the three distractors each had their own unique characteristics, highlighting the problem's disruptive nature. The entire multi-agent collaborative process was progressive, with each stage's problem version optimization fully demonstrating the indispensability of each stage and each AI agent.

[0104] In addition, refer to Figure 1 As shown, according to a second aspect of this embodiment, a storage medium is provided. The storage medium includes a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0105] Therefore, according to this embodiment, the diversity and logical rigor of reading comprehension test question generation can be improved, the coarse-grained retrieval problem existing in the prior art can be solved, and the cost of generating training samples can be reduced by automatically annotating and generating training samples for fine-tuning the pre-trained base model.

[0106] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0108] Example 2

[0109] Figure 6 A reading comprehension test item generation apparatus 600 based on a large language model multi-agent according to the first aspect of this embodiment is shown, which corresponds to the method described according to the first aspect of Embodiment 1. Reference Figure 6As shown, the device 600 includes: an acquisition module 610 for acquiring keywords input by the user corresponding to the reading comprehension questions to be generated; a question stem generation module 620 for generating question stem content corresponding to the reading comprehension questions based on the keywords using a RAG-based question stem generation model, wherein the question stem content includes reading material and questions; an option generation module 630 for generating an agent with multiple options to generate multiple sets of options corresponding to the questions based on the question stem content, wherein the multiple sets of options are different; a debate module 640 for debating which set of options is optimal among the multiple sets of options using two preset evaluation models, obtaining a debate record and the optimal option selected by the two evaluation models; a revision module 650 for revising the optimal option based on the debate record using a revision agent, obtaining the revised option; an answering module 660 for answering the question stem content and the revised option using a question-answering agent, obtaining the answer result; and a question generation module 670 for generating reading comprehension questions based on the question stem content, the revised option, and the answer result.

[0110] Optionally, the question stem generation module 620 is specifically used to: search a preset vector database based on the keyword vector corresponding to the keyword to obtain the corresponding search results; fill the search results into a preset prompt word template to obtain reconstructed prompt words, wherein the prompt word template contains real reading comprehension test questions as demonstrations; and input the reconstructed prompt words into the question stem generation model to generate the question stem content.

[0111] Optionally, before generating multiple sets of options corresponding to the question based on the question stem content, the device 600 further includes a training module 680, used to instantiate the option generation model based on the pre-trained basic model to obtain multiple option generation agents. The option generation model is obtained through the following operations: acquiring standard questions from a standard question set, which are obtained by converting real reading comprehension questions into structured data containing reading material fields, question fields, option fields, and question answer fields; generating corresponding option generation logic information based on the question stem content of the standard questions using a preset large language model, which represents the thought process of generating corresponding options according to the question stem content in the standard questions; constructing an option generation training set based on the question stem content of the standard questions in the standard question set, the corresponding options, and the corresponding option generation logic information; and fine-tuning the pre-trained basic model based on the option generation training set to obtain the option generation model.

[0112] Optionally, before obtaining the answer result by having the question-solving agent respond to the question stem and the revised options, the training module 680 is further used to instantiate the question-solving model based on the pre-trained basic model to obtain the question-solving agent. The question-solving model is obtained through the following operations: obtaining standard questions from a standard question set, which are obtained by converting real reading comprehension questions into structured data containing reading material fields, question fields, option fields, and question answer fields; generating question-solving strategy information and question answers selected from the options corresponding to the standard questions based on the question stem content of the standard questions through a preset large language model; constructing a question-solving training set based on the reading material, question, option, question answer, and question-solving strategy information corresponding to the standard questions, provided that the selected question answers match the question answers corresponding to the standard questions; and fine-tuning the pre-trained basic model based on the question-solving training set to obtain the question-solving model.

[0113] Optionally, the debate module 640 is specifically used for: In the initialization phase, the two judging models are divided into a positive judging model and a negative judging model, and the viewpoint information of the positive and negative judging models is determined respectively. The viewpoint information of the positive judging model includes the optimal option selected by the positive judging model from multiple sets of options and the corresponding reasons. The viewpoint information of the negative judging model includes the optimal option selected by the negative judging model from multiple sets of options and the corresponding reasons. The optimal option selected by the negative judging model is inconsistent with the optimal option selected by the positive judging model. In the debate phase, in each round of debate, based on the speaking information of the negative judging model in the previous round, the positive judging model's... The debate process involves several rounds of discussions. The discussion includes recording the statements and their corresponding confidence levels. The opposing side's evaluation model, based on the statements from the previous round's affirmative evaluation model, determines the statements and their corresponding confidence levels. The statements indicate whether the respective evaluation model agrees with the opposing side's viewpoint and whether it maintains its own. After each round of debate, the statements from both sides' evaluation models are stored to obtain a debate record. Furthermore, after each round, the confidence levels for the affirmative and opposing evaluation models are determined. If the confidence levels differ, the debate ends, and the optimal option selected by the evaluation model with the higher confidence level is used as the optimal option selected by both evaluation models. If the confidence levels are the same, the next round of debate begins.

[0114] Optionally, the revision module 650 is specifically used to revise the optimal option by revising the intelligent agent based on the opposing opinions for the optimal option in the debate record, so as to obtain the revised option.

[0115] Optionally, the option generation module 630 is specifically used to input the question stem into multiple option generation agents, each option generation agent outputting a corresponding set of options and corresponding test questions. The answering module 660 is specifically used to compose the question stem and revised options into a question to be answered, and to answer the question n times through a test-taking agent to obtain n corresponding answer results. The process of the test-taking agent answering n times is independent and does not share context. The test question generation module 670 is specifically used to count the most frequent answer among the n answer results, and to determine whether the most frequent answer matches the answer corresponding to the revised option. If they match, the reading comprehension test question is generated based on the question to be answered and the most frequent answer; if they do not match, the question to be answered is revised through the revision agent to obtain a revised question, and the test-taking agent answers the revised question again to obtain a second answer result. The reading comprehension test question is generated based on the second answer result and the revised question.

[0116] Therefore, according to this embodiment, the diversity and logical rigor of reading comprehension test question generation can be improved, the coarse-grained retrieval problem existing in the prior art can be solved, and the cost of generating training samples can be reduced by automatically annotating and generating training samples for fine-tuning the pre-trained base model.

[0117] Example 3

[0118] Figure 7 A reading comprehension test item generation apparatus 700 based on a large language model multi-agent according to the first aspect of this embodiment is shown, which corresponds to the method described according to the first aspect of Embodiment 1. Reference Figure 7 As shown, the device 700 includes: a processor 710; and a memory 720 connected to the processor 710, used to provide the processor 710 with instructions to process the following steps: acquiring user-input keywords corresponding to the reading comprehension questions to be generated; generating question stem content corresponding to the reading comprehension questions based on the keywords using a RAG-based question stem generation model, the question stem content including reading material and questions; generating an agent with multiple options, generating multiple sets of options corresponding to the questions based on the question stem content, wherein the multiple sets of options are different; debating which set of options is optimal among the multiple sets of options using two preset evaluation models, obtaining a debate record and the optimal option selected by the two evaluation models; revising the optimal option based on the debate record using a revision agent, obtaining the revised option; answering the question stem content and the revised option using a question-answering agent, obtaining the answer result; and generating the reading comprehension questions based on the question stem content, the revised option, and the answer result.

[0119] Optionally, the operation of generating question stem content corresponding to reading comprehension test questions using a RAG-based question stem generation model based on keywords includes: searching a preset vector database based on keyword vectors corresponding to the keywords to obtain the corresponding search results; filling the search results into a preset prompt word template to obtain reconstructed prompt words, wherein the prompt word template contains real reading comprehension test questions as demonstrations; and inputting the reconstructed prompt words into the question stem generation model to generate question stem content.

[0120] Optionally, before generating multiple sets of options corresponding to the question based on the question stem, the memory 720 is further configured to provide the processor 710 with instructions to process the following steps: instantiating the option generation model based on the pre-trained base model to obtain multiple option generation agents, wherein the option generation model is obtained through the following operations: acquiring standard questions from a standard question set, which are obtained by converting real reading comprehension questions into structured data containing reading material fields, question fields, option fields, and question answer fields; generating corresponding option generation logic information based on the question stem content of the standard questions using a pre-set large language model, which represents the thought process of generating the corresponding options according to the question stem content in the standard questions; constructing an option generation training set based on the question stem content of the standard questions in the standard question set, the corresponding options, and the corresponding option generation logic information; and fine-tuning the pre-trained base model based on the option generation training set to obtain the option generation model.

[0121] Optionally, before the intelligent agent answers the question stem and the revised options and obtains the answer result, the memory 720 is also used to provide the processor 710 with instructions to process the following steps: instantiating the question-solving model based on the pre-trained basic model to obtain the intelligent agent, wherein the question-solving model is obtained through the following operations: obtaining standard questions from a standard question set, which are obtained by converting real reading comprehension questions into structured data containing reading material fields, question fields, option fields, and question answer fields; generating question-solving strategy information and question answers selected from the options corresponding to the standard questions based on the question stem content of the standard questions through a preset large language model; constructing a question-solving training set based on the reading material, question, option, question answer, and question-solving strategy information corresponding to the standard questions when the selected question answers match the question answers corresponding to the standard questions; and fine-tuning the pre-trained basic model based on the question-solving training set to obtain the question-solving model.

[0122] Optionally, the process involves using two pre-set evaluation models to debate which set of options is optimal among multiple options, obtaining the debate record and the optimal option selected by the two evaluation models. Specifically, this includes: an initialization phase, where the two evaluation models are divided into a positive evaluation model and a negative evaluation model, and the viewpoint information of each model is determined. The positive evaluation model's viewpoint information includes the optimal option selected from multiple options and its corresponding reasons, while the negative evaluation model's viewpoint information includes the optimal option selected from multiple options and its corresponding reasons. The optimal option selected by the negative evaluation model may differ from that selected by the positive evaluation model. The debate phase involves, in each round of the debate, using the positive evaluation model, the optimal option is determined... Based on the statements of the opposing side's evaluation model in the previous round, the statements of the affirmative side's evaluation model and their corresponding confidence levels are determined. Similarly, based on the statements of the affirmative side's evaluation model in the previous round, the statements of the opposing side's evaluation model and their corresponding confidence levels are determined. The statements indicate whether the respective evaluation model agrees with the opposing side's viewpoint and whether it adheres to its own viewpoint. After each round of debate, the statements of both evaluation models are stored to obtain a debate record. After each round of debate, the confidence levels of the affirmative and opposing evaluation models are determined. If the confidence levels differ, the debate ends, and the optimal option selected by the evaluation model with the higher confidence level is taken as the optimal option selected by both evaluation models. If the confidence levels are the same, the next round of debate begins.

[0123] Optionally, the operation of revising the optimal option by the revising agent based on the debate record to obtain the revised option specifically includes: revising the optimal option by the revising agent based on the opposing opinions against the optimal option in the debate record to obtain the revised option.

[0124] Optionally, an intelligent agent generates multiple options based on the question stem, generating multiple sets of options corresponding to the question. Specifically, this includes: inputting the question stem into the intelligent agent, with each agent outputting a set of options and the corresponding answer; and having a question-answering agent respond to the question stem and the revised options to obtain the answer result. Specifically, this involves: forming the question stem and the revised options into a question to be answered; having the question-answering agent answer the question n times to obtain n corresponding answer results. The process of the question-answering agent answering the question n times is independent and does not share context. The process involves generating reading comprehension questions based on the question stem, revised options, and answers. Specifically, this includes: identifying the most frequent answer among n answers; determining if the most frequent answer matches the answer corresponding to the revised option; if they match, generating reading comprehension questions based on the question to be answered and the most frequent answer; and if they don't match, revising the question using a revision agent to obtain the revised question, then having the answering agent answer the revised question again to obtain the result, and finally generating reading comprehension questions based on the result of the second answer and the revised question.

[0125] Therefore, according to this embodiment, the diversity and logical rigor of reading comprehension test question generation can be improved, the coarse-grained retrieval problem existing in the prior art can be solved, and the cost of generating training samples can be reduced by automatically annotating and generating training samples for fine-tuning the pre-trained base model.

[0126] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0127] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0132] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for generating reading comprehension test questions based on a large language model and multiple agents, characterized in that, include: Obtain the keywords input by the user that correspond to the reading comprehension questions to be generated; Based on the keywords, a question stem content corresponding to the reading comprehension test question is generated using a RAG-based question stem generation model. The question stem content includes the reading material and the question. An intelligent agent is generated by using multiple options. Based on the question stem, multiple sets of options corresponding to the question are generated, wherein each set of options is different. Using two preset evaluation models, a debate is held to determine which set of options is optimal among the multiple sets of options, resulting in a debate record and the optimal option selected by the two evaluation models. The revised option is obtained by revising the optimal option based on the debate record by the revising agent; The intelligent agent responds to the question stem and the revised options to obtain the answer result; and Based on the question stem, the revised options, and the answers, the reading comprehension questions are generated, and wherein... The process involves debating which set of options is optimal among multiple sets of choices using two pre-defined evaluation models, obtaining a debate record, and identifying the optimal option selected by the two evaluation models. Specifically, this includes: In the initialization phase, the two evaluation models are divided into a positive evaluation model and a negative evaluation model, and the viewpoint information of the positive evaluation model and the negative evaluation model are determined respectively. During the debate phase, in each round of debate, the affirmative evaluation model, based on the statements made by the negative evaluation model in the previous round, determines the statements and corresponding confidence levels of the affirmative evaluation model; similarly, the negative evaluation model, based on the statements made by the affirmative evaluation model in the previous round, determines the statements and corresponding confidence levels of the negative evaluation model. After each round of debate, the confidence levels of the affirmative and negative evaluation models are determined. If the confidence levels of the two sides are different, the debate ends. The optimal option selected by the evaluation model with the higher confidence level is taken as the optimal option selected by the two evaluation models. If the confidence levels of the two sides are the same, the next round of debate begins.

2. The method according to claim 1, characterized in that, Based on the keywords, the operation of generating the question stem content corresponding to the reading comprehension question using a RAG-based question stem generation model includes: Based on the keyword vectors corresponding to the keywords, a search is performed in a preset vector database to obtain the corresponding search results; The search results are filled into a preset prompt word template to obtain reconstructed prompt words, wherein the prompt word template includes real reading comprehension test questions as examples; and The reconstructed prompt words are input into the question stem generation model to generate the question stem content.

3. The method according to claim 1, characterized in that, Before generating an intelligent agent from multiple options, based on the question stem, the method further includes: The option generation model based on the pre-trained base model is instantiated to obtain the plurality of option generation agents, wherein the option generation model is obtained through the following operations: Obtain standard questions from a standard question set, which are obtained by converting real reading comprehension test questions into structured data containing reading material fields, question fields, option fields, and test answer fields; Based on the content of the question stem of the standard question, a corresponding option generation logic information is generated using a pre-defined large language model. The option generation logic information is used to represent the thought process of generating the corresponding option according to the content of the question stem in the standard question. Based on the question stems, corresponding options, and corresponding option generation logic information of the standard questions in the standard question set, an option generation training set is constructed; and A training set is generated based on the options, and the pre-trained base model is fine-tuned to obtain the option-generated model.

4. The method according to claim 1, characterized in that, Before obtaining the answer result by having the question-answering agent respond to the question stem and the revised options, the method further includes: The problem-solving model based on the pre-trained base model is instantiated to obtain the problem-solving agent, wherein the problem-solving model is obtained through the following operations: Obtain standard questions from a standard question set, which are obtained by converting real reading comprehension test questions into structured data containing reading material fields, question fields, option fields, and test answer fields; Based on the question stem of the standard question, the system generates problem-solving strategies and answers selected from the options corresponding to the standard question through a pre-defined large language model. If the selected test question answers match the test question answers corresponding to the standard questions, a test question training set is constructed based on the reading materials, questions, options, test question answers, and problem-solving strategies corresponding to the standard questions; and Based on the training set, the pre-trained basic model is fine-tuned to obtain the problem-solving model.

5. The method according to claim 1, characterized in that, in, The opinion information of the affirmative evaluation model includes the optimal option selected by the affirmative evaluation model from the multiple options and the corresponding reasons. The opinion information of the negative evaluation model includes the optimal option selected by the negative evaluation model from the multiple options and the corresponding reasons. The optimal option selected by the negative evaluation model is inconsistent with the optimal option selected by the affirmative evaluation model. The speaking information is used to indicate whether the corresponding evaluation model agrees with the other party's point of view and whether it insists on its own point of view. After a round of debate, the speaking information of both sides' evaluation models is stored to obtain the debate record of a round of debate.

6. The method according to claim 1, characterized in that, The operation of revising the optimal option by the revising agent based on the debate record to obtain the revised option specifically includes: The revising agent revises the optimal option based on the opposing opinions in the debate record, resulting in a revised option.

7. The method according to claim 1, characterized in that, The process of generating an intelligent agent from multiple options, based on the question stem, involves generating multiple sets of options corresponding to the question, specifically including: The question stem is input into the multiple option generation agents. Each option generation agent outputs a corresponding set of options and the corresponding question answer, wherein... The operation of obtaining answer results by having a question-solving agent answer the question stem and the revised options specifically includes: forming a question to be answered from the question stem and the revised options; having the question-solving agent answer the question to be answered n times to obtain n corresponding answer results; wherein the process of the question-solving agent answering the question n times is independent and does not share context, and wherein... Based on the question stem, the revised options, and the answers, the reading comprehension questions are generated, specifically including: The most frequent answer among the n responses is counted, and it is determined whether the most frequent answer matches the answer corresponding to the revised option. If they match, the reading comprehension question is generated based on the question to be answered and the most frequent answer; and If there is a discrepancy, the revision agent revises the question to be answered to obtain a revised question. The answering agent then answers the revised question again to obtain a result. Based on the result of the second answer and the revised question, the reading comprehension test question is generated.

8. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, the method described in any one of claims 1 to 7 is performed by a processor.

9. A reading comprehension test question generation device based on a large language model and multiple agents, characterized in that, include: The acquisition module is used to acquire the keywords input by the user that correspond to the reading comprehension questions to be generated; The question stem generation module is used to generate question stem content corresponding to the reading comprehension test question based on the keywords and through the RAG-based question stem generation model. The question stem content includes reading material and questions. The option generation module is used to generate an intelligent agent from multiple options, and to generate multiple sets of options corresponding to the question based on the question stem content, wherein each set of options is different; The debate module is used to debate which set of options is optimal among multiple sets of options using two preset evaluation models, obtaining a debate record and the optimal option selected by the two evaluation models. Specifically, the debate module is used in the following stages: Initialization phase: In the initialization phase, the two evaluation models are divided into a positive evaluation model and a negative evaluation model, and the viewpoint information of the positive and negative evaluation models is determined respectively; Debate phase: In each round of debate in the debate phase, the positive evaluation model is used to analyze the statements made by the negative evaluation model in the previous round. The information determines the speaking information and corresponding confidence level of the affirmative evaluation model, and the speaking information and corresponding confidence level of the negative evaluation model based on the speaking information of the affirmative evaluation model in the previous round; and after each round of debate, the confidence levels corresponding to the affirmative and negative evaluation models are determined. If the confidence levels of the two sides are different, the debate ends, and the optimal option selected by the evaluation model with the higher confidence level is taken as the optimal option selected by the two evaluation models. If the confidence levels of the two sides are the same, the next round of debate begins. The revision module is used to revise the optimal option based on the debate record by a revision agent to obtain the revised option; The answering module is used to answer the question stem and the revised options through a question-answering agent, and obtain the answer result; and The question generation module is used to generate the reading comprehension questions based on the question stem, the revised options, and the answer results.

10. A reading comprehension test question generation device based on a large language model and multiple agents, characterized in that, include: processor; as well as A memory, connected to the processor, for providing the processor with instructions to perform the following processing steps: Obtain the keywords input by the user that correspond to the reading comprehension questions to be generated; Based on the keywords, a question stem content corresponding to the reading comprehension test question is generated using a RAG-based question stem generation model. The question stem content includes the reading material and the question. An intelligent agent is generated by using multiple options. Based on the question stem, multiple sets of options corresponding to the question are generated, wherein each set of options is different. Using two preset evaluation models, a debate is held to determine which set of options is optimal among the multiple sets of options, resulting in a debate record and the optimal option selected by the two evaluation models. The revised option is obtained by revising the optimal option based on the debate record by the revising agent; The intelligent agent responds to the question stem and the revised options to obtain the answer result; and Based on the question stem, the revised options, and the answers, the reading comprehension questions are generated, and wherein... The process involves debating which set of options is optimal among multiple sets of choices using two pre-defined evaluation models, obtaining a debate record, and identifying the optimal option selected by the two evaluation models. Specifically, this includes: In the initialization phase, the two evaluation models are divided into a positive evaluation model and a negative evaluation model, and the viewpoint information of the positive evaluation model and the negative evaluation model are determined respectively. During the debate phase, in each round of debate, the affirmative evaluation model, based on the statements made by the negative evaluation model in the previous round, determines the statements and corresponding confidence levels of the affirmative evaluation model; similarly, the negative evaluation model, based on the statements made by the affirmative evaluation model in the previous round, determines the statements and corresponding confidence levels of the negative evaluation model. After each round of debate, the confidence levels of the affirmative and negative evaluation models are determined. If the confidence levels of the two sides are different, the debate ends. The optimal option selected by the evaluation model with the higher confidence level is taken as the optimal option selected by the two evaluation models. If the confidence levels of the two sides are the same, the next round of debate begins.

Images