Computing devices, methods, computer-readable storage media, and computer program products for optimizing multiple-choice questions

Abstract

The present application relates to a computer system utilizing a computing model, and discloses a computing device, method, computer readable storage medium and computer program product for optimizing a multiple-choice question. The computing device for optimizing a multiple-choice question comprises a computing resource; a data pipeline executed on the computing resource to: obtain an original multiple-choice question from a data set; determine an initial valid pool of distractor options; iteratively update the initial valid pool; determine a final valid pool based on a history record of the valid pool; and generate a final multiple-choice question based on a stem, a correct option and the final valid pool. The computing device according to the present application overcomes the limitation that it is difficult for a technique of synthesizing a multiple-choice question based on a large language model as training data to quantify the misleadingness of distractor options to a target model, and replaces the one-time generation and end mechanism of the prior art with an iterative closed-loop optimization mechanism centered on model behavior feedback, thereby improving the ability to obtain high-quality multiple-choice questions for training a large language model.

Description

Technical Field

[0001] This invention generally relates to computer systems utilizing computational models, and more specifically to computational devices, methods, computer-readable storage media, and computer program products for optimizing multiple-choice questions. Background Technology

[0002] In recent years, data synthesis and augmentation based on large language models (LLM) has become a mainstream research direction. Wang et al.’s “A Survey on Data Synthesis and Augmentation for Large Language Models” (arXiv:2410.12896, 2024) systematically reviewed the relevant methods and challenges. Much of this work focuses on synthesizing training data such as instructions / questions, such as the Self-Instruct method proposed by Wang et al. (“Self-Instruct: Aligning Language Models with Self-Generated Instructions”, ACL 2023), the WizardLM method proposed by Xu et al. (“WizardLM: Empowering large pre-trained language models to follow complex instructions”, arXiv:2304.12244, 2025), and the Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation”, arXiv:2406.00179, 2024, etc.

[0003] In contrast, there is relatively little research on "fixed question stems and generation of high-quality distractors," and most of it is used to construct evaluation benchmarks or educational assessment scenarios. Relevant studies include Zhang et al.'s "Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation" (arXiv:2501.03225, 2025) and Offerijns et al.'s "Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question Filtering" (arXiv:2010.09598, 2020). Existing methods for generating and evaluating distractors typically rely on external text metrics (such as BLEU proposed by Papineni et al. ("BLEU: a Method for Automatic Evaluation of Machine Translation", ACL 2002) and ROUGE proposed by Lin et al. ("ROUGE: A Package for Automatic Evaluation of Summaries", ACL 2004)) or ranking / retrieval methods (such as Liang et al.'s "Distractor Generation for Multiple Choice Questions Using Learning to Rank" (BEA Workshop, 2018)) to measure quality. However, these metrics are difficult to directly characterize whether a target model will be truly "tricked" by distractors when answering questions, and they also lack an iterative closed-loop optimization mechanism centered on model behavior feedback.

[0004] There is a need in this field to improve techniques for optimizing multiple-choice questions as training data in at least one of the above aspects. Summary of the Invention

[0005] This invention is provided to offer further improved techniques for optimizing multiple-choice questions based on large language models.

[0006] One aspect of the present invention provides a computing device for optimizing multiple-choice questions, the computing device comprising: computing resources; and a data pipeline executing on the computing resources to: obtain original multiple-choice questions from a dataset, the original multiple-choice questions including a stem and a plurality of options associated with the stem, the plurality of options including a correct option and original distractor options; determine an initial valid pool of distractor options, the initial valid pool including a subset of the original distractor options; iteratively update the initial valid pool, including: invoking a generative model to generate an answer based on the stem as a candidate distractor option associated with the stem; and based on the correct option and the currently available... The following steps are taken: First, a test set is generated from the distractors in the effective pool and the candidate distractors. Then, an evaluation model is invoked to solve the current multiple-choice question a predetermined number of times, where the current multiple-choice question includes the question stem and the test set. Based on the number of times the candidate distractor is selected and the total number of times the correct option is not selected in the solution process, the strength of each candidate distractor is determined. Based on the strength of the candidate distractors, the current effective pool is selectively updated. The updated effective pool is added to the effective pool history record. The final effective pool is determined based on the effective pool history record. Finally, the final multiple-choice question is generated based on the question stem, the correct option, and the final effective pool.

[0007] In the computing device described above, the data pipeline is configured to: solve the original multiple-choice question a predetermined number of times using the evaluation model; and exclude distracting options that have never been selected by the evaluation model from the original distracting options to determine the initial valid pool of distracting options.

[0008] The computing device as described in any of the above, the data pipeline is further configured to compare the size of the current effective pool with the target number of interference options.

[0009] The computing device as described in any of the preceding claims, wherein the data pipeline is configured to iteratively update the initial valid pool in response to a comparison result indicating that the size of the current valid pool is less than the target number of distracting options, the data pipeline being configured to: generate a number of candidate distracting options to be generated based on the stem using the generation model, wherein the number of candidate distracting options to be generated is determined based on the difference between the size of the current valid pool and the target number of distracting options; and add candidate distracting options with a strength greater than zero from the candidate distracting options to the current valid pool.

[0010] The computing device as described in any of the preceding claims, wherein the data pipeline is configured to iteratively update the initial valid pool in response to a comparison result indicating that the size of the current valid pool is greater than or equal to the target number of distracting options, the data pipeline being configured to: identify the distracting option with the lowest intensity in the current valid pool; and the data pipeline being configured to: generate a predetermined number of candidate distracting options based on the question stem using the generative model; generate a test set based on the correct option, the distracting options in the valid pool after excluding the distracting option with the lowest intensity, and the predetermined number of candidate distracting options; and replace the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the distracting option with the lowest intensity.

[0011] The computing device as described in any of the above, wherein the predetermined number of candidate interference options includes a predetermined number of multiple candidate interference options, and the data pipeline is configured to: in response to determining that the candidate interference option with the greatest strength among the predetermined number of candidate interference options has a greater strength than the interference option with the least strength, replace the interference option with the candidate interference option with the greatest strength.

[0012] The computing device as described in any of the above, wherein the data pipeline is configured to iteratively update the initial valid pool a predetermined number of times.

[0013] The computing device as described in any of the above, wherein the data pipeline is configured to: select a candidate valid pool with the largest valid pool size from the valid pool history, wherein the final valid pool is determined at least in part based on the candidate valid pool.

[0014] The computing device as described in any of the above, wherein the data pipeline is configured to: determine the difficulty of the test set based on a predetermined number of times the evaluation model is solved and the total number of times the correct option is selected; and select the candidate valid pool as the valid pool with the largest valid pool size and the corresponding highest test set difficulty from the valid pool history.

[0015] The computing device as described in any of the above, wherein the data pipeline is configured to: in response to determining that the size of the selected effective pool is less than the target number, fill the candidate effective pool with at least a portion of the original interference options to obtain the final effective pool.

[0016] The computing device as described in any of the above, wherein the data pipeline is configured to: regenerate candidate interference options if the generated answer is the same as the correct option or at least one interference option in the current valid pool.

[0017] The computing device as described in any of the above claims, the computing device further includes the generation model and the evaluation model.

[0018] Another aspect of the present invention provides a method for optimizing multiple-choice questions, the method comprising: S1: obtaining original multiple-choice questions from a dataset, the original multiple-choice questions including a stem and multiple options associated with the stem, the multiple options including a correct option and original distractor options; S2: determining an initial valid pool of distractor options, the initial valid pool including a subset of the original distractor options; S3: iteratively updating the initial valid pool, S3 comprising: S31: generating an answer based on the stem using a generative model as a candidate distractor option associated with the stem; S32: based on the correct option, the distractor options in the current valid pool, and the candidate stem... S33: Generate a test set of distractor options; S34: Use an evaluation model to solve the current multiple-choice question a predetermined number of times, wherein the current multiple-choice question includes the question stem and the test set; S35: Determine the strength of each candidate distractor option based on the number of times the candidate distractor option is selected and the total number of times the correct option is not selected in the solution using the evaluation model; S36: Selectively update the current valid pool based on the strength of the candidate distractor option; S4: Add the updated valid pool to the valid pool history record; S5: Determine the final valid pool based on the valid pool history record; and S6: Generate the final multiple-choice question based on the question stem, the correct option, and the final valid pool.

[0019] As described above, S2 includes: S21: using the evaluation model to solve for a predetermined number of times the original multiple-choice question is asked; and S22: excluding interference options that have never been selected by the evaluation model from the original interference options to determine the initial valid pool of interference options.

[0020] The method described in any of the above methods further includes: S6: comparing the size of the current effective pool with the target number of interference options.

[0021] As described in any of the above methods, S3 is performed in response to the result of the comparison indicating that the size of the current valid pool is less than the target number of interference options, wherein S31 includes: S311: using the generation model to generate a number of candidate interference options to be generated based on the question stem, wherein the number of candidate interference options to be generated is determined based on the difference between the size of the current valid pool and the target number of interference options; and S35 includes: S351: adding candidate interference options with a strength greater than zero from the candidate interference options to the current valid pool.

[0022] As described in any of the above methods, step S3 is performed in response to the result of the comparison indicating that the size of the current valid pool is greater than or equal to the target number of interference options, and step S3 further includes: S37: identifying the interference option with the lowest intensity in the current valid pool; step S31 includes: S312: generating a predetermined number of candidate interference options based on the question stem using the generative model; step S32 includes: S322: generating a test set based on the correct option, the interference options in the valid pool after excluding the interference option with the lowest intensity, and the predetermined number of candidate interference options; and step S35 includes: S352: replacing the interference option with the lowest intensity with the candidate interference option in response to determining that the intensity of one of the candidate interference options in the predetermined number of candidate interference options is greater than that of the interference option with the lowest intensity.

[0023] As described in any of the above methods, the predetermined number of candidate interference options includes a predetermined number of multiple candidate interference options, and S352 includes: S3521: in response to determining that the candidate interference option with the greatest intensity among the predetermined number of candidate interference options has a greater intensity than the interference option with the least intensity, replacing the interference option with the candidate interference option with the greatest intensity.

[0024] As described in any of the above methods, S3 is performed a predetermined number of times.

[0025] As described in any of the above methods, S4 includes: S41: selecting a candidate valid pool with the largest valid pool size from the valid pool history, wherein the final valid pool is determined at least in part based on the candidate valid pool.

[0026] As described in any of the above methods, S3 further includes: S38: determining the difficulty of the test set based on the predetermined number of times the evaluation model is solved and the total number of times the correct option is selected; and S41 includes: S411: selecting the effective pool with the largest effective pool size and the corresponding test set difficulty as the candidate effective pool from the effective pool history.

[0027] As described in any of the above methods, S4 further includes: S42: in response to determining that the size of the selected effective pool is less than the target number, filling the candidate effective pool with at least a portion of the original interference options to obtain the final effective pool.

[0028] As described in any of the above methods, S31 includes: S313: if the generated answer is the same as the correct option or at least one interference option in the current valid pool, regenerate candidate interference options.

[0029] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0030] Another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0031] The computing device and method of the present invention overcome the limitations of current techniques that use multiple-choice questions synthesized based on large language models as training data, which make it difficult to quantify the deceptiveness of distracting options to the target model. Furthermore, by replacing the one-time generation and termination mechanism of the prior art with an iterative closed-loop optimization mechanism centered on model behavior feedback, the ability to obtain high-quality multiple-choice questions for training large language models is improved. Attached Figure Description

[0032] Figure 1 This is a block diagram of a computing device for optimizing multiple-choice questions according to some embodiments of the present invention.

[0033] Figure 2 A partial implementation pipeline diagram for optimizing multiple-choice questions according to some embodiments of the present invention is shown.

[0034] Figure 3 A schematic diagram illustrating interference generation options according to some embodiments of the present invention is shown.

[0035] Figure 4 A schematic diagram of an updated valid pool is shown according to some embodiments of the present invention.

[0036] Figure 5 A schematic diagram illustrating the selection of candidate valid pools from the valid pool history is shown according to some embodiments of the present invention.

[0037] Figure 6 This is a flowchart of a method for optimizing multiple-choice questions according to some embodiments of the present invention.

[0038] Figure 7 This is a flowchart of a first process associated with a method for optimizing multiple-choice questions according to some embodiments of the present invention.

[0039] Figure 8 This is a flowchart of a second process associated with a method for optimizing multiple-choice questions according to some embodiments of the present invention.

[0040] Figure 9 This is a flowchart of a third process associated with a method for optimizing multiple-choice questions, according to some embodiments of the present invention.

[0041] Figure 10This is a block diagram of a computer-readable storage medium according to some embodiments of the present invention.

[0042] Figure 11 This is a block diagram of a computer program product according to some embodiments of the present invention.

[0043] In the accompanying drawings, the same reference numerals refer to the same elements. Detailed Implementation

[0044] In this application, the term "data pipeline" refers to a logical collection of data processing steps that can be implemented and deployed via software code. A data pipeline can execute on the computing resources of a computing device. A data pipeline may include steps that call corresponding models via API interfaces. A data pipeline may also integrate corresponding tools (e.g., PDF readers, Python interpreters, calculators, etc.) to process various forms of input or implement corresponding functions.

[0045] In this application, the term "multiple-choice question" refers to the training input to a model (e.g., a large language model) used to perform multiple-choice question solving. A "multiple-choice question" may include a stem input to the model, and options associated with the stem for the model to choose from. Options may include a correct answer and "distractor options," i.e., incorrect options, that serve to confuse the model in solving the multiple-choice question.

[0046] In this application, ordinal numbers such as "first," "second," and "third" are used to distinguish different instances of objects with the same name. The ordinal numbers "first," "second," and "third" do not indicate a relative order of the indicated objects in time, space, sequence, or other aspects.

[0047] According to one aspect of the present invention, a computing device for optimizing multiple-choice questions is provided.

[0048] Figure 1 This is a block diagram of a computing device 100 for optimizing multiple-choice questions according to some embodiments of the present invention.

[0049] The computing device 100 may be a local or remote computer, server, etc. In some embodiments, the computing device 100 may include computing resources 110 and data pipelines 120. The data pipelines 120 may be executed on the computing resources 110 to enable the computing resources 110 to perform corresponding operations.

[0050] In some embodiments, computing resources 110 may include a central processing unit (CPU), a graphics processing unit (GPU), and various other processing units or cores (e.g., arithmetic logic units, integer units, floating-point units, tensor units, ray tracing cores, etc.).

[0051] In some embodiments, data pipeline 120 may include a step of calling a corresponding model via an API. As an example, the model may include a large language model, a multimodal model, a multimodal language model, etc. In some embodiments, the model that can be called in data pipeline 120 may be a generation model 130 and / or an evaluation model 140.

[0052] In some embodiments, computing device 100 may include a generation model 130 and an evaluation model 140. In some embodiments, models that can be invoked in data pipeline 120 (e.g., generation model 130 and / or evaluation model 140) may be deployed locally on computing device 100.

[0053] In some embodiments, computing device 100 may not include generation model 130 and evaluation model 140. In some embodiments, models that can be invoked in data pipeline 120 (e.g., generation model 130 and / or evaluation model 140) may be deployed remotely to computing device 100, for example, in the cloud.

[0054] In some embodiments, computing device 100 may include only one of the generation model 130 and the evaluation model 140. In some embodiments, some of the models that can be invoked in data pipeline 120 (e.g., generation model 130 and / or evaluation model 140) may be deployed locally on computing device 100, while others may be deployed remotely on computing device 100.

[0055] In some embodiments, the data pipeline 120 may integrate various tools to process various forms of input or implement corresponding functions. For example, the data pipeline 120 may integrate a Python interpreter to process Python code. The following will combine... Figures 2-5 Further describe the function and specific implementation of data pipeline 120.

[0056] Data pipeline 120 can be executed on computing resource 110 to obtain raw multiple-choice questions from a dataset. As an example, this dataset could be an existing open-source training dataset. The raw multiple-choice questions may include a stem and multiple options associated with the stem. The multiple options may include a correct answer and original distractor options.

[0057] Data pipeline 120 can be executed on computing resource 110 to determine an initial valid pool of interference options. This initial valid pool may include a subset of the original interference options.

[0058] Data pipeline 120 can be executed on computing resource 110 to iteratively update the initial valid pool.

[0059] In order to iteratively update the initial valid pool, in some embodiments, the data pipeline 120 may be executed on the computing resource 110 to perform the corresponding operations in each iteration.

[0060] In each iteration, data pipeline 120 can invoke generative model 130 to generate answers based on the question stem, serving as candidate distractors associated with the question stem. Data pipeline 120 can generate a test set based on the correct answer, distractors in the current valid pool, and candidate distractors. Data pipeline 120 can invoke evaluation model 140 to solve the current multiple-choice question a predetermined number of times. The current multiple-choice question may include the question stem and the test set. Data pipeline 120 can determine the strength of each candidate distractor based on the number of times the evaluation model 140 selects a candidate distractor and the total number of times the correct answer is not selected. Data pipeline 120 can selectively update the current valid pool based on the strength of the candidate distractors. Data pipeline 120 can add the updated valid pool to the valid pool history.

[0061] In some embodiments, after a predetermined number of iterations have been completed, the data pipeline 120 may be executed on computing resource 110 to determine the final valid pool based on the valid pool history. In some embodiments, the data pipeline 120 may be executed on computing resource 110 to generate the final multiple-choice question based on the question stem, the correct option, and the final valid pool. Through multiple iterations, the strength of the distractors in the final multiple-choice question is enhanced.

[0062] Some implementations use the frequency of selecting a certain distractor option under incorrect conditions as the strength of the distractor option, rather than relying solely on text similarity, to measure the strength of the distractor option based on model behavior. Other implementations use the evaluation results to drive the acceptance or rejection of candidate distractors in each round, thereby gradually improving the quality of distractors and achieving a closed loop of rejection sampling / iterative reinforcement in the option space.

[0063] In some embodiments, to determine an initial valid pool of distractors, data pipeline 120 may be configured to solve the original multiple-choice question a predetermined number of times using evaluation model 140. Data pipeline 120 may be configured to exclude distractors that were never selected by evaluation model 140 from the original distractors to determine an initial valid pool of distractors.

[0064] In some embodiments, data pipeline 120 may also be configured to compare the size of the current valid pool with the target number of interference options. Based on the comparison result, data pipeline 120 may determine whether to populate the valid pool with more interference options or replace existing interference options in the valid pool with stronger interference options. For example, when the size of the current valid pool is less than the target number of interference options, the valid pool can be updated by populating it with more interference options. As another example, when the size of the current valid pool is greater than or equal to the target number of interference options, the valid pool can be updated by replacing the weakest interference option in the current valid pool with a stronger interference option.

[0065] In cases where the valid pool is updated by filling it with distractor options, in some embodiments, data pipeline 120 may be configured to iteratively update the initial valid pool in response to a comparison result indicating that the current valid pool size is less than the target number of distractor options. Data pipeline 120 may be configured to generate a number of candidate distractor options to be generated based on the stem using generative model 130. The number of candidate distractor options to be generated may be determined based on the difference between the current valid pool size and the target number of distractor options. Data pipeline 120 may be configured to add candidate distractor options with a strength greater than zero to the current valid pool.

[0066] Some implementations, through batch generation and intensity screening, can quickly accumulate perturbation options that can deceive the model.

[0067] In cases where the effective pool is updated by replacing existing interference options in the effective pool, in some embodiments, data pipeline 120 may be configured to iteratively update the initial effective pool in response to a comparison result indicating that the current effective pool size is greater than or equal to the target number of interference options. Data pipeline 120 may be configured to identify the interference option with the lowest intensity in the current effective pool. Data pipeline 120 may be configured to generate a predetermined number of candidate interference options based on the prompt using generative model 130. Data pipeline 120 may be configured to generate a test set based on the correct option, interference options in the effective pool after excluding the interference option with the lowest intensity, and the predetermined number of candidate interference options. Data pipeline 120 may be configured to replace the interference option with the lowest intensity interference option with the candidate interference option in response to determining that the intensity of one of the candidate interference options in the predetermined number of candidate interference options is greater than that of the interference option with the lowest intensity.

[0068] In some embodiments, the predetermined number of candidate interference options may include a predetermined number of multiple candidate interference options. The data pipeline 120 may be configured to replace the least intense interference option with the most intense candidate interference option in response to determining that the intensity of the most intense candidate interference option among the predetermined number of candidate interference options is greater than that of the least intense interference option.

[0069] Some embodiments can progressively improve the overall quality of the effective pool by employing a monotonic improvement strategy of "accepting only stronger candidates" when the capacity is fixed.

[0070] The following will combine Figure 2 and Figure 4 The specific implementation of iteratively updating the effective pool based on the comparison results is further described.

[0071] In some embodiments, data pipeline 120 may be configured to iteratively update the initial valid pool a predetermined number of times. In some embodiments, data pipeline 120 may be configured to select a candidate valid pool with the largest valid pool size from the valid pool history. The final valid pool may be determined at least in part based on the candidate valid pools.

[0072] In some embodiments, data pipeline 120 may be configured to determine the difficulty of the test set based on a predetermined number of times the evaluation model 140 is solved and the total number of times the correct option is selected. Data pipeline 120 may be configured to select the candidate valid pool as the valid pool with the largest valid pool size and the highest corresponding test set difficulty from the valid pool history.

[0073] In some embodiments, data pipeline 120 may be configured to fill candidate effective pools with at least a portion of the original interference options to obtain a final effective pool in response to determining that the size of the selected effective pool is less than the target number.

[0074] In some embodiments, data pipeline 120 may be configured to regenerate candidate distractors if the generated answer is the same as the correct option or at least one distractor option in the current valid pool.

[0075] The process of optimizing the selection problem, which can be implemented through the data pipeline 120 of the computing device 100, is described in general terms in Algorithm 1 below.

[0076]

[0077]

[0078] It should be understood that the above algorithms are illustrative and not restrictive, and are not intended to limit the scope of the invention. Embodiments of the invention may include more or fewer steps than shown in the above algorithms, and these steps may not be performed in the order shown. The following will combine... Figures 2-5 Describe in detail the functions and implementation details of each functional module in Algorithm 1.

[0079] Figure 2 A partial implementation pipeline diagram for optimizing multiple-choice questions according to some embodiments of the present invention is shown. Figure 2 The process can be combined with the above. Figure 1The process of executing data pipeline 120 is described.

[0080] In one example, the original multiple-choice question includes four options associated with the stem, one of which is the correct answer and the other three are original distractors. It should be understood that the number of options is only an example, and the original multiple-choice question may include more or fewer options. In this example, the target number of options corresponds to line 3 in Algorithm 1. The answer is 4. As an example and not a limitation, the original multiple-choice question could be obtained from an open-source dataset.

[0081] During the initialization phase (as described in lines 5-12 of Algorithm 1), the data pipeline (e.g., data pipeline 120) constructs an initial set of options using the four options included in the original multiple-choice question. and use (For example, evaluate model 140) To obtain the strength of each interference option. In some embodiments, this can be achieved through... The strength of each distractor option is determined by the number of times the distractor option is selected and the total number of times the correct option is not selected, as shown in the formula in line 9 of Algorithm 1. express The number of times a certain distractor option is selected in the solution, and express The total number of times the correct option was not selected during the solution process.

[0082] As shown in line 10 of Algorithm 1, when a certain interfering option is not... When selected, the intensity of this interference option is 0. When all interference options are not selected... Selected (i.e., the) exist In the case where only the correct option is selected in each sampling, the intensity of each interference option is 0.

[0083] In this example, the strengths of the three original interference options are as follows: , , ,in , It equals 0, and Greater than 0. Accordingly, the effective pool is initialized to include only those with a strength of [value missing]. ( Greater than 0, that is, the corresponding option is in In the second sampling, it was Select the interference options (as described in line 11 of Algorithm 1).

[0084] Subsequently, the data pipeline (e.g., data pipeline 120) enters the iterative optimization phase (as described in lines 13-28 of Algorithm 1). In this phase, a predetermined number of iterations are performed. The number of target interference options can be calculated. In other words, the number of distractor options (deceptive options) needed in a multiple-choice question besides the correct answer. The number of distractor options that can be used to target the correct answer. Compare this to the number of interfering options included in the current valid pool (i.e., the size of the valid pool). As mentioned above... Figure 1 As explained, a data pipeline (e.g., data pipeline 120) can determine whether to enter fill mode or replacement mode based on the results of a comparison. This will be discussed in conjunction with... Figure 4 Further details on updating the effective pool in either fill or replacement mode.

[0085] In this example, the initialized effective pool size is 1, which is less than the number of target interference options, 3. Accordingly, the data pipeline (e.g., data pipeline 120) enters a fill mode (as described in lines 15-20 of Algorithm 1). In this mode, the data pipeline (e.g., data pipeline 120) uses... (For example, generative model 130) generated 2 (i.e., the number of target interference options). (Difference from the current effective pool size) Candidate interference options. The following will combine... Figure 3 The specific details of generating candidate interference options are further described. The correct options, the interference options in the initial valid pool, and the generated candidate options are together constructed into a test set.

[0086] Next, the data pipeline (e.g., data pipeline 120) calls... (For example, evaluation model 140) solves the current multiple-choice question (whose option set is the test set as described above) a predetermined number of times to determine the strength of each candidate distractor option. It should be understood that the strength of candidate distractors is determined in the same way as the strength of the original distractors was determined above. Distractors with a strength greater than 0 are added to the valid pool for use in the next iteration.

[0087] As an example, in the next iteration, if it is determined that the current effective pool size is greater than or equal to the target number of interference options... Then the data pipeline (e.g., data pipeline 120) enters replacement mode (as described in lines 21-28 of Algorithm 1). In replacement mode, the data pipeline (e.g., data pipeline 120) identifies the least intense interference option in the current valid pool. It should be understood that this option is... (For example, evaluation model 140) selects the least likely distractor option during the process of solving a multiple-choice question. The replacement pattern aims to replace the least strong distractor option with a stronger one to iteratively increase the strength of the distractor option (i.e., increase its deceptiveness to the evaluation model).

[0088] Data pipeline (e.g., data pipeline 120) calls (For example, generative model 130) generates one or more candidate interference options. Although described in Algorithm 1 as generating one candidate interference option per round in replacement mode, it is understandable that multiple candidate interference options can also be generated at once, and the one with the highest intensity can be selected to update the current effective pool.

[0089] The correct option, the interference options in the current valid pool excluding the interference option with the lowest intensity, and the generated candidate interference options are together constructed into a test set.

[0090] Data pipeline (e.g., data pipeline 120) calls (For example, evaluation model 140) solves the current multiple-choice question (whose option set is the test set as described above) a predetermined number of times to determine the strength of each candidate distractor option. If only one candidate distractor option is generated, and its strength is greater than the weakest distractor option identified in the current valid pool, then the weakest distractor option is replaced with the generated candidate distractor option to update the valid pool. If multiple candidate distractor options are generated, if any one of them has a strength greater than the weakest distractor option identified in the current valid pool, then the weakest distractor option is replaced with the strongest candidate distractor option generated.

[0091] In each round of iterative optimization, the difficulty of the corresponding test set can also be determined. For example, this difficulty can be determined by... The correct answer ratio is determined by the ratio of the number of times the correct option is selected to the total number of multiple-choice questions answered (hereinafter referred to as the "correct answer ratio"). With the total number of multiple-choice questions remaining constant, the fewer the number of times the correct option is selected, the smaller the correct answer ratio, indicating that the multiple-choice questions are less effective for answering multiple-choice questions. The more "difficult" it is.

[0092] Figure 3 A schematic diagram illustrating interference generation options according to some embodiments of the present invention is shown. Figure 3 The process shown in the figure can be Figure 2 The specific implementation details of the pipeline diagram are partially implemented, and can be Figure 1 The specific implementation details of data pipeline 120.

[0093] As mentioned above Figure 1 and Figure 2As described above, during the iterative optimization phase, the data pipeline (e.g., data pipeline 120) compares the number of target interference options with the current effective pool size, and enters either a fill mode or a replacement mode based on the comparison result. In both fill and replacement modes, the following is called: (For example, generating model 130) generates candidate interference options.

[0094] In some embodiments, conflict checks are performed during the generation of candidate interference options. Specifically, if If the answer generated from the question stem is the same as the correct option for a multiple-choice question or at least one distractor option in the current valid pool (determined as a "conflict"), then candidate distractor options are regenerated. Only when determined not to be a conflict does the data pipeline exit the loop that generates candidate distractor options to perform subsequent evaluation of the candidate distractor options and updates to the valid pool by filling and / or replacing them.

[0095] Figure 4 A schematic diagram of an updated valid pool is shown according to some embodiments of the present invention. Figure 4 The process shown in the figure can be Figure 2 The specific implementation details of the pipeline diagram are partially implemented, and can be Figure 1 The specific implementation details of data pipeline 120.

[0096] like Figure 4 As shown, during the iterative update phase of the effective pool, if the data pipeline enters a filling mode, candidate interference options with a strength greater than 0 (if present) are added to the effective pool. In this case, the size of the effective pool is incremented. Although in Figure 4 The value is shown as increasing by 1, but it should be understood that this is only an example and not a limitation, and the size of the effective pool can also increase by a larger number depending on the number of interference options with a strength greater than 0 among the candidate interference options.

[0097] When the data pipeline enters replacement mode, if the strength of the strongest candidate interference option among the generated candidate interference options is greater than that of the weakest candidate interference option in the current valid pool, then the candidate interference option with the strongest strength will replace the candidate interference option with the weakest strength in the current valid pool.

[0098] Figure 5 A schematic diagram illustrating the selection of candidate valid pools from the valid pool history is shown according to some embodiments of the present invention. Figure 5 The process shown in the figure can be Figure 2 The specific implementation details of the pipeline diagram are partially implemented, and can be Figure 1 The specific implementation details of data pipeline 120.

[0099] Figure 5This corresponds to lines 29-33 in the process algorithm 1 shown above. Figure 1 In each round of iterative optimization, the updated effective pool is added to the effective pool history. The largest effective pool is selected from the effective pool history as a candidate effective pool. Optionally, the effective pool with the smallest corresponding correct answer ratio is selected from the largest effective pool as a candidate effective pool.

[0100] It should be understood that after a predetermined number of iterations of optimization, the maximum size of the effective pool may still be smaller than the target number of distractors. In this case, the candidate effective pool can be filled with at least a portion of the original distractors to obtain the final effective pool. Accordingly, the final optimized options in the final effective pool, along with the question stem and the correct option, can be used to generate the final multiple-choice question.

[0101] Some embodiments use repeatable statistical metrics (intensity, pass rate) to drive the selection of synthetic data, rather than generating it all at once, offering the advantage of closed-loop quantifiability. This intensity metric comes directly from the behavior of the target / evaluation model, and the generated interference options are closer to its decision boundaries and common pitfalls, thus being more effective for the target model. Furthermore, embodiments of the present invention achieve the target number of options through a filling phase and continuously improve quality through a replacement phase, achieving robust output of training data.

[0102] According to another aspect of the present invention, a method for optimizing multiple-choice questions is provided.

[0103] Figure 6 This is a flowchart of a method for optimizing multiple-choice questions according to some embodiments of the present invention. The method can be found in reference... Figure 1 The computing device 100 described (e.g., its data pipeline 120) performs this operation.

[0104] In some examples, the method may include step S1: obtaining the original multiple-choice questions from the dataset. The original multiple-choice questions may include a stem and multiple options associated with the stem, including one correct option and original distractor options.

[0105] In some examples, the method may include step S2: determining an initial valid pool of interference options. The initial valid pool may include a subset of the original interference options.

[0106] In some examples, the method may include step S3: iteratively updating the initial valid pool.

[0107] In some examples, step S3 may include step S31: using a generative model to generate an answer based on the stem as a candidate distractor option associated with the stem.

[0108] In some examples, step S3 may include step S32: generating a test set based on the correct option, the distractors in the current valid pool, and the candidate distractors. In some examples, step S3 may include step S33: solving for the predetermined number of times the current multiple-choice question will be performed using an evaluation model. The current multiple-choice question may include a stem and a test set.

[0109] In some examples, step S3 may include step S34: determining the strength of each candidate interference option based on the number of times the evaluation model selected a candidate interference option in the solution and the total number of times the correct option was not selected. In some examples, step S3 may include step S35: selectively updating the current valid pool based on the strength of the candidate interference option. In some examples, step S3 may include step S36: adding the updated valid pool to the valid pool history.

[0110] In some examples, the method may include S4: determining the final valid pool based on the valid pool history. In some examples, the method may include S5: generating the final multiple-choice question based on the question stem, the correct options, and the final valid pool.

[0111] Some implementations use the frequency of selecting a certain distractor option under incorrect conditions as the strength of the distractor option, rather than relying solely on text similarity, to measure the strength of the distractor option based on model behavior. Other implementations use the evaluation results to drive the acceptance or rejection of candidate distractors in each round, thereby gradually improving the quality of distractors and achieving a closed loop of rejection sampling / iterative reinforcement in the option space.

[0112] In some examples, the method may optionally include step S6: comparing the size of the current effective pool with the target number of interference options.

[0113] In some examples, S31 may optionally include S313: if the generated answer is the same as the correct option or at least one distractor option in the current valid pool, regenerate candidate distractor options.

[0114] Figure 7 This is a flowchart of a first process associated with a method for optimizing multiple-choice questions according to some embodiments of the present invention. This first process may be... Figure 1 Data pipeline 120 in the middle is executed, and can be Figure 6 The specific implementation of step S2 in the method is described, but the scope of the present invention is not limited thereto.

[0115] In some embodiments, the first process may include step S21: solving for a predetermined number of times the original multiple-choice questions are evaluated using an evaluation model. In some embodiments, the first process may include step S22: excluding distractor options that have never been selected by the evaluation model from the original distractor options to determine an initial valid pool of distractor options.

[0116] Figure 8 This is a flowchart of a second process associated with a method for optimizing multiple-choice questions according to some embodiments of the present invention. This second process may be... Figure 1 Data pipeline 120 in the middle is executed, and can be Figure 6 The specific implementation of step S3 in the method is described, but the scope of the present invention is not limited thereto.

[0117] In some embodiments, the second process may include, at block 810, determining whether the result of the comparison indicates that the size of the current effective pool is less than the target number of interference options.

[0118] In some embodiments, where the comparison result indicates that the size of the current effective pool is less than the target number of distractor options, the second process may include step S311: generating the number of candidate distractor options to be generated based on the stem using a generative model. In some embodiments, step S311 may be... Figure 6 The specific implementation of step S31 in the method is as follows. In some embodiments, the second process may include step S351: adding candidate interference options with a strength greater than zero from the candidate interference options to the current valid pool. In some embodiments, step S351 may be... Figure 6 The specific implementation of step S35 in the method.

[0119] In some embodiments, where the comparison result indicates that the size of the current effective pool is greater than or equal to the target number of interfering options, the second process may include step S37: identifying the interfering option with the lowest strength in the current effective pool. In some embodiments, step S37 may be... Figure 6 The additional steps included in step S3 of the method. In some embodiments, the second process may include step S312: generating a predetermined number of candidate distractors based on the stem using a generative model. In some embodiments, step S312 may be... Figure 6 The specific implementation of step S31 in the method.

[0120] In some embodiments, the second process may include step S322: generating a test set based on the correct option, the interference options after excluding the interference option with the lowest intensity from the effective pool, and a predetermined number of candidate interference options. In some embodiments, step S322 may be... Figure 6 The specific implementation of step S32 in the method.

[0121] In some embodiments, the second process may include step S352: in response to determining that the strength of one of the candidate interference options among a predetermined number of candidate interference options is greater than that of the interference option with the lowest strength, replacing the interference option with the candidate interference option with the one that has the lowest strength. In some embodiments, step S352 may be... Figure 6The specific implementation of step S35 in the method.

[0122] In some embodiments, step S352 may include step S3521: in response to determining that the strongest candidate interference option among a predetermined number of candidate interference options has a greater strength than the weakest interference option, the candidate interference option with the strongest strength is replaced with the interference option with the weakest strength.

[0123] Figure 9 This is a flowchart of a third process associated with a method for optimizing multiple-choice questions, according to some embodiments of the present invention. This third process may be... Figure 1 Data pipeline 120 in the middle is executed, and can be Figure 6 The specific implementation of steps S3 and / or S4 in the method is described, but the scope of the present invention is not limited thereto.

[0124] In some embodiments, the third process may include step S41: selecting the effective pool with the largest effective pool size and the corresponding highest test set difficulty from the effective pool history as a candidate effective pool. In some embodiments, step S41 may be... Figure 6 The specific implementation of step S4 in the method.

[0125] In some embodiments, the candidate pool can be further determined based on the difficulty of the test set.

[0126] In some embodiments, the third process may include step S38: determining the difficulty of the test set based on a predetermined number of times the evaluation model has been solved and the total number of times the correct option has been selected. In some embodiments, step S38 may be... Figure 6 The additional steps included in step S3 of the method.

[0127] In some embodiments, step S41 may include step S411: selecting the effective pool with the largest effective pool size and the highest corresponding test set difficulty from the effective pool history as a candidate effective pool. In some embodiments, step S411 may be a specific implementation of the above step S41.

[0128] In some embodiments, the third process may include step S42: in response to determining that the size of the selected effective pool is less than the target number, filling the candidate effective pool with at least a portion of the original interference options to obtain the final effective pool. In some embodiments, step S42 may be... Figure 6 The specific implementation of step S4 in the method.

[0129] According to another aspect of the present invention, a computer-readable storage medium is provided.

[0130] Figure 10This is a block diagram of a computer-readable storage medium 1000 according to some embodiments of the present invention.

[0131] A computer-readable storage medium 1000 stores a computer program 1050. When executed by a processor, the computer program 1050 implements the above-mentioned... Figures 6-9 The steps of each method or process described.

[0132] According to another aspect of the present invention, a computer program product is provided.

[0133] Figure 11 This is a block diagram of a computer program product 1100 according to some embodiments of the present invention.

[0134] Computer program product 1100 may include computer program 1050. Computer program 1050, when executed by a processor, implements the above-mentioned... Figures 6-9 The steps of each method or process described.

[0135] Table 1 below shows the experimental results obtained using embodiments of the present invention.

[0136] Table 1: Downstream performance on medical datasets

[0137]

[0138] The first column of Table 1 shows the different datasets from which multiple-choice questions were obtained for training the Qwen2-7B and Llama-3.1-8B models. The second and fourth columns of Table 1 show the accuracy of the Qwen2-7B and Llama-3.1-8B models in answering multiple-choice questions after training them with multiple-choice questions obtained from different datasets as training data, without utilizing embodiments of the present invention. The third and fifth columns of Table 1 show the accuracy of the Qwen2-7B and Llama-3.1-8B models in answering multiple-choice questions after training them with multiple-choice questions obtained from different datasets using iterative perturbation option optimization, according to embodiments of the present invention.

[0139] As shown in Table 1, the model trained using the embodiments of the present invention to optimize the distractors in multiple-choice questions demonstrates a certain degree of accuracy improvement when performing multiple-choice question answering tasks on different datasets. Therefore, it is evident that training the model using the training dataset obtained from the embodiments of the present invention significantly enhances the model's ability to answer multiple-choice questions.

Claims

1. A computing device for optimizing multiple-choice questions, characterized in that, include: Computing resources; The data pipeline, executed on the computing resources, performs the following: The original multiple-choice questions are obtained from the dataset. The original multiple-choice questions include a stem and multiple options associated with the stem, including a correct option and original distractor options. Determine an initial valid pool of interference options, the initial valid pool comprising a subset of the original interference options; Iteratively updating the initial valid pool includes: The generative model is invoked to generate an answer based on the question stem, which serves as a candidate distractor option associated with the question stem; A test set is generated based on the correct option, the interference options in the current valid pool, and the candidate interference options; The evaluation model is invoked to calculate the predetermined number of times the current multiple-choice question will be answered, wherein the current multiple-choice question includes the question stem and the test set; The strength of each candidate interference option is determined based on the number of times the candidate interference option is selected and the total number of times the correct option is not selected in the solution, according to the evaluation model. The current valid pool is selectively updated based on the strength of the candidate interference options; Add the updated valid pool to the valid pool history; The final effective pool is determined based on the historical records of the effective pool; and The final multiple-choice questions are generated based on the question stem, the correct options, and the final valid pool. The data pipeline is also configured to: The size of the current effective pool is compared with the target number of interference options. The data pipeline is configured to iteratively update the initial effective pool in response to the comparison result indicating that the size of the current effective pool is greater than or equal to the target number of interference options. The data pipeline is configured to: Identify the interference option with the lowest intensity in the currently active pool; and The data pipeline is configured to: The generative model is used to generate a predetermined number of candidate distractors based on the question stem; A test set is generated based on the correct option, the interference options in the current valid pool after excluding the interference option with the lowest intensity, and the predetermined number of candidate interference options; and In response to determining that the strength of one of the predetermined number of candidate interference options is greater than that of the interference option with the lowest strength, the interference option with the lowest strength is replaced with the candidate interference option.

2. The computing device as described in claim 1, characterized in that, The data pipeline is configured to: The evaluation model is used to determine the predetermined number of times the original multiple-choice questions will be answered; and To determine the initial effective pool of interference options, interference options that were never selected by the evaluation model are excluded from the original interference options.

3. The computing device as described in claim 1, characterized in that, The data pipeline is configured to iteratively update the initial effective pool in response to a comparison result indicating that the size of the current effective pool is less than the target number of interference options. The data pipeline is configured to: The generative model is used to generate a number of candidate distractors based on the question stem, wherein the number of candidate distractors to be generated is determined based on the difference between the size of the current valid pool and the target number of distractors; and Add the candidate interference options with a strength greater than zero from the candidate interference options to the current valid pool.

4. The computing device as claimed in claim 1, characterized in that, The predetermined number of candidate interference options includes a predetermined number of multiple candidate interference options, and The data pipeline is configured to: In response to determining that the strongest candidate interference option among the predetermined number of candidate interference options has a greater strength than the weakest candidate interference option, the weakest candidate interference option is replaced with the strongest candidate interference option.

5. The computing device as claimed in claim 1, characterized in that, The data pipeline is configured to iteratively update the initial valid pool a predetermined number of times.

6. The computing device as described in claim 5, characterized in that, The data pipeline is configured to: The candidate valid pool with the largest valid pool size is selected from the valid pool history, wherein the final valid pool is determined at least in part based on the candidate valid pool.

7. The computing device as claimed in claim 6, characterized in that, The data pipeline is configured to: The difficulty of the test set is determined based on the predetermined number of times the evaluation model solves and the total number of times the correct option is selected; and The candidate valid pool is selected from the historical records of the valid pools, based on the largest valid pool size and the highest corresponding test set difficulty.

8. The computing device as claimed in claim 6, characterized in that, The data pipeline is configured to: In response to determining that the size of the selected effective pool is less than the target number, the candidate effective pool is filled with at least a portion of the original interference options to obtain the final effective pool.

9. The computing device as claimed in claim 1, characterized in that, The data pipeline is configured to: If the generated answer is the same as the correct option or at least one of the distractors in the current valid pool, a new candidate distractor option is generated.

10. The computing device as claimed in claim 1, characterized in that, The computing device also includes the generation model and the evaluation model.

11. A method for optimizing multiple-choice questions, characterized in that, Includes the following steps: S1: Obtain the original multiple-choice questions from the dataset. The original multiple-choice questions include a stem and multiple options associated with the stem. The multiple options include a correct option and original distractor options. S2: Determine the initial valid pool of interference options, the initial valid pool comprising a subset of the original interference options; S3: Iteratively update the initial valid pool, wherein S3 includes: S31: Use a generative model to generate an answer based on the question stem, as a candidate distractor option associated with the question stem; S32: Generate a test set based on the correct option, the interference options in the current valid pool, and the candidate interference options; S33: Use an evaluation model to determine the predetermined number of times the current multiple-choice question will be answered, wherein the current multiple-choice question includes the question stem and the test set; S34: Based on the number of times the candidate interference option is selected and the total number of times the correct option is not selected in the solution using the evaluation model, determine the strength of each candidate interference option; S35: Based on the strength of the candidate interference options, selectively update the current valid pool; S36: Add the updated valid pool to the valid pool history; S4: Determine the final effective pool based on the historical records of the effective pool; and S5: Generate the final multiple-choice questions based on the question stem, the correct options, and the final valid pool. The method further includes: S6: Compare the size of the current effective pool with the target number of interference options. S3, in response to the result of the comparison, indicates that the size of the current effective pool is greater than or equal to the target number of interference options, and that this is executed. S3 further includes: S37: Identify the interference option with the lowest intensity in the currently valid pool; S31 includes: S312: Using the generative model, generate a predetermined number of candidate distractors based on the question stem. S32 includes: S322: Generate a test set based on the correct option, the interference options after excluding the interference option with the lowest intensity in the current valid pool, and the predetermined number of candidate interference options; and The S35 includes: S352: In response to determining that the strength of one of the predetermined number of candidate interference options is greater than that of the interference option with the lowest strength, the interference option with the lowest strength is replaced with the candidate interference option.

12. The method as described in claim 11, characterized in that, S2 includes: S21: Use the evaluation model to determine the predetermined number of times the original multiple-choice questions will be answered; and S22: Exclude interference options that were never selected by the evaluation model from the original interference options to determine the initial effective pool of interference options.

13. The method as described in claim 11, characterized in that, S3 responds to the result of the comparison by indicating that the size of the current effective pool is less than the target number of interference options, and this is executed. S31 includes: S311: Using the generative model, generate a number of candidate distractors based on the question stem, wherein the number of candidate distractors to be generated is determined based on the difference between the size of the current valid pool and the target number of distractors; and The S35 includes: S351: Add the candidate interference options with a strength greater than zero from the candidate interference options to the current valid pool.

14. The method as described in claim 11, characterized in that, The predetermined number of candidate interference options includes a predetermined number of multiple candidate interference options, and S352 includes: S3521: In response to determining that the strongest candidate interference option among the predetermined number of candidate interference options has a greater strength than the weakest interference option, the weakest interference option is replaced with the strongest candidate interference option.

15. The method as described in claim 11, characterized in that, S3 is executed a predetermined number of times.

16. The method as described in claim 15, characterized in that, S4 includes: S41: Select the candidate valid pool with the largest valid pool size from the valid pool history, wherein the final valid pool is determined at least in part based on the candidate valid pool.

17. The method as described in claim 16, characterized in that, S3 further includes: S38: Determine the difficulty of the test set based on the predetermined number of times the evaluation model has been solved and the total number of times the correct option has been selected; and S41 includes: S411: Select the candidate valid pool as the one with the largest valid pool size and the highest corresponding test set difficulty from the valid pool history.

18. The method as described in claim 16, characterized in that, S4 further includes: S42: In response to determining that the size of the selected effective pool is less than the target number, the candidate effective pool is filled with at least a portion of the original interference options to obtain the final effective pool.

19. The method as described in claim 11, characterized in that, S31 includes: S313: If the generated answer is the same as the correct option or at least one of the distractors in the current valid pool, regenerate the candidate distractor option.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 11-19.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 11-19.

Images