Reinforcement learning system for automatically generating interference items of shape completion and gap filling

Abstract

The invention discloses a reinforcement learning system for automatically generating a complete blank-filling interference item, which is characterized in that a fixed high reward is given to an artificial standard interference item through a double reward structure, a high-quality standard is defined, a dynamic reward is given to an interference candidate item generated by a model based on a confidence score, and the accuracy of the interference item is improved. The model can accurately judge the quality levels of the candidate items according to different reward feedbacks, and quality irregularity is reduced; the self-adaptive reward scaling module adjusts reward intensity according to model performance, amplifies reward and punishment signals in an initial training stage, helps the model to quickly learn basic generation capability, refines reward differences in a convergence stage, and improves quality stability of candidate items; besides, a self-adaptive reward scaling module continuously optimizes a dynamic reward mapping rule, a conversion parameter is iteratively adjusted by comparing historical and current conditions, a reward mechanism adapts to model change, effective guidance is provided for the model, the model more accurately captures high-quality interference item features and optimizes a generation process, and therefore relevant features are better learned.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a reinforcement learning system for automatically generating interference items for cloze questions. Background Art

[0002] In the context of the deep integration of educational assessment and natural language processing technology, cloze questions, as an important question type for language ability assessment, have a direct impact on the accuracy and effectiveness of the assessment in terms of the quality of their distractor generation. Traditional pre-trained language models can capture rich language knowledge and semantic information by learning from large amounts of text data, thereby generating distractor candidates with a certain degree of rationality. However, the existing distractor generation methods based on pre-trained language models still have many shortcomings: on the one hand, the quality of the distractor candidates generated by the model varies greatly, and some candidates may be too close to the correct answer, lack sufficient confusion, or do not match the context, affecting the effectiveness of the question; on the other hand, during the model training process, there is a lack of effective mechanisms to guide the model to learn the characteristics of high-quality distractor items, resulting in the distractor items generated by the model being difficult to reach the level of manual design. Summary of the Invention

[0003] In view of this, the present invention proposes a reinforcement learning system for automatically generating interference items for cloze questions, which can effectively solve the defects of the existing technology, such as the uneven quality of the generated interference candidates and the lack of an effective mechanism to guide the model to learn the characteristics of high-quality interference items.

[0004] The technical solution of the present invention is achieved as follows:

[0005] A reinforcement learning system for automatically generating distractor items for cloze questions, comprising:

[0006] Interference item candidate generation module, used to generate interference items including manually constructed standard interference items and interference candidates generated by pre-trained language models;

[0007] A dual reward structure module is used to assign a fixed high reward to manually constructed standard distractors and a dynamic reward to distractor candidates generated by the pre-trained language model. The comparison between the standard fixed reward and the dynamic reward of the distractor candidates generated by the pre-trained language model is used to guide the pre-trained language model to learn high-quality distractor features.

[0008] Adaptive reward scaling module, used by the reinforcement learning policy gradient calculation module to adjust the reward intensity of the pre-trained language model-generated items based on the pre-trained language model performance to guide the learning direction;

[0009] The inference module selects the optimal interference items from the candidates generated by the pre-trained language model based on the confidence of the pre-trained language model.

[0010] As a further optional solution of the reinforcement learning system for automatically generating distractor items for cloze questions, the distractor candidate generation module includes:

[0011] A standard interference item generation submodule is used to generate standard interference items;

[0012] The initial candidate generation submodule is used to generate initial interference candidates using the pre-trained language model;

[0013] The candidate item screening submodule is used to filter out the candidates that are completely identical in semantics or form to the standard distractors;

[0014] A supplementary generation submodule is used to continue generating until a sufficient number of candidate items are obtained when the remaining candidate items after screening are less than a preset number;

[0015] The training data construction submodule is used to include a preset number of candidate labels for each training sample, including a preset number of standard interference items and candidate items generated by the pre-trained language model.

[0016] As a further optional solution of the reinforcement learning system for automatically generating distractor items for cloze questions, the dual reward structure module includes:

[0017] The standard reward assignment submodule is used to assign a fixed high reward value to the manually constructed standard distractors. This fixed high reward value serves as the learning goal for the pre-trained language model to generate high-quality distractors.

[0018] A dynamic reward calculation submodule is used to assign a dynamic reward value to the interference candidate generated by the pre-trained language model based on the confidence score of the pre-trained language model as a valid interference item, wherein the dynamic reward value is adjusted as the model confidence level changes, and the higher the confidence level, the higher the dynamic reward value;

[0019] The reward comparison and learning guidance submodule is used to guide the pre-trained language model to learn the characteristics of high-quality interference items based on the comparison results between the standard fixed reward value and the dynamic reward value of the interference candidate items generated by the pre-trained language model.

[0020] As a further optional solution of the reinforcement learning system for automatically generating distractor items for cloze questions, the dynamic reward calculation submodule includes:

[0021] A confidence evaluation unit is used to input the candidate distractor to be evaluated and the context text of the cloze question into the pre-trained language model, calculate the generation probability of the candidate item in the cloze position based on the masked language modeling head of the pre-trained language model, and obtain the confidence score of the candidate distractor;

[0022] A dynamic reward mapping unit is used to dynamically adjust and assign a dynamic reward value corresponding to the candidate interference item according to the confidence score, wherein the dynamic reward value is adjusted as the confidence of the pre-trained language model changes, and the higher the confidence, the higher the dynamic reward value.

[0023] As a further optional solution of the reinforcement learning system for automatically generating distractors for cloze questions, the adaptive reward scaling module includes:

[0024] The performance monitoring submodule is used to obtain the performance indicators of the pre-trained language model in real time during the training phase;

[0025] Dynamic scaling factor calculation submodule, used to dynamically calculate the reward scaling factor based on performance indicators;

[0026] The reward intensity adjustment submodule is used to multiply the original dynamic reward value by the scaling factor to obtain the adjusted reward value;

[0027] The learning direction guidance submodule is used to feed back the adjusted reward value to the reinforcement learning policy gradient calculation module, which guides the learning direction of the pre-trained language model.

[0028] As a further optional solution of the reinforcement learning system for automatically generating distractor items for cloze questions, the dynamic scaling factor calculation submodule uses a sigmoid function to calculate the reward scaling factor, specifically:

[0029] ;

[0030] in, represents the reward scaling factor, is the minimum reward scaling value, is the maximum reward scaling value, is the sigmoid function, Controls the steepness of reward changes, represents the average loss of the current training stage, Represents the average loss value.

[0031] A reinforcement learning method for automatically generating distractor items for cloze questions, specifically comprising:

[0032] Generates interference candidates including manually constructed standard interference items and those generated by the pre-trained language model;

[0033] A fixed high reward is given to manually constructed standard distractors, and a dynamic reward is given to distractor candidates generated by the pre-trained language model. Based on the comparison between the standard fixed reward and the dynamic reward of distractor candidates generated by the pre-trained language model, the pre-trained language model is guided to learn high-quality distractor features.

[0034] The reinforcement learning policy gradient calculation module adjusts the reward intensity of the pre-trained language model-generated items based on the pre-trained language model performance to guide the learning direction;

[0035] The optimal interference items are selected from the candidates generated by the pre-trained language model based on the confidence of the pre-trained language model.

[0036] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the reinforcement learning method for automatically generating distractor items for cloze questions are implemented.

[0037] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned reinforcement learning method for automatically generating distractor items for cloze questions.

[0038] The beneficial effects of the present invention are as follows: fixed high rewards are given to artificially constructed standard interference items through the dual reward structure module, and dynamic rewards are given to interference candidates generated by the pre-trained language model. The standard interference items are carefully designed by humans and have high deceptiveness and rationality. Giving fixed high rewards can clarify the standards of high-quality interference items, and the interference candidates generated by the model are evaluated through dynamic rewards. The dynamic rewards are given based on the model's confidence score that the candidate is a valid interference item. This distinction enables the model to more accurately judge the quality level of the candidate items based on different reward feedback when generating interference candidates, and avoids confusing candidates with large quality differences, thereby ensuring the relative stability of the quality of the generated candidate items to a certain extent and reducing the uneven quality. At the same time, the adaptive reward scaling module adjusts the reward intensity of the model-generated items based on the model performance. At different stages of model training, its ability and level of generating interference candidates are different. The adaptive reward scaling module can adjust the reward intensity of the model-generated items based on the model training performance. The performance indicators in the training phase (such as the average loss value) dynamically adjust the amplitude of the reward value. During the initial training of the model, the penalty signal of low-confidence items is amplified and the incentive signal of high-confidence items is strengthened, so that the model can quickly learn the basic interference item generation ability and generate relatively reasonable and high-quality candidates. During the model convergence stage, the reward differences are refined so that the model can further distinguish and optimize the quality of the candidates, making the quality of the generated interference candidates more stable and high-quality. In addition, the adaptive reward scaling module continuously optimizes the dynamic reward mapping rules during the model training process. By comparing the historical reward distribution effects with the current model generation quality, it iteratively adjusts the conversion parameters from confidence scores to reward values. This enables the reward mechanism to continuously adapt to the changes of the model as the model is trained, and always provides the model with effective guidance signals. In the process of continuously adjusting the reward parameters, the model can more accurately capture the characteristics of high-quality interference items and optimize its own generation process based on these characteristics, thereby better learning the characteristics of high-quality interference items. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A schematic diagram of the composition of a reinforcement learning system for automatically generating distractor items for cloze questions according to the present invention;

[0041] Figure 2 Schematic diagram of the composition of the interference candidate generation module of the present invention;

[0042] Figure 3 This is a schematic diagram of the composition of the dual reward structure module of the present invention;

[0043] Figure 4 Schematic diagram of the composition of the adaptive reward scaling module of the present invention;

[0044] Figure 5 This is a schematic diagram of the composition of the dynamic reward calculation submodule of the present invention. DETAILED DESCRIPTION

[0045] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] refer to Figures 1 to 5 ,A reinforcement learning system for automatically generating distractors for cloze questions, including a distractor candidate generation module, a dual reward structure module, an adaptive reward scaling module and a reasoning module.

[0047] A distractor candidate generation module is configured to generate distractor candidate items including manually constructed standard distractors and distractor candidate items generated by a pre-trained language model. In some embodiments, the distractor candidate generation module includes:

[0048] A standard interference item generation submodule is used to generate standard interference items;

[0049] The initial candidate generation submodule is used to generate initial interference candidates using the pre-trained language model;

[0050] The candidate item screening submodule is used to filter out the candidates that are completely identical in semantics or form to the standard distractors;

[0051] A supplementary generation submodule is used to continue generating until a sufficient number of candidate items are obtained when the remaining candidate items after screening are less than a preset number;

[0052] The training data construction submodule is used to include a preset number of candidate labels for each training sample, including a preset number of standard interference items and candidate items generated by the pre-trained language model.

[0053] Specifically, the standard distractor generation submodule is specifically used to generate standard distractors. Standard distractors are usually artificially constructed or rigorously screened distractors with high deceptiveness and rationality. They provide a quality benchmark for the entire distractor set. By introducing standard distractors, the generated distractor set is ensured to contain high-quality options, improving the overall quality of the distractors and making the generated cloze test more valuable.

[0054] Use pre-trained language models to generate initial distractor candidates. Leveraging its learning on large-scale text data, pre-trained language models can capture rich linguistic knowledge and semantic information, thereby generating diverse initial candidates. This provides rich material for subsequent screening and supplementation, increases the diversity of distractor items, and helps meet the needs of different cloze test questions.

[0055] The initially generated candidate items are screened to remove those that completely duplicate the semantics or form of the standard distractors. This step avoids redundancy in distractors and ensures the uniqueness and effectiveness of each distractor. At the same time, the screening process also helps to remove some low-quality or non-compliant candidates, thereby improving the overall quality of the distractor set.

[0056] When the number of remaining candidates after screening is less than the preset number, generation continues until a sufficient number of candidates are obtained. This mechanism ensures the stability of the number of distractors and meets the requirements of the cloze test for the number of distractors. Moreover, during the supplementary generation process, the pre-trained language model can further optimize the generation strategy based on the previous screening results and contextual information to generate distractors that better meet the needs.

[0057] A dual reward structure module is configured to assign a fixed high reward to manually constructed standard distractors and a dynamic reward to distractor candidates generated by the pre-trained language model, and to guide the pre-trained language model to learn high-quality distractor features based on a comparison of the standard fixed reward and the dynamic reward of the distractor candidates generated by the pre-trained language model. In some embodiments, the dual reward structure module includes:

[0058] The standard reward assignment submodule is used to assign a fixed high reward value to the manually constructed standard distractors. This fixed high reward value serves as the learning goal for the pre-trained language model to generate high-quality distractors.

[0059] A dynamic reward calculation submodule is used to assign a dynamic reward value to the interference candidate generated by the pre-trained language model based on the confidence score of the pre-trained language model as a valid interference item, wherein the dynamic reward value is adjusted as the model confidence level changes, and the higher the confidence level, the higher the dynamic reward value;

[0060] The reward comparison and learning guidance submodule is used to guide the pre-trained language model to learn the characteristics of high-quality interference items based on the comparison results between the standard fixed reward value and the dynamic reward value of the interference candidate items generated by the pre-trained language model.

[0061] Specifically, the standard reward assignment submodule assigns a fixed high reward value to artificially constructed standard distractors and uses it as a learning target for the pre-trained language model to generate high-quality distractors. Standard distractors are carefully designed to be highly confusing, reasonable, and well-matched with the context. They represent the standard of high-quality distractors. By assigning a fixed high reward, the model is provided with a clear and high-quality learning benchmark, enabling it to clearly understand what kind of distractors are high-quality during training, thereby guiding the model to learn and optimize towards generating similar high-quality distractors, effectively improving the quality benchmark of the model-generated distractors.

[0062] The dynamic reward calculation submodule assigns a dynamic reward value based on the confidence score of the candidate as a valid distractor according to the pre-trained language model. The dynamic reward value is adjusted as the model confidence changes. The higher the confidence, the higher the dynamic reward value. This dynamic reward mechanism can reflect the model's assessment of the quality of its own generated candidate items in real time and accurately. During the training process, the model can adjust its generation strategy in a timely manner based on the dynamic reward value, giving more attention and strengthening to candidates with high confidence, and improving or eliminating candidates with low confidence. This enables the model to better adapt to different contexts and question requirements, enhances its adaptability to various situations, and thus generates distractors that better meet actual needs.

[0063] The reward comparison and learning guidance submodule guides the pre-trained language model to learn the characteristics of high-quality interference items based on the comparison results of the standard fixed reward value and the dynamic reward value of the interference candidate items generated by the pre-trained language model. Through comparison, the model can intuitively understand the gap between the interference items generated by itself and the standard interference items, and clarify the direction that needs improvement. This comparison guidance mechanism can enhance the model's learning of high-quality interference item features, such as semantic confusion, grammatical correctness, and logical coherence with the context, so that the model can more accurately capture and use these features in the subsequent generation process, further improving the quality of generated interference items.

[0064] In some embodiments, the dynamic reward calculation submodule includes:

[0065] A confidence evaluation unit is used to input the candidate distractor to be evaluated and the context text of the cloze question into the pre-trained language model, calculate the generation probability of the candidate item in the cloze position based on the masked language modeling head of the pre-trained language model, and obtain the confidence score of the candidate distractor;

[0066] A dynamic reward mapping unit is used to dynamically adjust and assign a dynamic reward value corresponding to the candidate interference item according to the confidence score, wherein the dynamic reward value is adjusted as the confidence of the pre-trained language model changes, and the higher the confidence, the higher the dynamic reward value.

[0067] Specifically, the confidence assessment unit inputs the candidate distractor to be evaluated and the context text of the cloze test into the pre-trained language model, and calculates the generation probability of the candidate in the fill-in-the-blank position based on its masked language modeling head, thereby obtaining a confidence score. This assessment method fully utilizes the pre-trained language model's understanding and modeling capabilities of language. The masked language modeling head can predict the most appropriate vocabulary for the fill-in-the-blank position based on contextual information. By calculating the generation probability of the candidate, the degree of match between the candidate and the context and its rationality as a distractor can be accurately measured. For example, if the generation probability of the candidate in the context is high, it means that it is more consistent with the semantics and grammar of the context and has a high degree of deceptiveness, thus enabling a more accurate assessment of the quality of the candidate.

[0068] The dynamic reward mapping unit dynamically adjusts and assigns dynamic reward values ​​corresponding to candidate interference items based on the confidence scores, and the reward values ​​are adjusted as the confidence of the pre-trained language model changes. The higher the confidence, the higher the dynamic reward value. This dynamic adjustment mechanism makes the reward closely related to the quality of the candidate items. During the model training process, high-confidence candidates receive higher rewards, which will guide the model to generate more similar high-quality candidates in subsequent generations, while low-confidence candidates receive lower rewards, prompting the model to reduce the generation of such poor-quality candidates. Through this dynamic feedback, the model can continuously optimize its own generation strategy, gradually learn the skills and methods to generate high-quality interference items, and improve the overall quality of generated interference items.

[0069] An adaptive reward scaling module is used to reinforce the learning policy gradient calculation module to adjust the reward intensity of the pre-trained language model generated items based on the pre-trained language model performance to guide the learning direction. In some embodiments, the adaptive reward scaling module includes:

[0070] The performance monitoring submodule is used to obtain the performance indicators of the pre-trained language model in real time during the training phase;

[0071] Dynamic scaling factor calculation submodule, used to dynamically calculate the reward scaling factor based on performance indicators;

[0072] The reward intensity adjustment submodule is used to multiply the original dynamic reward value by the scaling factor to obtain the adjusted reward value;

[0073] The learning direction guidance submodule is used to feed back the adjusted reward value to the reinforcement learning policy gradient calculation module, which guides the learning direction of the pre-trained language model.

[0074] Specifically, the performance monitoring submodule can obtain the performance indicators of the pre-trained language model during the training phase in real time, such as the loss function value and accuracy rate. These performance indicators reflect the performance of the model in the current training state and provide a key data foundation for the subsequent dynamic calculation of the reward scaling factor. Through real-time monitoring, changes in model performance can be captured in a timely manner, so that reward adjustments can keep pace with the model's learning progress and ensure that the reward mechanism always matches the actual needs of the model.

[0075] The dynamic scaling factor calculation submodule dynamically calculates the reward scaling factor based on the performance indicators obtained by the performance monitoring submodule. In the early stages of model training, performance indicators may be poor. At this time, the calculated scaling factor will amplify the penalty signal of low-confidence items and strengthen the incentive signal of high-confidence items, prompting the model to quickly learn basic features and accelerate convergence. As training progresses, the model performance gradually improves, and the scaling factor will be adjusted accordingly to refine the reward differences, allowing the model to more accurately distinguish the rewards of candidates of different qualities and further optimize the generation strategy. This dynamic calculation method enables the reward scaling factor to flexibly change according to the actual performance of the model, improving the flexibility and adaptability of reward adjustment;

[0076] The reward intensity adjustment submodule multiplies the original dynamic reward value by a scaling factor to obtain an adjusted reward value. This allows precise adjustment of reward intensity based on model performance. For example, when model performance is poor, the reward difference is appropriately amplified to allow the model to more clearly understand the difference between high-quality and low-quality candidates. When the model is nearing convergence, the reward difference is reduced, allowing the model to further optimize the generated results with subtle changes in rewards. This precise reward adjustment can provide more targeted learning feedback to the model, helping it better understand which interference items are more suitable for the requirements.

[0077] The learning direction guidance submodule feeds back the adjusted reward value to the reinforcement learning policy gradient calculation module to guide the learning direction of the pre-trained language model. The reinforcement learning policy gradient calculation module updates the model parameters according to the adjusted reward value, so that the model is optimized towards generating higher-quality interference items. Through continuous feedback and adjustment, the model can gradually learn the characteristics and generation rules of high-quality interference items, improve the quality and confusion of the generated interference items, and thus better meet the requirements of cloze questions for interference items.

[0078] In some embodiments, the dynamic scaling factor calculation submodule uses a sigmoid function to calculate the reward scaling factor, specifically:

[0079] ;

[0080] in, represents the reward scaling factor, is the minimum reward scaling value, is the maximum reward scaling value, is the sigmoid function, Controls the steepness of reward changes, represents the average loss of the current training stage, Represents the average loss value.

[0081] Specifically, in the formula Represents the average loss of the current training stage, which reflects the performance of the model in the training process in real time. As the training progresses, It will change gradually. In the early stage of model training, the prediction effect is poor. Larger, as the model continues to learn, Will gradually decrease, the reward scaling factor The calculation depends on , so it can be dynamically adjusted according to the different training states of the model. For example, in the early stage of training, the larger will make The value of is large, combined with And the role of the sigmoid function, the reward scaling factor will be in a suitable range to amplify the reward difference and help the model learn quickly, and in the later stage of training, If the reward is smaller, the reward scaling factor will be adjusted accordingly to refine the reward difference and further optimize the model;

[0082] The sigmoid function has the characteristic of smoothness. This part is done by the sigmoid function The sigmoid function can map the input to between 0 and 1, and the change process is smooth, which makes the adjustment of the reward scaling factor not sudden, but as This smooth adjustment avoids large fluctuations in the reward amplitude and provides a relatively stable learning environment for the model, enabling the model to converge more stably and avoiding training instability caused by sudden changes in rewards.

[0083] In the formula Control the steepness of the reward change by adjusting The value of can change the shape of the sigmoid function, thereby affecting the reward scaling factor If you want to change the rate of When the change is small, the reward scaling factor has a large change, which can be increased On the contrary, if you want the reward scaling factor to change more smoothly, you can reduce This flexibility enables the technical solution to adjust the sensitivity of reward changes according to specific model and task requirements, and better guide model training;

[0084] Represents the average loss value, which sets a benchmark for adjusting the reward scaling factor. Greater than hour, is positive, it will have a positive impact on the reward scaling factor; when Less than hour, If it is negative, it will have a reverse effect. , it can be determined under what circumstances to magnify or reduce the reward, so that the calculation of the reward scaling factor is more in line with the actual needs of model training, further improving the effect of model training.

[0085] The inference module selects the optimal interference items from the candidates generated by the pre-trained language model based on the confidence of the pre-trained language model.

[0086] Specifically, a preset number of distractor candidates are generated for each cloze question, candidates that are exactly the same as the correct answer are filtered out, duplicates are removed, and they are sorted according to model confidence. The top N candidates with the highest confidence are selected as the final distractor, where N is a preset natural number.

[0087] A reinforcement learning method for automatically generating distractor items for cloze questions, specifically comprising:

[0088] Generates interference candidates including manually constructed standard interference items and those generated by the pre-trained language model;

[0089] A fixed high reward is given to manually constructed standard distractors, and a dynamic reward is given to distractor candidates generated by the pre-trained language model. Based on the comparison between the standard fixed reward and the dynamic reward of distractor candidates generated by the pre-trained language model, the pre-trained language model is guided to learn high-quality distractor features.

[0090] The reinforcement learning policy gradient calculation module adjusts the reward intensity of the pre-trained language model-generated items based on the pre-trained language model performance to guide the learning direction;

[0091] The optimal interference items are selected from the candidates generated by the pre-trained language model based on the confidence of the pre-trained language model.

[0092] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the reinforcement learning method for automatically generating distractor items for cloze questions are implemented.

[0093] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned reinforcement learning method for automatically generating distractor items for cloze questions.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A reinforcement learning system for automatically generating distractor items for cloze questions, characterized in that: include: Interference item candidate generation module, used to generate interference items including manually constructed standard interference items and interference candidates generated by pre-trained language models; A dual reward structure module is used to assign a fixed high reward to manually constructed standard distractors and a dynamic reward to distractor candidates generated by the pre-trained language model. The comparison between the standard fixed reward and the dynamic reward of the distractor candidates generated by the pre-trained language model is used to guide the pre-trained language model to learn high-quality distractor features. Adaptive reward scaling module, used by the reinforcement learning policy gradient calculation module to adjust the reward intensity of the pre-trained language model-generated items based on the pre-trained language model performance to guide the learning direction; The inference module selects the optimal interference items from the candidates generated by the pre-trained language model based on the confidence of the pre-trained language model.

2. The reinforcement learning system for automatically generating distractor items for cloze questions according to claim 1, characterized in that: The interference item candidate generation module includes: A standard interference item generation submodule is used to generate standard interference items; The initial candidate generation submodule is used to generate initial interference candidates using the pre-trained language model; The candidate item screening submodule is used to filter out the candidates that are completely identical in semantics or form to the standard distractors; A supplementary generation submodule is used to continue generating until a sufficient number of candidate items are obtained when the remaining candidate items after screening are less than a preset number; The training data construction submodule is used to include a preset number of candidate labels for each training sample, including a preset number of standard interference items and candidate items generated by the pre-trained language model.

3. The reinforcement learning system for automatically generating distractor items for cloze questions according to claim 2, characterized in that: The dual reward structure module includes: The standard reward assignment submodule is used to assign a fixed high reward value to the manually constructed standard distractors. This fixed high reward value serves as the learning goal for the pre-trained language model to generate high-quality distractors. A dynamic reward calculation submodule is used to assign a dynamic reward value to the interference candidate generated by the pre-trained language model based on the confidence score of the pre-trained language model as a valid interference item, wherein the dynamic reward value is adjusted as the model confidence level changes, and the higher the confidence level, the higher the dynamic reward value; The reward comparison and learning guidance submodule is used to guide the pre-trained language model to learn the characteristics of high-quality interference items based on the comparison results between the standard fixed reward value and the dynamic reward value of the interference candidate items generated by the pre-trained language model.

4. The reinforcement learning system for automatically generating distractor items for cloze questions according to claim 3, characterized in that: The dynamic reward calculation submodule includes: A confidence evaluation unit is used to input the candidate distractor to be evaluated and the context text of the cloze question into the pre-trained language model, calculate the generation probability of the candidate item in the cloze position based on the masked language modeling head of the pre-trained language model, and obtain the confidence score of the candidate distractor; A dynamic reward mapping unit is used to dynamically adjust and assign a dynamic reward value corresponding to the candidate interference item according to the confidence score, wherein the dynamic reward value is adjusted as the confidence of the pre-trained language model changes, and the higher the confidence, the higher the dynamic reward value.

5. The reinforcement learning system for automatically generating distractor items for cloze questions according to claim 4, characterized in that: The adaptive reward scaling module includes: The performance monitoring submodule is used to obtain the performance indicators of the pre-trained language model in real time during the training phase; Dynamic scaling factor calculation submodule, used to dynamically calculate the reward scaling factor based on performance indicators; The reward intensity adjustment submodule is used to multiply the original dynamic reward value by the scaling factor to obtain the adjusted reward value; The learning direction guidance submodule is used to feed back the adjusted reward value to the reinforcement learning policy gradient calculation module, which guides the learning direction of the pre-trained language model.

6. The reinforcement learning system for automatically generating distractor items for cloze questions according to claim 5, characterized in that: The dynamic scaling factor calculation submodule uses the sigmoid function to calculate the reward scaling factor, specifically: ； in, represents the reward scaling factor, is the minimum reward scaling value, is the maximum reward scaling value, is the sigmoid function, Controls the steepness of reward changes, represents the average loss of the current training stage, Represents the average loss value.

7. A reinforcement learning method for automatically generating distractors for cloze test questions, characterized in that: Specifically include: Generates interference candidates including manually constructed standard interference items and those generated by the pre-trained language model; A fixed high reward is given to manually constructed standard distractors, and a dynamic reward is given to distractor candidates generated by the pre-trained language model. Based on the comparison between the standard fixed reward and the dynamic reward of distractor candidates generated by the pre-trained language model, the pre-trained language model is guided to learn high-quality distractor features. The reinforcement learning policy gradient calculation module adjusts the reward intensity of the pre-trained language model-generated items based on the pre-trained language model performance to guide the learning direction; The optimal interference items are selected from the candidates generated by the pre-trained language model based on the confidence of the pre-trained language model.

8. A computing device, characterized in that The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method implements the steps of the reinforcement learning method for automatically generating distractor items for cloze questions as described in claim 7.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, implements the steps of the reinforcement learning method for automatically generating distractor items for cloze questions as described in claim 7.

Images