A multi-interference term generation method based on a large language model and reinforcement learning

Abstract

The application discloses a multi-interference term generation method based on a large language model and reinforcement learning, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining EQG-RACE data, dividing the EQG-RACE data into a training set and a verification set; establishing an interference term generation model based on a text instruction; inputting the training set into the interference term generation model and performing supervised fine-tuning processing to obtain a supervised fine-tuned interference term generation model; inputting the verification set into the supervised fine-tuned interference term generation model, and performing sorting according to a generation result; constructing a reward model, obtaining a training data set of the reward model according to a sorting result, and calculating a pairwise ranking loss; further training the supervised fine-tuned interference term generation model based on the reward model and reinforcement learning to obtain multi-interference terms with the highest quality. The application reaches a level satisfying real test question setting requirements in terms of interference term generation quality.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a method for generating multiple perturbation terms based on a large language model and reinforcement learning. Background Technology

[0002] English reading comprehension questions typically appear on exams in the form of multiple-choice questions. The options other than the correct answer are called distractors. Distractors are placed alongside the correct answer to confuse test-takers. Therefore, distractors are an important component of reading comprehension questions, and their quality determines the quality of a single-choice question. English is a key subject for students of all ages in my country, requiring a large number of high-quality new reading comprehension multiple-choice questions for practice and assessment each year. Among these, the most tedious and labor-intensive part of the writing process is creating the distractors.

[0003] In recent years, there has been only a limited amount of research. Generally speaking, previous work has applied the sequence-to-sequence model, the most common text generation task, to the task of automatically generating distractors. The sequence-to-sequence model consists of an encoder and a decoder. Researchers have fed the article, question, and answer as input to the encoder, and the model uses encoder-decoder attention during decoding to select appropriate contextual information from the input to generate distractors. This invention, however, considers that the model's input includes three different types of input: article, question, and answer. Therefore, it adjusts the encoder structure and the model's attention mechanism, hoping that the model can correctly model the relationship between distractors and questions / answers, enhance the model's ability to understand long texts, and find the content basis for generating distractors within the text.

[0004] In order to reduce the workload of test creators and improve their work efficiency, it is necessary to study artificial intelligence-based automatic generation technology for distractors in real-world test creation scenarios.

[0005] Therefore, proposing a multi-perturbation generation method based on a large language model and reinforcement learning to solve the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a method for generating multiple perturbation terms based on a large language model and reinforcement learning, which is used to solve the technical problems existing in the prior art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for generating multiple perturbations based on a large language model and reinforcement learning includes the following steps:

[0009] Obtain EQG-RACE data and divide it into training and validation sets;

[0010] Establish a distractor generation model based on text instructions;

[0011] The training set is input into the disturbance term generation model and supervised fine-tuning is performed to obtain the supervised fine-tuned disturbance term generation model.

[0012] The validation set is input into the supervised fine-tuned interference terms to generate the model, and the generated results are sorted.

[0013] Construct a reward model, obtain the training dataset for the reward model based on the ranking results, and calculate the pairwise ranking loss;

[0014] The interference generation model is further trained with supervised fine-tuning based on the reward model and reinforcement learning to obtain the highest quality multiple interference items.

[0015] Optionally, the specific content of establishing a distractor generation model based on text instructions is as follows:

[0016] First, the reading comprehension passage, questions, answers, and the number of distractors are written into a text instruction template to obtain text instructions. Then, the text instructions are input into the distractor generation model after fine-tuning the instructions. Based on the required response format and the number of distractors to be generated in the instructions, a response text containing a specified number of distractors is generated. Finally, the generated distractors are extracted from the response text.

[0017] Optionally, the specific content of the supervised fine-tuned disturbance generation model is as follows:

[0018] Using text commands as existing text, predict each word in the response text. When generating a model with supervised fine-tuning of perturbations, combine the text commands and response text into a target text. Employ Teacher-Forcing technique to predict the word at the next time step based on the target's historical context, thus transforming the text generation task into a word classification task. The classification target at each time step is the next word in the target text, and the classification loss at each time step is calculated.

[0019] Optionally, the specific content of the training dataset for the reward model based on the ranking results is as follows: construct the interference evaluation criteria, articles, questions, answers and interference items into text instructions, use ChatGPT to rank the samples, and obtain the ranking results to construct the training dataset.

[0020] Optionally, sorting can be performed based on the evaluation criteria for individual interference items.

[0021] Optional evaluation criteria for individual interference items include: legitimacy and deceptiveness.

[0022] Optionally, the supervised fine-tuned interference generation model and reward model both include: a word embedding layer, several decoders, and a linear layer.

[0023] Optionally, the specific content of training the reward model based on pairwise ranking loss is as follows:

[0024] The reward model combines the textual instructions for individual distractors and the distractors themselves into the input text, specifically:

[0025] Given a reading comprehension passage P, a reading comprehension question Q constructed based on the passage, the correct answer A, and distractors D generated by a distractor generation model, the input text T of the reward model is denoted as T = [P; Q; A; D]. Given an input text of length n, T = (t1, t2, ..., t... n The input reward model first passes through a word embedding layer to obtain the feature matrix M, as shown in the formula:

[0026] M = Embedding(T), M ∈ R n×d

[0027] Where d is the feature dimension of the hidden layer of the model;

[0028] Assuming the reward model has L decoders, the output of the last decoder is represented as:

[0029] H L =Decorder L (H L-1 ), H L ∈R n×d

[0030] Among them, H i This is the output of the i-th decoder;

[0031] The output of the final decoder layer contains the hidden layer vector corresponding to each input word, and is therefore represented as H. L = (h1, h2, ..., h n ),h i ∈R d Extract the hidden layer vector corresponding to the last word. With learnable parameters Multiply to get the bonus points, the formula is:

[0032]

[0033] Therefore, the formula for calculating pairwise ranking loss is:

[0034]

[0035] Where, r aThe reward score r is the score awarded to the highest-ranking distractor. b The bonus points awarded to low-ranking distractors.

[0036] Optionally, the specific details of further training a supervised fine-tuned disturbance generation model based on the reward model and reinforcement learning are as follows:

[0037] First, several corresponding interference items are generated according to the text instructions. Then, the interference items are input into the reward model to obtain the reward score for the quality of a single interference item. At the same time, the diversity score among multiple interference items is calculated. The two scores are input into the objective function to obtain the final score. Finally, the score of the objective function is optimized by gradient descent.

[0038] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method for generating multiple perturbation terms based on a large language model and reinforcement learning, the beneficial effects of which are:

[0039] 1) Based on the model and training method of this invention, the quality of generated distractors reaches a level that meets the needs of real-world question setting;

[0040] 2) The method of the present invention can reduce the workload of the question setter and improve his work efficiency. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 A flowchart of a multi-interference generation method based on a large language model and reinforcement learning provided by the present invention;

[0043] Figure 2 The flowchart for generating interference terms provided by this invention;

[0044] Figure 3 This is a schematic diagram of supervised fine-tuning provided by the present invention;

[0045] Figure 4 A schematic diagram of the reward model provided by this invention;

[0046] Figure 5 This is a schematic diagram of the reinforcement learning process provided by the present invention;

[0047] Figure 6 This is a statistical chart showing the ranking of interference items based on manual evaluation, provided by the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] See Figure 1 As shown, this invention discloses a method for generating multiple perturbation terms based on a large language model and reinforcement learning, comprising the following steps:

[0050] Obtain EQG-RACE data and divide it into training and validation sets;

[0051] Establish a distractor generation model based on text instructions;

[0052] The training set is input into the disturbance term generation model and supervised fine-tuning is performed to obtain the supervised fine-tuned disturbance term generation model.

[0053] The validation set is input into the supervised fine-tuned interference terms to generate the model, and the generated results are sorted.

[0054] Construct a reward model, obtain the training dataset for the reward model based on the ranking results, and calculate the pairwise ranking loss;

[0055] The interference generation model is further trained with supervised fine-tuning based on the reward model and reinforcement learning to obtain the highest quality multiple interference items.

[0056] Specifically, multiple-choice questions in English reading comprehension exams typically have multiple distractors, which, along with the correct answer, constitute the answer choices. This research aims to generate multiple high-quality distractors. Specifically, given a reading comprehension passage P, a reading comprehension question Q, and the correct answer A, the task of generating n distractors is defined by the following formula: Where D represents the text containing n distractors, D = (D1, D2, ..., Dn). n ).

[0057] Further, see Figure 2 As shown, the specific content of establishing a text-based distractor generation model is as follows: First, the reading comprehension article, questions, answers, and the number of distractors are written into the text instruction template to obtain text instructions. Then, the text instructions are input into the distractor generation model after instruction fine-tuning. According to the response format and the number of distractors required in the instructions, a response text containing a specified number of distractors is generated. Finally, the generated distractors are extracted from the response text.

[0058] Specifically, the text instruction template includes the definition and requirements of the distractor generation task, especially specifying the deceptiveness requirements of each distractor and the diversity requirements among multiple distractors. The translated text instruction template is shown in Table 1, while the actual instruction template is in English.

[0059] Table 1. Text Instruction Template for Interference Item Generation Task

[0060]

[0061] In practical use, replace "{{N}}" in the template with the number of distractors to generate, replace "{{passage}}" with the actual article content, replace "{{question}}" with the question, and replace "{{answer}}" with the answer to obtain the text instruction. Input the written text instruction into the distractor generation model. The distractor generation model generates a text containing the specified number of distractors based on the required response format and distractor generation number in the instruction. Finally, it extracts the generated distractors from the text.

[0062] Further, see Figure 3 As shown, the specific content of the supervised fine-tuned disturbance term generation model is as follows:

[0063] Using text commands as existing text, predict each word in the response text. When generating a model with supervised fine-tuning of perturbations, combine the text commands and response text into a target text. Employ Teacher-Forcing technique to predict the word at the next time step based on the target's historical context, thus transforming the text generation task into a word classification task. The classification target at each time step is the next word in the target text, and the classification loss at each time step is calculated.

[0064] Furthermore, based on the ranking results, the specific content of the training dataset for the reward model is as follows: the evaluation criteria for distractors, articles, questions, answers, and distractors are constructed as text instructions, and ChatGPT is used to rank the samples, as shown in Table 2, to obtain the ranking results and thus construct the training dataset.

[0065] Table 2. Text command template for sorting interference items using ChatGPT

[0066]

[0067] Furthermore, the items are ranked according to the evaluation criteria for individual interference items.

[0068] Furthermore, the evaluation criteria for individual interference items include: legitimacy and deceptiveness.

[0069] Specifically, in addition to the criteria for the legitimacy and deceptiveness of a single distractor, there is also the diversity among multiple distractors. These three points constitute the evaluation criteria for distractors.

[0070] Furthermore, both the supervised fine-tuned interference generation model and the reward model include: a word embedding layer, several decoders, and a linear layer.

[0071] See Figure 4 The diagram shown is a schematic of the reward model.

[0072] Furthermore, the specific details of training the reward model based on pairwise ranking loss are as follows:

[0073] The reward model combines the textual instructions for individual distractors and the distractors themselves into the input text, specifically:

[0074] Given a reading comprehension passage P, a reading comprehension question Q constructed based on the passage, the correct answer A, and distractors D generated by a distractor generation model, the input text T of the reward model is denoted as T = [P; Q; A; D]. Given an input text of length n, T = (t1, t2, ..., t... n The input reward model first passes through a word embedding layer to obtain the feature matrix M, as shown in the formula:

[0075] M = Embedding(T), M ∈ R n×d

[0076] Where d is the feature dimension of the hidden layer of the model;

[0077] Assuming the reward model has L decoders, the output of the last decoder is represented as:

[0078] H L =Decorder L (H L-1 ), H L ∈R n×d

[0079] Among them, H i This is the output of the i-th decoder;

[0080] The output of the final decoder layer contains the hidden layer vector corresponding to each input word, and is therefore represented as H. L = (h1, h2, ..., h n ),h i ∈R d Extract the hidden layer vector corresponding to the last word. With learnable parameters Multiply to get the bonus points, the formula is:

[0081]

[0082] Therefore, the formula for calculating pairwise ranking loss is:

[0083]

[0084] Where, r a The reward score r is the score awarded to the highest-ranking distractor. b The bonus points awarded to low-ranking distractors.

[0085] Specifically, first, the difference in reward scores between the high and low ranking interference items is calculated. Then, the difference is input into the sigmoid function to obtain a value between 0 and 1. Finally, the negative logarithm is calculated as the loss value.

[0086] Further, see Figure 5 As shown, the specific content of the interference term generation model, which is further trained and fine-tuned based on the reward model and reinforcement learning, is as follows:

[0087] First, several corresponding interference items are generated according to the text instructions. Then, the interference items are input into the reward model to obtain the reward score for the quality of a single interference item. At the same time, the diversity score among multiple interference items is calculated. The two scores are input into the objective function to obtain the final score. Finally, the score of the objective function is optimized by gradient descent.

[0088] Specifically, this invention uses proximal policy optimization as the reinforcement learning algorithm. Let the text instruction received by the interference model be x, and the generated n interference terms be d = (d1, d2, ..., dn). n The objective function formula is shown below:

[0089]

[0090] In the formula, To generate a model that receives interference terms from reinforcement learning training; π SFT For the supervised fine-tuning of the disturbance term generation model; D KL denoted as Klbek-Leibler divergence, or KL divergence; D is the diversity score; α, β, γ, and δ are all hyperparameters.

[0091] In summary, the objective function consists of the mean reward score of each perturbation, the minimum reward score of each perturbation, the diversity score among multiple perturbations, and the KL divergence between the perturbation generation results in the reinforcement learning and supervised fine-tuning perturbation generation models.

[0092] Specifically, in the reinforcement learning process, two sets of hyperparameters for the objective function are used to train two interference generation models with different emphases on the quality of a single interference and the diversity of multiple interferences.

[0093] Generative models that focus on the quality of individual interference items are called During reinforcement learning, the weight of the mean reward score is α = 0.2, the weight of the minimum reward score is β = 0.1, and the weight of the diversity score is γ = 0.2.

[0094] Generative models that emphasize the diversity of multiple interference terms are called During reinforcement learning, the weight of the mean reward score is α = 0.2, and the weight of the minimum reward score is β = 0.

[0095] After experimental comparison, DISTINCT-1 was selected as the diversity score, resulting in the best training effect. Therefore, both models mentioned above use DISTINCT-1 as the diversity score when receiving reinforcement learning.

[0096] The supervised fine-tuning of the disturbance generation model, reward model, and reinforcement learning all employ LoRA as an efficient parameter fine-tuning method. The disturbance generation model after supervised fine-tuning is called DGRL. SFT .

[0097] Specifically, the comparative experiments are as follows:

[0098] HSA: It uses a hierarchical encoder to model long texts at both the word and sentence levels. A static attention mechanism is also designed to prevent the model from generating distractors that are too similar to the answer.

[0099] CHN: Building upon HSA, we propose a hierarchical encoder enhanced with a collaborative attention mechanism to better capture the relationship between articles and questions, thereby guiding the decoder to generate more coherent distractors. By calculating semantic similarity loss, we make the generated distractors more relevant to the articles.

[0100] HMD-Net: It uses an attention mechanism with gating in the encoder and uses multiple decoders to generate diverse noise items.

[0101] EDGE: An encoding module for articles and questions was designed based on attention and gating mechanisms to improve the generation of distractors.

[0102] TMCA: Based on HSA, topic information and topic attention are introduced to enhance the generation of distractors.

[0103] MSG proposes a diverse content selection model, which selects diverse content from the text as the basis for generating distractors to enhance the diversity among distractors.

[0104] QDG proposes a joint problem-distraction generation model to generate both the problem and distractors. Specifically, it calculates the average value between each generated distractor and the three distractors in the dataset. This invention also calculates the overall average value for comparison. The statistical results are shown in Table 3.

[0105] Table 3 Comparison of experimental results

[0106]

[0107] As can be seen, except for the third interference term where the ROUGE-L score is slightly lower than that of MSG, DGRL... SFT The three generated interference terms all outperformed the previously highest-scoring model in both BLEU1-4 and ROUGE-L scores on the test set. Since BLEU and ROUGE scores have limited reference value, we will continue to evaluate the generation quality using other methods.

[0108] The most common evaluation metrics for automation are BLEU and ROUGE. These two metrics were initially used for machine translation and summarization tasks, and subsequently, researchers have tended to use their levels to demonstrate the state of their proposed methods in other generation tasks. The vast majority of researchers use BLEU and ROUGE in their experiments to measure the level of interference generated.

[0109] BLEU and ROUGE essentially reflect the word-level similarity between the generated result and the target text; that is, the higher the BLEU and ROUGE values, the more similar the generated result is to the target text. The distractor generation task belongs to the open-domain generation task. The human-written samples in the dataset are only a relatively high-quality reference, but not the only correct answer. Even assuming that the reference samples in the dataset are the highest-quality perfect answers, it does not mean that being 90% similar to the reference result is "90% perfect." A simple experiment further demonstrates that the quality of distractors is unrelated to their similarity to the distractor samples in the dataset.

[0110] In the RACE dataset, each sample comes with three human-written distractors. Consider a distractor generation scenario: assuming each sample in the RACE dataset has only two distractors, a distractor model named "Genius" generates the exact same result as the third distractor in the RACE dataset. Its average BLEU and ROUGE scores on the EQG-RACE test set are shown in Table 4.

[0111] Table 4. Genius's BLEU and ROUGE scores on the EQG-RACE test set.

[0112] Genius 31.55 15.28 10.18 8.0 22.89

[0113] This demonstrates that calculating the BLEU and ROUGE scores between the generated results and the reference samples given in the dataset cannot accurately measure the quality of the distractors in the distractor generation task.

[0114] Next, we will use multiple evaluation methods to compare DGRL. SFT and DGRL RL The quality of the generated interfering terms will be compared with the quality of human-written interfering terms in the EQG-RACE test set.

[0115] Evaluating the quality of a single disturbance item using a reward model:

[0116] The reward model is used in reinforcement learning to represent the quality of individual distractors by generating scores within a real-valued range. Therefore, the reward model is used to calculate the scores of each distractor in the two DGRL generation results and the EQG-RACE test set to represent the overall level of the distractors. It should be noted that the samples evaluated by the reward model are taken from the EQG-RACE test set and the distractor generation model's output on the test set. The evaluation samples do not appear in the training or validation sets of either the reward model or the distractor generation model. Therefore, for the reward model, the distractors from all four sources are unfamiliar and have similar data distributions.

[0117] To comprehensively compare the quality of interference items between the model and the dataset, the mean, maximum, and minimum scores of each interference item group were calculated. Since the reward model's output is in the real number range, the real scores were first fed into a sigmoid function to reduce the scores to the range of 0-1 before calculating the mean. Then, the mean was multiplied by 100 to obtain a score within the range of 0-100. Specifically, n interference items d = (d1, d2, ..., dn) were calculated. n The average score r of the reward model is shown in the following formula, and the results are shown in Table 5.

[0118]

[0119] Table 5. Reward Model Scoring Table

[0120]

[0121] Table 5 shows that, in terms of individual distractor quality, both the supervised fine-tuning model and the model further refined through reinforcement learning outperformed the distractor samples in the EQG-RACE dataset on the EQG-RACE test set. This indicates that the quality of distractor samples in the EQG-RACE dataset, or rather the RACE dataset, does not meet the ideal requirements for human question generation. Further research on high-quality distractor generation is necessary, building upon supervised fine-tuning using datasets. Meanwhile, DGRL... SFT The score is higher than EQG-RACE, which proves the effectiveness of the proposed disturbance generation method based on textual instructions and instruction fine-tuning of large language models.

[0122] Evaluate the quality of individual interference items using a rating system:

[0123] Besides evaluating the quality of individual distractors through reward model scoring, a grading system can also be used. Based on the quality requirements of individual distractors in real-world question scenarios, distractor quality can be divided into three levels:

[0124] (1) Strongly deceptive distractors. One of the goals of generating these options is to effectively confuse the test taker when they appear alongside the correct answer. Typically, strongly deceptive distractors can be a possible answer to the question without the context of the passage, or even if the content they describe appears in the passage but is not what the question is specifically asking about.

[0125] (2) Simple distractors. Simple distractors are usually easy for test takers to eliminate compared to the correct answer. The content of these distractors is usually illogical, even absurd, or too far removed from the content described in the article.

[0126] (3) Illegal distractors. Distractors are semantically too similar to the correct answer to be distinguishable, making it impossible for test takers to select the unique correct answer. Considering the high cost and long evaluation time of manual testing, ChatGPT based on GPT4 was used to replace humans to evaluate the EQG-RACE test set and DGRL. SFT , and A total of 11,256 interference items from the four sets of samples were evaluated for their levels.

[0127] The definitions of the three interference quality levels mentioned above, combined with interference items generated by the model or in the dataset, are presented as text instructions and passed to ChatGPT. ChatGPT is then tasked with automatically rating the given interference items based on the quality definitions, article content, and question answers. To ensure the reliability of the rating results, the text instructions sent to ChatGPT require it to first provide the reasoning for the rating in the form of a thought process chain, followed by the rating. This is used in conjunction with the EQG-RACE test set and DGRL. SFT and DGRL RL The number and percentage of each rating category were used to further compare the quality of individual interference items. The results are shown in Table 6.

[0128] Table 6. Results of Interference Item Level Assessment

[0129]

[0130] We can see the interference samples of EQG-RACE and DGRL SFT The generated distractors are generally distributed evenly across all levels. Approximately two-thirds are highly deceptive distractors, about one-third are simple distractors, and a small number are illegal distractors.

[0131] DGRL after reinforcement learning RL Among the generated distractors, the proportion of highly deceptive distractors increased significantly, while the proportion of simple distractors decreased significantly, further demonstrating that reinforcement learning improved the overall quality of the distractors.

[0132] However, the number of illegal interference items in EQG-RACE is significantly less than that generated by DGRL, so there is still room for improvement in DGRL in terms of controlling the number of illegal interference items.

[0133] The quality of multiple interference items is evaluated by calculating diversity evaluation indicators:

[0134] Real-world scenarios require not only deceptiveness and legitimacy of distractors, but also diversity among multiple distractors. Therefore, we will measure distractor diversity by calculating common automated evaluation metrics. Specifically, we will calculate the following four diversity scores:

[0135] 1. Self-BLEU: Calculates the average BLEU score among multiple samples, specifically using BLEU-4 as the calculation metric. The lower the Self-BLEU value, the lower the similarity between samples and the stronger the diversity.

[0136] 2. DISTINCT-1: Calculates the percentage of unique words across multiple samples. A higher DISTINCT-1 value indicates a greater number of unique words and stronger diversity.

[0137] 3. DISTINCT-2: Calculates the percentage of unique bigrams across multiple samples. Similarly, the more unique bigrams there are, the greater the diversity.

[0138] 4. Mean Cosine Distance: Calculates the cosine distance between the vector representations of multiple samples. The specific formula is as follows:

[0139]

[0140] Where n is the number of distractors, d i Let be the vector representation of the i-th interference term. The larger the cosine distance, the stronger the semantic diversity among the interference terms.

[0141] The results of the diversity evaluation index calculation are shown in Table 7.

[0142] Table 7 Calculation Results of Diversity Evaluation Indicators

[0143]

[0144] Overall, the diversity of interference terms generated by DGRL is significantly lower than that of the interference term samples in EQG-RACE. Among the three DGRL models, The model achieved the best results in every diversity score, indicating that the hyperparameter settings of the objective function that emphasize diversity played a role, and the diversity of the model was further improved with reinforcement learning. While achieving a significant improvement in the quality of individual interference items, its overall diversity score is comparable to DGRL. SFT It remained at the same level.

[0145] Evaluating the overall quality of distractors using a single-choice question answering model:

[0146] To further evaluate the applicability of DGRL in real-world question generation and answering scenarios, BERT was selected as the single-choice question answering model. First, it was fine-tuned on the EQG-RACE training set using a single-choice question task. Then, it was tested on the test set and on the results generated by DGRL's distractors, and its accuracy was statistically analyzed. The results are shown in Table 8.

[0147] Table 8 shows the accuracy of BERT in responding to distractor generation results.

[0148]

[0149] The above results indicate that using The reading comprehension multiple-choice questions constructed from the generated distractors were the most difficult, while the samples in EQG-RACE were the least difficult, which proves the effectiveness of reinforcement learning.

[0150] The overall quality of interference items was evaluated through manual assessment.

[0151] Finally, experienced English test creators were invited to manually evaluate 10 samples from the EQG-RACE test set. The evaluators will assess the deceptiveness and legitimacy of the manually crafted distractors in the 10 selected samples using DGRL (Discretionary Grading). SFT and The generated distractors are then ranked. Unlike the reward model data construction, this ranking will have 9 candidate distractors for each sample. After ranking, the evaluators will also need to assign a score ranging from 0 to 5 to the diversity of the two DGRL distractors.

[0152] Plot the frequency of each interference item at each ranking position as a bar chart, see [link / reference]. Figure 6 As shown in the figure. Blank bars represent human-written noise in EQG-RACE, shaded bars represent noise generated by DGRLSFT, and black bars represent... Generated interference items. From Figure 6 It can be seen from this that The generated interference items ranked highest overall.

[0153] Table 9 shows the total number of times the EQG-RACE test set and the two DGRLs achieved first and top three rankings, as well as the diversity score results.

[0154] Table 9 shows the results of manual evaluation of interference items.

[0155]

[0156] The number of times the first and third generated interference items were the highest, reaching 8 and 15 times respectively. (DGRL) SFT Although it never achieved first place, it ranked second with a total of 9 appearances in the top three. EQG-RACE had the lowest overall ranking but the highest diversity score.

[0157] Based on the above, the following conclusions can be drawn:

[0158] (1) Focusing on the quality of individual interference items While maintaining the basic diversity, it is possible to generate more deceptive and higher-quality individual distractors.

[0159] (2) Emphasizing the diversity of interference items With the quality of individual interference items remaining essentially unchanged, the diversity is compared to DGRL. SFT Further improvements have been achieved.

[0160] (3) In EQG-RACE, the human-generated noise samples, in terms of the quality of individual noise items, especially their deceptiveness, are not as good as those generated by the DGRL model. However, in terms of diversity, human-generated noise items still have a very obvious advantage, and in terms of legitimacy, they are more stable than those generated by the DGRL model.

[0161] (4) The proposed distractor generation model based on a large language model and the method based on reinforcement learning are effective and advanced. They can be used in real-world question-setting scenarios as an automatic distractor generation assistant to provide distractor samples and references for question setters.

[0162] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0163] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating multiple perturbation terms based on a large language model and reinforcement learning, characterized in that, Includes the following steps: Obtain EQG-RACE data and divide it into training and validation sets; Establish a distractor generation model based on text instructions; The training set is input into the disturbance term generation model and supervised fine-tuning is performed to obtain the supervised fine-tuned disturbance term generation model. The validation set is input into the supervised fine-tuned interference terms to generate the model, and the generated results are sorted. Construct a reward model, obtain the training dataset for the reward model based on the ranking results, and calculate the pairwise ranking loss; The supervised fine-tuned disturbance generation model is further trained based on the reward model and reinforcement learning to obtain the highest quality multiple disturbances. The specific content of establishing a distractor generation model based on text instructions is as follows: First, the reading comprehension passage, questions, answers, and the number of distractors are written into the text instruction template to obtain text instructions. Then, the text instructions are input into the distractor generation model after the instructions have been fine-tuned. Based on the response format and the number of distractors required in the instructions, a response text containing a specified number of distractors is generated. Finally, the generated distractors are extracted from the response text. The specific content of the training dataset for the reward model based on the ranking results is as follows: the evaluation criteria for distractors, articles, questions, answers and distractors are constructed as text instructions, ChatGPT is used to rank the samples, and the ranking results are used to construct the training dataset. The specific content of the disorder generation model, which is further trained and fine-tuned based on the reward model and reinforcement learning, is as follows: First, several corresponding interference items are generated according to the text instructions. Then, the interference items are input into the reward model to obtain the reward score for the quality of a single interference item. At the same time, the diversity score among multiple interference items is calculated. The two scores are input into the objective function to obtain the final score. Finally, the score of the objective function is optimized by gradient descent.

2. The method for generating multiple perturbation terms based on a large language model and reinforcement learning according to claim 1, characterized in that, The specific content of the supervised fine-tuned disturbance generation model is as follows: Using text commands as existing text, predict each word in the response text. When generating a model with supervised fine-tuning of perturbations, combine the text commands and response text into a target text. Employ Teacher-Forcing technique to predict the word at the next time step based on the target's historical context, thus transforming the text generation task into a word classification task. The classification target at each time step is the next word in the target text, and the classification loss at each time step is calculated.

3. The method for generating multiple perturbation terms based on a large language model and reinforcement learning according to claim 1, characterized in that, Sorting is based on the evaluation criteria for individual interference items.

4. The method for generating multiple perturbation terms based on a large language model and reinforcement learning according to claim 3, characterized in that, The evaluation criteria for a single interference item include: legitimacy and deceptiveness.

5. The method for generating multiple perturbations based on a large language model and reinforcement learning according to claim 1, characterized in that, The supervised fine-tuned interference generation model and reward model both include: word embedding layer, several decoders, and linear layer.

6. The method for generating multiple perturbations based on a large language model and reinforcement learning according to claim 5, characterized in that, The specific content of training the reward model based on pairwise ranking loss is as follows: The reward model combines the textual instructions for individual distractors and the distractors themselves into the input text, specifically: Given a reading comprehension passage P, a reading comprehension question Q constructed based on the passage, the correct answer A, and distractors D generated by a distractor generation model, the input text T of the reward model is denoted as T=[P; Q; A; D]. Given an input text of length n, T=(t1, t2, ..., t... n The input reward model first passes through a word embedding layer to obtain the feature matrix M, as shown in the formula: Where d is the feature dimension of the hidden layer of the model; Assuming the reward model has L decoders, the output of the last decoder is represented as: in, This is the output of the i-th decoder; The output of the final decoder layer contains the hidden layer vector corresponding to each input word, and is therefore represented as Extract the hidden layer vector corresponding to the last word. , and learnable parameters Multiply to get the bonus points, the formula is: Therefore, the formula for calculating pairwise ranking loss is: in, The bonus points are awarded to the highest-ranking distractors. The bonus points awarded to low-ranking distractors.

7. The method for generating multiple perturbation terms based on a large language model and reinforcement learning according to claim 6, characterized in that, The specific content of the disorder generation model, which is further trained and fine-tuned based on the reward model and reinforcement learning, is as follows: First, several corresponding interference items are generated according to the text instructions. Then, the interference items are input into the reward model to obtain the reward score for the quality of a single interference item. At the same time, the diversity score among multiple interference items is calculated. The two scores are input into the objective function to obtain the final score. Finally, the score of the objective function is optimized by gradient descent.

Images