A diversity question generation method based on concept structuring reward and quality perception gating

Abstract

This invention relates to a method for generating diverse questions based on conceptual structured rewards and quality-aware gating, belonging to the field of diverse question generation technology. The method includes: constructing and training a supervised fine-tuning question generation model and a reward function model; performing two-stage candidate question generation to obtain candidate samples and extract four-dimensional quality signals, constructing a conceptual structured reward, then introducing quality-aware gating to construct a quality-aware gating reward for the candidate samples, and combining this with the basic reward of the candidate samples to obtain the final reward value; performing reinforcement learning on the trained supervised fine-tuning question generation model and reward function model based on the final reward value to obtain a diverse question generation model; inputting the dataset of questions to be generated into the diverse question generation model to obtain several questions, which are then sorted to obtain an ordered candidate question list. This method aims to solve the technical problems of existing technologies' difficulty in uniformly representing the multi-attribute combination features of questions and the lack of quality constraint mechanisms.

Description

Technical Field

[0001] This invention relates to a method for generating diversity problems based on conceptual structured rewards and quality-aware gating, belonging to the field of diversity problem generation technology. Background Technology

[0002] The technology of automatically generating diverse questions aims to automatically generate multiple questions with different expressions, reasonable semantics, fluency, and answerability based on the context and answer of a given text. Traditional question generation methods often focus on generating the most likely question for each (context, answer) pair, ignoring the diversity and flexibility of question expression, resulting in monotonous and rigid generation results. However, in real-world applications, the same answer often corresponds to multiple reasonable ways of asking questions. For example, in intelligent education, teachers need to ask questions from different angles to test students' depth of understanding; in dialogue systems, the system also needs to use varied questions to maintain natural interaction. Therefore, how to effectively improve the diversity of generated questions without sacrificing semantic correctness and fluency has become one of the current research hotspots in the field of natural language processing, especially in scenarios such as intelligent education, reading comprehension, question-answering systems, dialogue systems, and intelligent tutoring, where it has significant application value.

[0003] With the development of sequence-to-sequence pre-trained models, question generation techniques have been able to learn the mapping relationship between context, answer, and question to a certain extent. Although recent research has used reinforcement learning to design various reward models to capture the attributes of ideal questions, existing methods are limited by the scalar reward paradigm, making it difficult to fully characterize the multi-attribute concept combination features of high-quality questions. This easily leads to templated candidate questions, resulting in an imbalance between quality and diversity. Secondly, to improve the difference between candidate questions, existing techniques usually adopt methods such as increasing sampling temperature, expanding the search space, introducing repetition penalties, or post-processing rewriting. Although these methods can superficially improve the difference between outputs, they cause phenomena such as disconnect between questions and answers, mismatch between questions and context, semantic incompleteness, and even linguistic incoherence, thus producing pseudo-diversity. Therefore, how to maintain the quality of candidate questions while enhancing question diversity is a key challenge in existing techniques. In addition, existing reinforcement learning training often only focuses on the current batch of samples, and does not make full use of historical high-quality candidate samples. Even if there is an experience replay mechanism, it is mostly used as a sample cache rather than a concept reference pool, making it difficult to form a stable dynamic evaluation benchmark. This can lead to instability in the model's criteria for judging "high quality and diversity" at different stages, affecting the robustness and convergence of training.

[0004] To address the aforementioned issues, a new problem generation technology is urgently needed to structurally represent the multidimensional quality attributes in problem generation and achieve effective diversity enhancement under quality constraints, thereby improving the comprehensive balance between quality and diversity in the generated results. Summary of the Invention

[0005] This invention aims to solve the following problems existing in the problem generation process of the prior art:

[0006] First, existing technologies rely heavily on a single scalar reward to optimize candidate questions when using reinforcement learning to train models, making it difficult to uniformly represent the combination of multiple attributes such as answerability, fluency, relevance, and diversity of questions.

[0007] Secondly, existing technologies often lack quality constraint mechanisms, which can easily lead to a decline in problem quality while increasing the diversity of candidates, resulting in an imbalance between quality and diversity.

[0008] To achieve the above objectives, the technical solution of this invention is: a method for generating diverse questions based on conceptually structured rewards and quality-aware gating. Given a context and answer conditions, this method first generates reasoning arguments, then generates questions, and extends traditional scalar rewards to conceptually structured rewards by jointly modeling signals such as answerability, fluency, relevance, and diversity. Simultaneously, it conditionally amplifies diversity benefits using quality-aware gating, thereby improving the semantic quality, naturalness of expression, and candidate diversity of the generated questions, and enhancing the interpretability of the model training process. The method includes the following steps:

[0009] Step 1: Obtain the original dataset for the question generation task, preprocess the original dataset to obtain the sample dataset; construct a supervised fine-tuning question generation model and a reward function model, and train the supervised fine-tuning question generation model and the reward function model using the sample dataset;

[0010] Step 2: Based on the sample dataset, perform two-stage candidate question generation to obtain candidate samples. Obtain the basic reward of the candidate samples through the trained reward function model, extract the four-dimensional quality signal of the candidate samples, construct a concept-structured reward, introduce quality-aware gating, obtain the quality-aware gating reward of the candidate samples based on the four-dimensional quality signal and the concept-structured reward, weight the quality-aware gating reward with the basic reward to obtain the final reward value, and perform reinforcement learning on the trained supervised fine-tuning question generation model and reward function model based on the final reward value to obtain a diverse question generation model;

[0011] Step 3: Input the dataset of questions to be generated into the diversity question generation model to obtain several questions. Calculate the confidence of all the questions and sort them from high to low confidence to obtain an ordered list of candidate questions.

[0012] Optionally, obtaining the original dataset for the question generation task and preprocessing the original dataset to obtain the sample dataset specifically involves:

[0013] Define the original dataset used for the problem generation task as follows: The expression is:

[0014]

[0015] in, Represents the total number of samples. This represents the context text of the i-th sample. This represents the highlighted context text of the i-th sample. This represents the answer text for the i-th sample. This represents the target question text for the i-th sample. The text representing the reasoning basis for the i-th sample;

[0016] The context text, highlighted context text, answer text, target question text, and reasoning basis text are segmented and encoded to obtain the corresponding token sequence representations, as shown in the expression:

[0017]

[0018] in, For word segmentation encoder, This represents the context encoding sequence of the i-th sample. This represents the highlighted context encoding sequence of the i-th sample. This represents the answer encoding sequence for the i-th sample. This represents the target question encoding sequence of the i-th sample. This represents the encoding sequence of the reasoning basis for the i-th sample.

[0019] Optionally, the step of constructing a supervised fine-tuning problem generation model and a reward function model, and training the supervised fine-tuning problem generation model and the reward function model using the sample dataset, specifically involves:

[0020] The supervised fine-tuning question generation model employs a pre-trained sequence generation model based on an encoder-decoder architecture, and performs supervised fine-tuning based on the pre-trained parameters. By introducing reasoning text, the question generation process is represented as a structured sequence learning task that first generates reasoning and then generates the question. This allows the supervised fine-tuning question generation model to learn the intermediate semantic information between the context, the answer, and the question in the initial stage. For the i-th training sample, the structured target output sequence in the supervised fine-tuning stage... Defined as:

[0021]

[0022] in, <r>Indicates that the reasoning is based on the starting marker. <q>Indicates the starting marker of the question. Indicates the end of the sequence;

[0023] The supervised fine-tuning question generation model predicts the token at the current position at each time step based on the highlighted context, the answer, and tokens generated before the current time step. The loss function of the supervised fine-tuning stage is then applied. Defined as cross-entropy loss, the expression is:

[0024]

[0025] in, This represents the total number of samples used for supervised fine-tuning training. This represents the length of the structured target output sequence corresponding to the i-th training sample. The parameters represent the model generated from the problem in the supervised fine-tuning phase. The conditional probability distribution given by the defined supervised fine-tuning problem generation model. This represents the target character at position t in the structured target output sequence of the i-th training sample. This represents the target character subsequence that has been given up to the t-th position in the i-th training sample;

[0026] The reward function model includes a question quality reward model and a reasoning consistency scoring model. The question quality reward model uses an encoder-decoder model as its basic structure and calculates the conditional log probability of each character based on the context encoding results and the concatenated target sequence. This process progressively scores the question-answer joint sequence to obtain the final temperature-scaled question quality reward. The expression is:

[0027]

[0028] in, For the candidate question sequence, Indicates the length of the candidate problem sequence. For the answer sequence, Indicates the length of the answer sequence. The parameters represent the problem quality reward model. This represents the temperature parameter of the problem quality reward model, where c represents the context text. This represents the character at position t in the question-answer sequence z. This represents the prefix character subsequence in the combined sequence z preceding the t-th position;

[0029] The reasoning consistency scoring model employs a natural language reasoning model, feeding the context text and the reasoning text as a pair of inputs into a text entailment classifier, and using the log probability of the entailment category as the reasoning consistency reward. Thus, the reward function of the reward function model is obtained as follows: .

[0030] Optionally, the step of generating candidate questions in two stages based on the sample dataset to obtain candidate samples specifically involves:

[0031] Given the highlighted context text and the answer text, a sequence of reasoning basis is first generated, then a sequence of questions is generated based on the sequence of reasoning basis, and the sequence of reasoning basis and the sequence of questions are connected by a special delimiter symbol to obtain candidate samples.

[0032] Optionally, the step of extracting the four-dimensional quality signal of the candidate sample and constructing a conceptual structured reward specifically involves:

[0033]

[0034] In the formula, For four-dimensional quality signals, This represents the answerability signal of candidate sample x. Indicates smoothness signal, Indicates correlation signal, Indicates diversity signals;

[0035] The expression for the responsiveness signal is as follows:

[0036]

[0037] In the formula, This represents an overlap function based on a character set. Rewards for quality issues. , , Let represent the weighting coefficients of the responsiveness signal, and satisfy . =1;

[0038] The expression for the smoothness signal is as follows:

[0039]

[0040] In the formula, This indicates the penalty for repeating candidate questions. This represents the penalty for deviation of the candidate problem from the preset length interval. Indicates penalty for abnormal characters. , , As weight;

[0041] After obtaining the four-dimensional quality signal of the candidate samples, a concept-structured reward is constructed based on axiomatic fuzzy sets, specifically as follows:

[0042] First, the historical distribution of the four-dimensional quality signal is cached separately, and then the low threshold and high threshold of each dimension are obtained by the quantile method, forming the segmentation boundary of the three categories of low, medium and high concepts.

[0043] Secondly, three levels of concepts—low, medium, and high—are constructed for the four dimensions of the four-dimensional quality signal, and positive and negative concepts are assigned to each level to form a set of basic concepts.

[0044] Then, on the joint universe of discourse formed by the current batch of samples and the experience replay samples, the axiomatic fuzzy set structure generating function is invoked to generate a set of composite concepts. ;

[0045] Finally, after the composite concept is constructed, the concept structure reward is obtained by aggregating the candidate samples according to their membership in the preferred concept set. The expression is:

[0046]

[0047] in, Indicates sample For the first Membership degree of each preferred concept, This represents the set of preferred concepts after filtering.

[0048] Optionally, the introduction of quality-aware gating, specifically obtaining the quality-aware gating reward for candidate samples based on the four-dimensional quality signal and the concept-structured reward, involves:

[0049] The four-dimensional quality signal is standardized using the following expression:

[0050]

[0051] in, Denotes the standardized answerability term of candidate sample x. Indicates the standardized fluency item. Represents standardized relevance terms. For standardized functions;

[0052] This yields the overall quality score of the candidate samples. The expression is:

[0053]

[0054] in, , , Let be the weighting coefficient for the overall quality score, and satisfy . + + =1;

[0055] The overall quality score By inputting a preset gating function (Sigmoid), the gating value of the candidate samples is obtained, which controls the diversity signal of the candidate samples. Perform standardization processing to obtain standardized diversity items. The standardized diversity term is then gated using the gating values ​​of candidate samples to obtain the quality-perceived gating reward for the candidate samples. The expression is:

[0056]

[0057] in, This represents the coefficient of the quality enhancement term. This represents the coefficient of the diversity enhancement term. This represents the coefficient for the low-quality penalty term.

[0058] Optionally, the step of performing reinforcement learning on the trained supervised fine-tuning problem generation model and reward function model based on the final reward value to obtain a diverse problem generation model specifically involves:

[0059] An experience replay module is constructed based on candidate samples. During reinforcement learning training, online policy sampling and offline policy experience replay sampling are used alternately to update model parameters. A diversity intensity scheduling function is designed to achieve progressive optimization that prioritizes quality in the early stages of reinforcement learning training and gradually increases diversity in the later stages. Specifically, the experience replay module is as follows:

[0060] The generated sequence, final reward, four-dimensional quality signal, concept-structured reward, and basic reward corresponding to candidate sample x are jointly written into the experience replay module, along with the corresponding replay entries. The expression is:

[0061]

[0062] in, This represents the final reward value;

[0063] For new candidate samples and existing candidate samples If the following conditions are met:

[0064]

[0065] Then determine and The differences between them are insufficient, belonging to approximately duplicate samples; among them, This represents the edit distance function. This represents the similarity tolerance coefficient;

[0066] When near-duplicate samples appear, only the candidate sample with the higher final reward is retained;

[0067] When the number of samples in the experience replay module reaches the preset capacity limit, a retention and sampling mechanism based on reward priority is adopted, defining the final reward of the i-th candidate sample in the experience replay module as follows: Then the probability of the i-th candidate sample being far from the policy sampling probability The expression is:

[0068]

[0069] in, Represents an exponential function. This represents the minimum final reward value among all candidate samples in the experience replay module. This represents the final reward value corresponding to the k-th candidate sample in the experience replay module.

[0070] The beneficial effects of this invention are as follows: This invention maps the answerability, fluency, relevance, and diversity of candidate questions into multi-dimensional quality signals, and further utilizes historical signal distribution to construct low, medium, and high-level concept boundaries. On the joint domain of discourse formed by the current batch of samples and historical candidate samples, composite concepts are generated through axiomatic fuzzy set structures and EI algebras, forming interpretable concept structured rewards. On this basis, a comprehensive quality score is constructed based on answerability, relevance, and fluency. The diversity enhancement term is dynamically adjusted through a gating function, so that high-quality samples receive more effective diversity incentives and low-quality samples are suppressed, providing a new theoretical path for quality-diversity co-optimization in the generation of diverse questions. This invention can be widely applied to scenarios such as automatic question generation and knowledge point assessment in intelligent education, multi-turn questioning and interactive guidance in dialogue systems, reading comprehension data augmentation, and personalized practice in intelligent tutoring systems. It can effectively improve the content generation capability and user experience of the system without increasing the cost of manual annotation. Attached Figure Description

[0071] Figure 1 This is a detailed flowchart of the present invention. Detailed Implementation

[0072] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0073] Example 1: As Figure 1 As shown, a method for generating diverse questions based on conceptual structured rewards and quality-aware gating includes the following steps:

[0074] Step 1: Obtain the original dataset for the question generation task, preprocess the original dataset to obtain the sample dataset; construct a supervised fine-tuning question generation model and a reward function model, and train the supervised fine-tuning question generation model and the reward function model using the sample dataset;

[0075] Step 1.1: Define the original dataset for the problem generation task as follows The expression is:

[0076]

[0077] in, Represents the total number of samples. This represents the context text of the i-th sample. This represents the highlighted context text of the i-th sample. This represents the answer text for the i-th sample. This represents the target question text for the i-th sample. The text representing the reasoning basis for the i-th sample;

[0078] It is understandable that this embodiment can provide complete information support for the subsequent supervised fine-tuning stage and reinforcement learning stage through the five-tuple structure of the original dataset. Among them, highlighting the context helps the model focus on the answer-related area, the answer text helps constrain the direction of question generation, and the reasoning basis helps to explicitly introduce the explanation path in the generation process, thereby improving the controllability of the question generation process.

[0079] After obtaining the original dataset, preprocessing is performed on it to obtain the sample dataset. This preprocessing includes text normalization, length truncation, token encoding, and attention mask construction. Specifically, the context text, highlighted context text, answer text, target question text, and reasoning basis text are each segmented and encoded to obtain the corresponding token sequence representation, expressed as:

[0080]

[0081] in, For word segmentation encoder, This represents the context encoding sequence of the i-th sample. This represents the highlighted context encoding sequence of the i-th sample. This represents the answer encoding sequence for the i-th sample. This represents the target question encoding sequence of the i-th sample. This represents the encoding sequence of the reasoning basis for the i-th sample.

[0082] Furthermore, to ensure that the encoder can distinguish between valid text positions and padding positions during the calculation process, this embodiment also constructs an attention mask based on the context encoding sequence, with the expression as follows:

[0083]

[0084] in, This represents the attention mask corresponding to the context-encoded sequence. The symbol represents the padding marker, and 1() represents the indicator function, which takes the value 1 when the corresponding position is not a padding marker, and 0 otherwise.

[0085] It is understandable that the attention mask constructed in this embodiment can ensure that the model performs semantic modeling only on valid text tokens during the encoding process, without being affected by the padding position, thus laying a unified data foundation for subsequent supervised fine-tuning and optimization of diversity issues.

[0086] Step 1.2: Construct and train a supervised fine-tuning question generation model to obtain the initial policy model required for the subsequent reinforcement learning stage. The supervised fine-tuning question generation model is used to learn the structured output sequence mapping relationship from the highlighted context text and answer text to the <reasoning basis, question> tuple, so that the model has basic question generation ability and explicit intermediate reasoning expression ability before entering reinforcement learning optimization.

[0087] Optionally, the supervised fine-tuning question generation model employs a pre-trained sequence generation model based on an encoder-decoder architecture, such as the T5-large model, and performs supervised fine-tuning based on the pre-trained parameters. Unlike the traditional approach of directly generating question text, this embodiment introduces reasoning text, representing the question generation process as a structured sequence learning task that first generates reasoning and then generates the question. This allows the model to learn the intermediate semantic information between the context, the answer, and the question in the initial stage. For the i-th training sample, the structured target output sequence in the supervised fine-tuning stage... Defined as:

[0088]

[0089] in, <r>Indicates that the reasoning is based on the starting marker. <q>Indicates the starting marker of the question. This indicates the end of the sequence.

[0090] It is understood that this embodiment does not treat the question generation task as a simple single-segment text generation task. Instead, it introduces special structural markers to organize the reasoning basis and the question into a unified hierarchical output sequence, so that the supervised fine-tuning question generation model can learn the generation pattern of reasoning to the question. This design is consistent with the two-stage candidate generation mechanism adopted in the subsequent reinforcement learning stage, which is conducive to ensuring the consistency of the goals and the continuity of the structure before and after the model training.

[0091] Furthermore, during supervised fine-tuning training, the highlighted context encoding sequence of the i-th sample is... With the answer encoding sequence As conditional input, the structured target output sequence As the supervision objective, the parameters of the supervised fine-tuning problem generation model are trained, and the supervised fine-tuning problem generation model is denoted as... ,in, Let represent the model parameters. Then, the conditional probability distribution expression for the supervised fine-tuning problem generation model learning is:

[0092]

[0093] Understandably, the above formula indicates that the learning objective of the supervised fine-tuning question generation model is to construct a joint generation distribution of reasoning basis and target question, given the highlighted context and answer. Simultaneously, due to the output sequence... The reasoning has already been organized in the order of inference first, followed by the question. Therefore, the above formula also implicitly indicates that the model learns during training to first predict the reasoning and then the conditional generation relationship of the question. Correspondingly, for the entire training set, the standard maximum likelihood estimation approach is adopted in the supervised fine-tuning stage. By modeling the conditional probabilities of the structured output sequences of all training samples, the initial question generation strategy is learned. The training objective of the supervised fine-tuning question generation model is to maximize the conditional log-likelihood of the structured output sequences of all samples, expressed as:

[0094]

[0095] in, This represents the total number of samples used for supervised fine-tuning training;

[0096] Furthermore, the supervised fine-tuning question generation model employs an autoregressive decoding approach to progressively generate a structured output sequence. Specifically, at each time step, it predicts the token for the current position based on the highlighted context, the answer, and tokens already generated before the current time step. The loss function of the supervised fine-tuning stage is then applied... Defined as cross-entropy loss, the expression is:

[0097]

[0098] in, This represents the length of the structured target output sequence corresponding to the i-th training sample. The parameters represent the model generated from the problem in the supervised fine-tuning phase. The conditional probability distribution given by the defined supervised fine-tuning problem generation model. This represents the target character at position t in the structured target output sequence of the i-th training sample. It represents the target character subsequence given before the t-th position in the i-th training sample, that is, the prefix sequence formed by the 1st position to the t-1th position.

[0099] Furthermore, during training, the model parameters are optimized by minimizing the negative log-likelihood of all positions in the structured output sequence. Continuous updates are made to improve the model's ability to generate correct reasoning and target questions given the highlighted context and answer conditions. After training, the optimized supervised fine-tuning problem generation model parameters are denoted as... And the corresponding model is recorded as the initial strategy model. This serves as the initialization result of the policy model during the reinforcement learning training phase.

[0100] Furthermore, after constructing the supervised fine-tuning question generation model, a reward function model is constructed and trained to evaluate the generation quality of candidate samples in the subsequent reinforcement learning stage. The reward function model does not directly participate in candidate question generation but serves as an evaluator in reinforcement learning training, scoring different quality attributes of candidate samples to provide optimization signals for policy model parameter updates. In this embodiment, the reward function model comprises two parts: a question quality reward model, used to evaluate the matching degree between candidate questions and answers; and a reasoning consistency scoring model, used to evaluate the semantic consistency between generated reasoning and context. The question quality reward model corresponds to the question quality reward in subsequent reinforcement learning training, and the reasoning consistency scoring model corresponds to the reasoning consistency reward. By using these two reward components together, the answerability and reasoning rationality of candidate samples can be constrained simultaneously, thereby preventing the model from generating seemingly reasonable questions that are disconnected from the context or answer, merely based on superficial linguistic form.

[0101] Specifically, for the question quality reward model, an encoder-decoder model of the same type as the supervised fine-tuning question generation model is used as the basic structure, and the quality of candidate questions is evaluated through the joint conditional probability of the question text and the answer text. The candidate question sequence is defined as follows: The answer sequence is The problem quality reward model is denoted as , The parameters representing the problem quality reward model are then the problem quality reward. The expression is:

[0102]

[0103] Where c represents the context text. The above formula shows that the question quality reward is essentially a conditional scoring function, whose inputs are candidate questions, answers, and context, and whose output is a reward value reflecting the degree of semantic matching between the candidate questions and answers.

[0104] Furthermore, in this embodiment, to specifically calculate the question quality reward, the candidate question sequence and the answer sequence are first concatenated to construct a joint question-answer input sequence. The concatenated input sequence is defined as z, and its expression is:

[0105]

[0106] in, <sep>This represents the separator between the question and the answer. For each candidate question, it is concatenated with its corresponding answer using a separator character, forming a unified input to the reward model.

[0107] Furthermore, in the decoding stage, the question quality reward model calculates the conditional log probability of each character based on the context encoding results and the concatenated target sequence. Simultaneously, the question quality reward model also employs an autoregressive conditional modeling approach to progressively score the question-answer joint sequence. Correspondingly, the average log probability of the joint sequence can be further decomposed into the question component score and the answer component score, and constructed as the final temperature-scaled question quality reward. The expression is:

[0108]

[0109] in, Indicates the length of the candidate problem sequence. Indicates the length of the answer sequence. The temperature parameter represents the problem quality reward model. This temperature parameter is used to adjust the range of reward values, facilitating training stability during subsequent reinforcement learning phases. This represents the character at position t in the question-answer sequence z. This represents the prefix character subsequence in the combined sequence z that precedes the t-th position.

[0110] It is understandable that the temperature-scaled question quality reward not only depends on the reasonableness of the question's generation, but also considers the degree of matching between the generated answer and the question conditions, which can more accurately reflect whether the question and the answer are truly compatible.

[0111] Furthermore, in addition to the question quality reward model, this embodiment constructs a reasoning consistency scoring model to evaluate whether the reasoning in candidate samples maintains semantic consistency with the context text. The reasoning consistency scoring model employs a natural language reasoning model, whereby the context text and the reasoning text are fed as a pair of inputs into a text entailment classifier, and the log probability of the entailment category is used as the reasoning consistency reward.

[0112] Optionally, in this embodiment, the existing pre-trained model robotic-large-snli_mnli_fever_anli_R1_R2_R3-nl is used, and the model is not directly trained.

[0113] Based on the pre-trained model, a consistency score for reasoning is calculated. The consistency reward for reasoning after handling interrogative sentences is defined as follows: .

[0114] Furthermore, after constructing the problem quality reward model and the reasoning consistency scoring model, this embodiment uses both as the basis for the reward function model in the subsequent reinforcement learning stage. For candidate sample x, the reward function of the reward function model is: .

[0115] Furthermore, after completing the construction of the supervised fine-tuning problem generation model and the reward function model, the parameters and functional modules required for the reinforcement learning training phase are initialized based on Step 1.1 and Step 1.2, so as to perform the optimization of diverse problem generation based on concept-structured reward and quality-aware regulation. Among them, the problem quality reward model and the reasoning basis consistency scoring model will be used as the problem quality reward and reasoning basis reward in the reinforcement learning phase, respectively. Together with the initial supervised fine-tuning problem generation model obtained in the supervised fine-tuning phase, they constitute the basic components of the subsequent diverse problem generation reinforcement learning optimization framework.

[0116] Step 2: Based on the sample dataset, perform two-stage candidate question generation to obtain candidate samples. Obtain the basic reward of the candidate samples through the trained reward function model, extract the four-dimensional quality signal of the candidate samples, construct a concept-structured reward, introduce quality-aware gating, obtain the quality-aware gating reward of the candidate samples based on the four-dimensional quality signal and the concept-structured reward, weight the quality-aware gating reward with the basic reward to obtain the final reward value, and perform reinforcement learning on the trained supervised fine-tuning question generation model and reward function model based on the final reward value to obtain a diverse question generation model;

[0117] Step 2.1: After initializing the parameters and functional modules required for the reinforcement learning training phase, a two-stage candidate question generation is performed: given the highlighted context text... and answer text Under the given conditions, a sequence of reasoning evidence is first generated, then a sequence of questions is generated based on the sequence of reasoning evidence, and finally, the sequence of reasoning evidence and the sequence of questions are connected by a special delimiter to obtain candidate samples. The two-stage candidate generation process is represented as follows:

[0118]

[0119]

[0120] in, Indicates the text in the given highlighted context. Given the answer text a, generate the reasoning basis. ; This means that after generating the reasoning basis, candidate questions are generated by combining the highlighted context text, the answer text, and the already generated reasoning basis. Therefore, the complete structured output sequence of a single candidate sample can be written as:

[0121]

[0122] in, Indicates that the reasoning is based on the starting marker. Indicates the starting marker of the question. This indicates the end of the output.

[0123] Furthermore, in the specific implementation, the initial decoding prompt is first constructed based on the answer encoding, and then in the reasoning basis generation stage... The flag indicates the start of decoding; once the switching condition is met, the state is changed to [a new state]. The process is initiated by marking the candidate action sequence and entering the question generation phase. The forward log probabilities for the reasoning basis and the question are recorded separately. In this embodiment, to control the exploratory nature and diversity of candidate samples, maximum generation length, minimum generation length, and temperature parameters are set for the reasoning basis and question phases respectively during the generation process. A kernel sampling threshold top-p constraint is also introduced in the question generation phase. Furthermore, in the off-policy training mode, candidate action sequences sampled from experience replay can be directly fed into the generation function to re-evaluate their log probabilities under the current policy model parameters.

[0124] Furthermore, after the candidate samples are generated, a basic reward is calculated for each candidate sample, which consists of a problem quality reward. Rewards based on consistency with reasoning Together they constitute. Specifically, for the problems generated in candidate sample x. By constructing a problem quality reward model Scoring is performed, and a reward is given for the quality of the questions; the reasoning basis generated from candidate sample x The reasoning consistency scoring model was constructed to evaluate the reasoning basis. The semantic consistency with the context text c is scored to obtain a reasoning consistency reward. The question quality reward is calculated based on the conditional log probability of the question-answer joint sequence.

[0125] Step 2.2: After generating candidate samples and obtaining the basic reward, extract the four-dimensional quality signal of the candidate samples and construct a conceptual structured reward, specifically as follows:

[0126] It is important to understand that the extraction of the four-dimensional quality signal in this embodiment differs from the traditional method of directly compressing candidate samples into a single scalar reward. Instead, it requires starting from multiple quality dimensions of the candidate samples, performing structured characterization, and then further mapping the obtained multi-dimensional quality signals onto the axiomatic fuzzy set concept space, so that the subsequent reward calculation can more fully characterize the distribution features of high-quality and diverse candidate problems.

[0127] Specifically, for any candidate sample, its structured output is first parsed, dividing the generated result into a reasoning basis part and a question part; then, by combining the question quality reward, reasoning basis consistency reward, context text, answer text, generated reasoning basis, and generated question, answerability signal, fluency signal, relevance signal, and diversity signal are extracted, and the four-dimensional quality signal is uniformly represented as the quality signal vector of the candidate sample, with the expression:

[0128]

[0129] In the formula, For four-dimensional quality signals, This represents the answerability signal of candidate sample x. Indicates smoothness signal, Indicates correlation signal, Indicates diversity signals;

[0130] Understandably, this embodiment does not compress the quality of candidate samples into a single numerical value. Instead, it first constructs a multi-dimensional quality representation from four aspects: answerability, text naturalness, semantic relevance, and candidate diversity, thereby providing richer input for subsequent conceptual structure modeling. Specifically, answerability of candidate questions is characterized by analyzing the semantic relationships between answers and questions, and between answers and context; relevance of candidate questions is characterized by analyzing the semantic correspondence between reasoning and questions, and between context and questions; fluency of candidate questions is characterized by examining the degree of repetition, length rationality, and abnormal expressions in the question text; and diversity of candidate questions is characterized by comparing the similarity between the current candidate sample and other candidate samples in the same batch, as well as the degree of difference between the current candidate sample and historical candidate samples.

[0131] Specifically, the answerability signal is calculated by weighting the question quality reward, the lexical overlap between the answer and the question, and the lexical overlap between the answer and the context, as expressed in the following expression:

[0132]

[0133] In the formula, This represents an overlap function based on a character set. Rewards for quality issues. , , Let represent the weighting coefficients of the responsiveness signal, and satisfy . =1;

[0134] Specifically, the fluency signal is constructed based on the repetition rate, length deviation, and abnormal character occurrences in the candidate questions. First, the unique character ratio of the candidate questions is defined, and then the fluency term is constructed from the repetition penalty, length penalty, and abnormal character penalty, expressed as:

[0135]

[0136] in, This indicates the penalty for repeating candidate questions. This represents the penalty for deviation of the candidate problem from the preset length interval. Indicates penalty for abnormal characters. , , As weight. The expression indicates that if a candidate question has too many repetitions, is too short or too long, or has abnormal formatting, its fluency score will decrease.

[0137] Specifically, the correlation signal is fused based on the consistency reward of reasoning basis, the overlap between context and question, and the overlap between reasoning basis and question, reflecting the degree of consistency between candidate question and context semantics and intermediate reasoning path;

[0138] Specifically, the diversity signal is constructed based on the differences between candidate samples in the current batch and the novelty between historical candidate samples in the experience replay. The differences between candidate samples in the current batch can be obtained by back-calculating the approximate sentence-level similarity based on unigram and bigram models, and the novelty between historical candidate samples in the experience replay can be obtained by normalizing the edit distance.

[0139] Furthermore, after obtaining the four-dimensional quality signal of the candidate samples, a concept-structured reward is constructed based on axiomatic fuzzy sets, specifically as follows:

[0140] First, a historical cache of the four-dimensional quality signals is maintained. For each dimension of the signal, historical values ​​appearing during training are stored, and concept splitting points are constructed based on the historical distribution. Let the historical cache set of the k-th signal dimension be denoted as... Then its low threshold and high threshold The expression is:

[0141] ), )

[0142] in, and These represent the 30% quantile and the 70% quantile, respectively; further, an intermediate threshold is defined. .

[0143] It is important to understand that the simple concept boundaries in the axiomatic fuzzy set are not limited to a single source. Besides being based on expert knowledge or pre-defined rules, this embodiment employs a quantile-based segmentation method based on historical signal distribution to instantiate simple concepts using data-driven methods, enhancing the adaptability of the concept boundaries to changes in candidate sample distribution. Based on this, low, medium, and high-level simple concepts are constructed for the four-dimensional quality signal, and corresponding complementary concepts are configured to form the basic concept set for subsequent concept structure representation. Subsequently, the current batch of candidate samples and historical candidate samples from experience replay together constitute the universe of discourse for concept structure generation and membership calculation, and an axiomatic fuzzy set structure is generated on this universe of discourse based on the four-dimensional signal matrix. Further, based on the simple concept set, a set of composite concept expressions is generated using operations such as contradiction resolution in EI algebra, and these composite concept expressions are deduplicated and filtered. For any composite concept... Define the membership degree of a candidate sample x to it as Then, by combining the standardized results of the basic reward, the concept utility value is calculated, thereby selecting the preferred concept set that can better represent the high-quality candidate distribution.

[0144] Define the preferred concept set after filtering as The conceptual structured reward for candidate sample x can be obtained by aggregating the membership degrees of the preferred concepts and taking the logarithm, as expressed in the following expression:

[0145]

[0146] Logarithmic transformation is used to enhance the discriminative power between different samples and suppress the instability caused by extreme values. The higher the reward value, the more the current candidate sample conforms to the combination of high-value concepts such as "high answerability, high relevance, good fluency and high diversity".

[0147] Understandable The expression indicates that the obtained concept structure reward is not directly given by a single quality signal, but is determined by the common belonging degree of candidate samples to multiple efficient composite concepts.

[0148] Step 2.3: After obtaining the conceptual structured reward for candidate samples, a quality-aware gating system is introduced based on the overall quality level of the candidate samples. This system selectively amplifies or suppresses diversity to prevent the model from optimizing diversity independently of quality constraints during reinforcement learning training. This prevents low-quality pseudo-diversity issues such as semantic shifts, unnatural questions, or mismatches with answers from occurring in the generated results. In this embodiment, the quality-aware gating system further discriminates the overall quality level of candidate samples based on the obtained conceptual structured reward, and determines whether the diversity signal should be enhanced, by how much, or whether suppression and penalties are necessary. Specifically:

[0149] First, construct the comprehensive quality score: For any candidate sample x, perform standardization on the answerability signal, fluency signal, and relevance signal in the four-dimensional quality signal to construct the comprehensive quality score, expressed as:

[0150]

[0151] in, Denotes the standardized answerability term of candidate sample x. Indicates the standardized fluency item. Represents standardized relevance terms. For standardized functions;

[0152] Furthermore, the standardized quality signals are weighted and fused according to preset weights to obtain the comprehensive quality score of the candidate samples. This is used to characterize the overall semantic quality level of the current candidate sample, and its expression is:

[0153]

[0154] in, , , Let be the weighting coefficient for the overall quality score, and satisfy . + + =1;

[0155] Then, the gating value is calculated: in obtaining the overall quality score. After that, By inputting a preset gating function (Sigmoid), the gating value of the candidate samples is obtained, thus characterizing whether the current candidate samples meet the quality conditions for enhancing diversity. The overall quality score of the candidate samples... When the overall quality score is greater than the preset threshold, the gating value is close to 1. At this time, the quality of the candidate sample is relatively high, and the diversity benefits can be enhanced. When the overall quality score is less than the preset threshold, the gating value is close to 0. At this time, it is considered that although the candidate sample may have differences in surface form, its diversity benefits should not be further amplified to avoid introducing pseudo-diversity problems. Thus, a gating regulation relationship of "high quality allows enhanced diversity, low quality restricts diversity" is established.

[0156] Finally, the diversity signals are modulated and fused to obtain the final reinforcement learning reward: the diversity signal of the candidate samples. Perform standardization processing to obtain standardized diversity items. Furthermore, gating is used to scale the standardized diversity term, thereby constructing a gated diversity enhancement term. To further ensure that high-quality candidate samples receive positive quality improvement while low-quality candidate samples are suppressed, this embodiment also constructs a quality enhancement term and a low-quality penalty term based on the conceptually structured reward, and integrates them with the gated diversity enhancement term to obtain a quality-perceived gated reward for candidate samples. The expression is:

[0157]

[0158] in, This represents the coefficient of the quality enhancement term. This represents the coefficient of the diversity enhancement term. This represents the coefficient for the low-quality penalty term. Part One Used to preserve structured evaluation results in the concept space of axiom fuzzy sets; Part Two Used to directly increase the reward value of high-quality candidate samples; Part Three Used for conditional enhancement of diversity terms under gating control; Part Four This is used to penalize candidate samples with small gating values, i.e., low overall quality, thereby suppressing the introduction of low-quality pseudo-diversity.

[0159] Furthermore, quality perception gating rewards With basic rewards Weighted fusion is performed to obtain the final reward value used for subsequent reinforcement learning training. The expression is:

[0160]

[0161] in, This represents the combined weight of the concept-structured reward and the quality-perceived adjustment reward in the final reward. This final reward value will be used in subsequent experience replay updates and online / offline joint reinforcement learning optimizations, thereby driving the policy model to gradually learn to generate more diverse and interpretably supported candidate questions while ensuring question quality.

[0162] Step 2.4: Based on the obtained candidate samples, an experience replay module is constructed. During the reinforcement learning training process, online policy sampling and offline policy experience replay sampling are used alternately to update the model parameters. In addition, a diversity intensity scheduling function is designed to achieve progressive optimization that prioritizes quality in the early stage of training and gradually increases diversity in the later stage of training. Specifically:

[0163] First, an experience replay module is constructed and updated: After obtaining the final reward for the candidate samples, an experience replay module is built based on the obtained candidate samples to store historical high-value candidate samples and their related information, and to support off-policy sampling and the construction of the joint domain of discourse in the concept space during subsequent reinforcement learning training. Specifically, for any input condition, the context text... With answer text Construct replay key value To identify the set of candidate samples under the same semantic condition, the expression is:

[0164]

[0165] in, This represents the concatenation of context text and answer text.

[0166] Furthermore, for any candidate sample x, its corresponding generated sequence, final reward, four-dimensional quality signal, conceptual structured reward, and basic reward are jointly written into the experience replay module, and the corresponding replay entries are... The expression is:

[0167]

[0168] It is understood that the experience replay module constructed in this embodiment does not only store the candidate question text itself, but also a set of structured entries containing sequence, reward, signal, and concept information. This ensures that the experience replay samples can not only be used for off-policy training, but also serve as historical reference samples in the subsequent construction of concept-structured rewards. During the writing process, to avoid storing a large number of semantically similar but slightly different duplicate candidates in the replay module, the similarity between the new candidate samples and the existing candidate samples is judged by normalized edit distance.

[0169] Furthermore, for new candidate samples and existing candidate samples If the following conditions are met:

[0170]

[0171] Then determine and The differences between them are insufficient, belonging to approximately duplicate samples; among them, This represents the edit distance function. This represents the similarity tolerance coefficient. When nearly identical samples appear, only candidate samples with higher final rewards are retained, improving the representativeness and effectiveness of samples in the experience replay module. Furthermore, when the number of samples in the replay module reaches a preset capacity limit, a retention and sampling mechanism based on reward priority is adopted, defining the final reward of the i-th candidate sample in the experience replay module as... Then the probability of the i-th candidate sample being far from the policy sampling probability The expression is:

[0172]

[0173] in, Represents an exponential function. Let represent the minimum final reward value among all candidate samples in the experience replay module, j represent the index used to traverse all candidate samples in the experience replay module, and k represent the index used to traverse all candidate samples in the experience replay module. This represents the final reward value corresponding to the k-th candidate sample in the experience replay module. The expression indicates that historical candidate samples with higher final rewards will have a higher sampling probability in subsequent training, thus enabling the model to learn the distribution of historical high-value candidates more fully.

[0174] Then, online / offline joint reinforcement learning optimization is performed: after the experience replay module is completed, online policy sampling and experience replay off-policy sampling are used alternately to update the model parameters, so as to balance the current policy exploration capability and the utilization efficiency of historical high-value candidate samples.

[0175] Specifically, in the online policy sampling mode, based on the current policy model The current input sample is directly subjected to two-stage candidate generation to obtain a new candidate sample x, and its basic reward, concept structuring reward, quality-aware regulation reward and final reward are calculated. Then, the candidate sample is written into the experience playback module, and its log probability under the current policy and the final reward are used to construct the loss function.

[0176] Specifically, in the off-policy sampling mode, data is collected from the experience replay module according to... The priority probability sampling of historical candidate sample sequences defined in the expression is used as a fixed action sequence and input into the current policy model. Without resampling candidates, the parameters of the current model are recalculated. The log-forward probability, basic reward, conceptual structured reward, and final reward are defined under the current policy model. The joint log-forward probability of a candidate sample x under the current policy model is denoted as... Define the output of the normalization term estimation module as The reinforcement learning optimization loss corresponding to a single candidate sample is... The expression:

[0177]

[0178] in, Indicates the parameters of the normalization term estimation module;

[0179] Furthermore, for the current batch candidate sample set B, the corresponding batch loss is... Represented as:

[0180]

[0181] in, This indicates the number of candidate samples in the current batch.

[0182] It is understandable that during the training process, this embodiment ensures that the model can continuously explore new candidate samples in the current policy space through the online sampling mode, and ensures that the model can repeatedly use historical high-value samples for stable optimization through the off-policy sampling mode. Therefore, the alternating use of the two constitutes the joint reinforcement learning optimization mechanism of this embodiment.

[0183] Finally, diversity scheduling is performed to obtain the final diverse problem generation model: To avoid the model prematurely emphasizing the differences between candidate problems in the early stages of training, which could affect the basic quality of the problems, this embodiment designs diversity intensity scheduling to progressively control the sampling parameters and reward parameters related to diversity during training, specifically as follows:

[0184] In the early stages of training, the model is still learning the basic generation rules and semantic constraints of questions. At this time, priority is given to ensuring the answerability, relevance, and fluency of the generated questions, so diversity is kept at a low level. As training progresses and the model has developed a more stable question generation ability, the diversity intensity is gradually increased, allowing the model to learn to generate multiple candidate questions with differences while maintaining quality. Furthermore, diversity intensity scheduling not only affects the candidate question generation stage but also the reward regulation stage. In the candidate question generation stage, as training progresses, the sampling space is gradually widened to increase the exploration degree of candidate generation, while the repetition penalty and intra-batch difference constraint are gradually strengthened, making the model more inclined to generate candidate expressions that are different from each other. In the reward regulation stage, as training progresses, the role of diversity enhancement terms in the overall reward is gradually increased, and the quality enhancement terms and gating strength are adjusted accordingly, so that the model pays more attention to diversity under high-quality conditions in the later stages.

[0185] Step 3: Input the dataset of questions to be generated into the diversity question generation model to obtain several questions. Calculate the confidence of all the questions and sort them from high to low confidence to obtain an ordered list of candidate questions.

[0186] Step 3.1: Load the trained diversity question generation model and read the test samples from the dataset containing the questions to be generated. Each test sample includes highlighted context text, answer text, and reference question text. Then, perform the same input preprocessing as during training on the test samples. The highlighted context text serves as the encoder input, the answer text as the decoding prompt input, and the reference question text is mainly used for subsequent evaluation. For any test sample, under the condition that the model parameters are fixed and not updated, repeat the forward sampling multiple times to obtain multiple candidate questions. Define the i-th test sample as . The corresponding set of candidate questions is denoted as . The expression is:

[0187]

[0188] in, Indicates the number of times a single test sample is sampled. Indicates that the i-th test sample is in the i-th position. Candidate questions are generated under each sampling. Optionally, in this embodiment, the number of samplings is set to 5. During each sampling generation process, the two-stage structured generation mechanism of the training phase is used, i.e., first generating the reasoning basis, then generating the question, and using special markings. <r>and <q>By separating intermediate states, only the question portion is retained as candidate question text in the final output, thereby enabling the generation of diverse candidate questions for the same test sample, providing a basis for subsequent confidence ranking and quality evaluation.

[0189] Step 3.2: For the generated candidate problem set The confidence score of each candidate problem is calculated, and the candidates are sorted from highest to lowest confidence score to obtain an ordered list of candidate problems. Specifically, for any candidate problem... The confidence level is determined by the logarithmic probability of the reasoning basis and the logarithmic probability of the question component when the candidate was generated. The confidence level is defined as... The expression is:

[0190]

[0191] in, This represents the reasoning basis generated for the i-th test sample during the k-th sampling. This represents the log-forward probability of the reasoning basis. This represents the log-forward probability of the problem part. The expression indicates that the measurement of confidence in candidate questions does not rely solely on the question text itself, but simultaneously considers the generation stability of both the reasoning basis and the question itself. This allows the confidence ranking result to reflect the comprehensive characteristics of whether the generation path is stable and whether the question output is reliable. Furthermore, by ranking all candidate questions generated from the same test sample according to their confidence from high to low, we obtain... ,satisfy This indicates the candidate problem that is ranked first after sorting. The most reliable output result for the test sample under the current model is considered. After sorting, the ordered candidate question list corresponding to each test sample and the reference question are written together into the results file to form the standardized output format required for subsequent evaluation, and then written into a JSONL file.

[0192] Step 3.3: Based on the output of ordered candidate questions, the model's inference results are quantitatively evaluated using relevant metrics reflecting question quality and candidate diversity. Specifically, this embodiment uses the Top-1 metric, Oracle metric, and Self metric to systematically evaluate the trained diverse question generation model from three dimensions: the quality of the most confident candidate, the optimal potential of the candidate set, and the diversity within the candidate set. This comprehensively verifies the overall optimization effect achieved by the method of this invention in terms of question quality and candidate diversity. Specifically, the Top-1 metric evaluates the overall matching degree between the candidate question with the highest confidence in each test sample and the reference question; the Oracle metric evaluates the optimal generation potential achievable by the question closest to the reference question in the candidate question set of each test sample; and the Self metric evaluates the similarity between multiple candidate questions within the same test sample to reflect the diversity level of the candidate question set.

[0193] Furthermore, to verify the effectiveness of the present invention, this embodiment conducted an experiment on automatic machine evaluation. The evaluation was performed on three datasets: SquAD1, SquAD2, and NewsQA, and three evaluation metrics: Top-1, Oracle, and Self were used. The experimental results are shown in Table 1.

[0194] Table 1 Experimental Results

[0195]

[0196] Table 1 above lists the machine-automated evaluation results of the model and several methods of this invention (the model reduction forms in the table are all existing problem generation models): Through experimental verification, this invention demonstrates that on three public datasets, the method of this invention exhibits the best performance on the diversity metric Self, and also shows excellent performance on other metrics. * indicates experimental results obtained using the original code of this invention. The model of this invention has better experimental results than the state-of-the-art methods. Therefore, this invention can improve the problem of relying on scalar rewards, which makes it difficult to characterize the multi-attribute combination features of high-quality problems and easily leads to candidate problem templates, resulting in an imbalance between quality and diversity, thereby improving the diversity and quality of problem generation.

[0197] In summary, this invention first constructs data including context, highlighted context, answer, question, and reasoning basis. Then, based on a sequence-to-sequence generation model, it uses a decoding mechanism to generate candidate questions and calculates a basic reward for the candidate results by combining question quality rewards and reasoning basis rewards. Furthermore, it uses axiomatic fuzzy set concepts to map the multidimensional quality attributes of candidate questions to a concept space and uses complex concepts to characterize the distribution characteristics of high-quality candidate questions, improving the quality of question generation. Further, it applies quality-aware gating to candidate questions during the generation stage, using gating to guide the model to learn a more stable candidate distribution, thereby improving the diversity of candidate questions while maintaining question quality. This invention can achieve a synergistic improvement in question generation quality and diversity, and is applicable to various scenarios such as educational assessment, intelligent question answering, reading comprehension, dialogue systems, and knowledge augmentation generation.

[0198] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.< / q> < / r> < / sep> < / q> < / r> < / q> < / r>

Claims

1. A method for generating diverse questions based on conceptually structured rewards and quality-aware gating, characterized in that, The method includes the following steps: Step 1: Obtain the original dataset for the question generation task, preprocess the original dataset to obtain the sample dataset; construct a supervised fine-tuning question generation model and a reward function model, and train the supervised fine-tuning question generation model and the reward function model using the sample dataset; Step 2: Based on the sample dataset, perform two-stage candidate question generation to obtain candidate samples. Obtain the basic reward of the candidate samples through the trained reward function model, extract the four-dimensional quality signal of the candidate samples, construct a concept-structured reward, introduce quality-aware gating, obtain the quality-aware gating reward of the candidate samples based on the four-dimensional quality signal and the concept-structured reward, weight the quality-aware gating reward with the basic reward to obtain the final reward value, and perform reinforcement learning on the trained supervised fine-tuning question generation model and reward function model based on the final reward value to obtain a diverse question generation model; Step 3: Input the dataset of questions to be generated into the diversity question generation model to obtain several questions. Calculate the confidence of all the questions and sort them from high to low confidence to obtain an ordered candidate question list.

2. The method for generating diverse questions based on conceptual structured rewards and quality-aware gating according to claim 1, characterized in that, The process of obtaining the original dataset for the problem generation task and preprocessing the original dataset to obtain the sample dataset specifically involves: Define the original dataset used for the problem generation task as follows: The expression is: ； in, Represents the total number of samples. This represents the context text of the i-th sample. This represents the highlighted context text of the i-th sample. This represents the answer text for the i-th sample. This represents the target question text for the i-th sample. The text representing the reasoning basis for the i-th sample; The context text, highlighted context text, answer text, target question text, and reasoning basis text are segmented and encoded to obtain the corresponding token sequence representations, as shown in the expression: ； in, For word segmentation encoder, This represents the context encoding sequence of the i-th sample. This represents the highlighted context encoding sequence of the i-th sample. This represents the answer encoding sequence for the i-th sample. This represents the target question encoding sequence of the i-th sample. This represents the encoding sequence of the reasoning basis for the i-th sample.

3. The method for generating diverse problems based on conceptual structured rewards and quality-aware gating according to claim 2, characterized in that, The specific steps of constructing the supervised fine-tuning problem generation model and the reward function model, and training the supervised fine-tuning problem generation model and the reward function model using the sample dataset are as follows: The supervised fine-tuning question generation model employs a pre-trained sequence generation model based on an encoder-decoder architecture, and performs supervised fine-tuning based on the pre-trained parameters. By introducing reasoning text, the question generation process is represented as a structured sequence learning task that first generates reasoning and then generates the question. This allows the supervised fine-tuning question generation model to learn the intermediate semantic information between the context, the answer, and the question in the initial stage. For the i-th training sample, the structured target output sequence in the supervised fine-tuning stage... Defined as: ； in, <r>Indicates that the reasoning is based on the starting marker. <q>Indicates the starting marker of the question. Indicates the end of the sequence;< / q> < / r> The supervised fine-tuning question generation model predicts the token at the current position at each time step based on the highlighted context, the answer, and tokens generated before the current time step. The loss function of the supervised fine-tuning stage is then applied. Defined as cross-entropy loss, the expression is: ； in, This represents the total number of samples used for supervised fine-tuning training. This represents the length of the structured target output sequence corresponding to the i-th training sample. The parameters represent the model generated from the problem in the supervised fine-tuning phase. The conditional probability distribution given by the defined supervised fine-tuning problem generation model. This represents the target character at position t in the structured target output sequence of the i-th training sample. This represents the target character subsequence that has been given up to the t-th position in the i-th training sample; The reward function model includes a question quality reward model and a reasoning consistency scoring model. The question quality reward model uses an encoder-decoder model as its basic structure and calculates the conditional log probability of each character based on the context encoding results and the concatenated target sequence. This process progressively scores the question-answer joint sequence to obtain the final temperature-scaled question quality reward. The expression is: ； in, For the candidate question sequence, Indicates the length of the candidate problem sequence. For the answer sequence, Indicates the length of the answer sequence. The parameters represent the problem quality reward model. The temperature parameter represents the problem quality reward model, and c represents the context text. This represents the character at position t in the question-answer sequence z. This represents the prefix character subsequence in the combined sequence z preceding the t-th position; The reasoning consistency scoring model employs a natural language reasoning model, feeding the context text and the reasoning text as a pair of inputs into a text entailment classifier, and using the log probability of the entailment category as the reasoning consistency reward. Thus, the reward function of the reward function model is obtained as follows: .

4. The method for generating diverse questions based on conceptual structured rewards and quality-aware gating according to claim 3, characterized in that, The two-stage candidate question generation based on the sample dataset, to obtain candidate samples, specifically involves: Given the highlighted context text and the answer text, a sequence of reasoning basis is first generated, then a sequence of questions is generated based on the sequence of reasoning basis, and the sequence of reasoning basis and the sequence of questions are connected by a special delimiter symbol to obtain candidate samples.

5. The method for generating diverse questions based on conceptual structured rewards and quality-aware gating according to claim 4, characterized in that, The extraction of the four-dimensional quality signal of the candidate samples and the construction of the conceptual structured reward are specifically as follows: ； In the formula, For four-dimensional quality signals, This represents the answerability signal of candidate sample x. Indicates smoothness signal, Indicates correlation signal, Indicates diversity signals; The expression for the responsiveness signal is as follows: ； In the formula, This represents an overlap function based on a character set. Rewards for quality issues. , , Let represent the weighting coefficients of the responsiveness signal, and satisfy . =1; The expression for the fluency signal is: ； In the formula, This indicates the penalty for repeating candidate questions. This represents the penalty for deviation of the candidate problem from the preset length interval. Indicates penalty for abnormal characters. , , As weight; After obtaining the four-dimensional quality signal of the candidate samples, a concept-structured reward is constructed based on axiomatic fuzzy sets, specifically as follows: First, the historical distribution of the four-dimensional quality signal is cached separately, and then the low threshold and high threshold of each dimension are obtained by the quantile method, forming the segmentation boundary of the three categories of low, medium and high concepts. Secondly, three levels of concepts—low, medium, and high—are constructed for the four dimensions of the four-dimensional quality signal, and positive and negative concepts are assigned to each level to form a set of basic concepts. Then, on the joint universe of discourse formed by the current batch of samples and the experience replay samples, the axiomatic fuzzy set structure generating function is invoked to generate a set of composite concepts. ; Finally, after the composite concept is constructed, the concept structure reward is obtained by aggregating the candidate samples according to their membership in the preferred concept set. The expression is: ； in, Indicates sample For the first Membership degree of each preferred concept, This represents the set of preferred concepts after filtering.

6. The method for generating diverse questions based on conceptual structured rewards and quality-aware gating according to claim 5, characterized in that, The introduction of quality-aware gating, specifically the quality-aware gating reward for candidate samples obtained based on the four-dimensional quality signal and the concept-structured reward, is as follows: The four-dimensional quality signal is standardized using the following expression: ； in, Denotes the standardized answerability term of candidate sample x. Indicates the standardized fluency item. Represents standardized relevance terms. For standardized functions; This yields the overall quality score of the candidate samples. The expression is: ； in, , , Let be the weighting coefficient for the overall quality score, and satisfy . + + =1; Overall quality score By inputting a preset gating function (Sigmoid), the gating value of the candidate samples is obtained, which controls the diversity signal of the candidate samples. Perform standardization processing to obtain standardized diversity items. The standardized diversity term is then gated using the gating values ​​of candidate samples to obtain the quality-perceived gating reward for the candidate samples. The expression is: ； in, This represents the coefficient of the quality enhancement term. This represents the coefficient of the diversity enhancement term. This represents the coefficient for the low-quality penalty term.

7. The method for generating diverse questions based on conceptual structured rewards and quality-aware gating according to claim 6, characterized in that, The step of performing reinforcement learning on the trained supervised fine-tuning problem generation model and reward function model based on the final reward value to obtain a diverse problem generation model is as follows: An experience replay module is constructed based on candidate samples. During reinforcement learning training, online policy sampling and offline policy experience replay sampling are used alternately to update model parameters. A diversity intensity scheduling function is designed to achieve progressive optimization that prioritizes quality in the early stages of reinforcement learning training and gradually increases diversity in the later stages. Specifically, the experience replay module is as follows: The generated sequence, final reward, four-dimensional quality signal, concept-structured reward, and basic reward corresponding to candidate sample x are jointly written into the experience replay module, along with the corresponding replay entries. The expression is: ； in, This represents the final reward value; For new candidate samples and existing candidate samples If the following conditions are met: ； Then determine and The differences between them are insufficient, belonging to approximately duplicate samples; among them, This represents the edit distance function. This represents the similarity tolerance coefficient; When near-duplicate samples appear, only the candidate sample with the higher final reward is retained; When the number of samples in the experience replay module reaches the preset capacity limit, a retention and sampling mechanism based on reward priority is adopted, defining the final reward of the i-th candidate sample in the experience replay module as follows: Then the probability of the i-th candidate sample being far from the policy sampling probability The expression is: ； in, Represents an exponential function. This represents the minimum final reward value among all candidate samples in the experience replay module. This represents the final reward value corresponding to the k-th candidate sample in the experience replay module.

Images