A question generation method and device based on reinforcement learning and a storage medium

Abstract

The application belongs to the field of education and discloses a question generation method and device based on reinforcement learning and a storage medium. The method comprises the following steps: obtaining a question and answer data set, wherein the question and answer data set comprises a plurality of data composed of questions, answers and subgraphs; using the subgraphs to guide the generation of initial questions based on a supervised fine-tuning training mode and the context information of answers; outputting to-be-scored questions according to the answers and subgraphs of the initial questions, labeling and scoring the questioning styles of the to-be-scored questions; labeling the question quality of the to-be-scored questions; calculating the reward scores of the labeled to-be-scored questions; training a question generation model using a reinforcement learning model; and correcting the bias of the scored questions according to the feedback of prior knowledge by the reinforcement learning model. The application uses reinforcement learning to train a question generation model, increases the diversity of question generation and improves the question quality.

Description

Technical Field

[0001] This application belongs to the field of educational technology, and in particular relates to a problem generation method, apparatus and storage medium based on reinforcement learning. Background Technology

[0002] As people's demand for and reliance on online education continues to grow, improving the quality of online education courses and user experience has become a focus. The application of knowledge graph-based intelligent question-answering systems in online education has attracted considerable attention, as it can promptly and accurately resolve problems encountered by students during the learning process.

[0003] Automatic question generation is a natural language question technique that generates questions based on given data formats (e.g., images, text, tables, knowledge graphs). This technology has wide applications in the question-answering (QA) field, enabling the automatic generation of questions for training and enhancing QA systems. It can also assist dialogue agents in formulating questions during human-computer interaction.

[0004] However, traditional rule-based or template-based knowledge graph intelligent question-answering systems have limitations, failing to handle complex natural language problems and even exhibiting ambiguity. Recently, deep learning-based question-answering systems have made new progress, but still face many challenges, such as low answer quality and the existence of flattery bias. This occurs because, as human annotators label data according to predefined priority criteria, the machine gradually learns these priorities through multiple interactions and outputs the content most valued by the annotators. While this is beneficial in aligning machine and human preferences, it can be overdone, sometimes even ignoring other criteria—a phenomenon known as "flattery." For example, if the criterion of "human-like questioning style" is considered the highest priority, the model may overemphasize this criterion in later feedback phases, producing outputs that more closely match human preferences while ignoring other factors, such as the relevance of the question to the answer. Alignment of machine and human preferences can be beneficial, but sometimes it can become so obvious that it ignores other criteria, thus affecting the quality of the text itself.

[0005] Therefore, knowledge base question generation (KBQG) has attracted increasing interest in recent years. Research on KBQG mainly falls into two categories: template-based models and neural network-based models. The former relies heavily on manually created templates, resulting in questions that are often stereotyped, lack diversity, and are not scalable. Neural models take triples related to the answer as input and employ a Seq2Seq architecture to automatically generate questions. While these methods can generate fluent questions, they also have a significant drawback: the generated questions often lack diversity, making these models less applicable in real-world scenarios. Knowledge base question generation (KBQG) aims to output natural language questions based on knowledge graphs (KGs) and target answers. However, existing KBQG solutions suffer from low diversity.

[0006] Figure 1 This further illustrates the problem. The right side of the diagram presents questions posed by humans, possessing unique and realistic logic and style. In contrast, while the questions generated by existing models (left side of the diagram) are meaningful and conform to the given "standard questions," most are unnatural and lack diversity.

[0007] In recent years, influenced by GPT, there has been a growing interest in reinforcement learning from prior knowledge feedback. Early research focused primarily on training agents to achieve predefined goals through feedback. Some studies utilized expert demonstrations and preference feedback, employing imitation learning strategies for initialization to further improve policy performance. However, feedback efficiency remains a concern. Some literature combines unsupervised pre-training and experience relabeling techniques to improve feedback efficiency. Other research employs semi-supervised learning and data augmentation to learn reward functions. Still others leverage the uncertainty of rewards to reduce the number of preference labels, promoting exploration. However, these methods only focus on guiding reward learning through supervised loss between preferences and estimated preferences. When feedback samples are limited, estimation bias can be significant, leading to biased reward learning and affecting the final result. This results in confirmation bias, where one neural network overfits another. Summary of the Invention

[0008] In view of this, in order to solve this problem and make the problem more realistic, inspired by reinforcement learning based on prior knowledge feedback (RLHF, the underlying algorithm of ChatGPT), this application proposes a reinforcement learning framework for bias correction in KBQG.

[0009] This application trains a cold-start question generation model, QG-SFT (Cold Start Question Generation Model with Supervised Fine-Tuning), capable of generating basic, coherent question sentences. Then, based on prior knowledge and inherent question quality, two reward models are trained: a question quality reward model to score questions generated by QG-SFT; and a question style reward model to encourage diverse question generation. The question quality reward model mitigates flattery bias by emphasizing question quality, such as punctuation, vocabulary, grammar, and whether the question matches the answer. Furthermore, to alleviate annotation bias during the annotation process, this application employs a game theory mechanism to improve annotation quality by coordinating two competing annotation models. Finally, QG-SFT is used to initialize BC-RLHF, generating initial candidate questions for the reinforcement learning process. To mitigate confirmation bias, this application proposes a method to correct confirmation bias and mitigate the impact of scarce feedback annotation data, combining it with the two reward models to generate a bias correction reward. Subsequently, the agent receives reward feedback for bias correction and updates its parameters, ultimately arriving at a more realistic and coherent problem.

[0010] It is worth noting that, unlike the original RLHF model, which can become inaccurate when feedback is limited or of low quality, this application designs a reward model and inference procedure to avoid potential bias.

[0011] To achieve the above objectives, the first aspect of this application discloses a problem generation method based on reinforcement learning, comprising the following steps:

[0012] Obtain a question-and-answer dataset, which includes multiple datasets consisting of questions, answers, and subgraphs;

[0013] Based on supervised fine-tuning training, the initial question is generated using the contextual information of the subgraph to guide the answer.

[0014] Based on the answers to the initial questions and the subgraph, output the questions to be scored; annotate and score the question style of the questions to be scored; annotate the question quality of the questions to be scored; calculate the bonus score for the annotated questions to be scored.

[0015] A problem generation model is trained using a reinforcement learning model, which corrects for bias in the scored problems based on prior knowledge feedback.

[0016] After multiple rounds of training, a trained model is obtained. The collected real-world problems are then input into the model, and the final problem is output.

[0017] Preferably, question style annotation and question quality annotation are used to automatically assign reward scores to the generated questions. That is, the answer and subgraph information are used as prompts to encode the previously generated initial questions into vector representations, which are then processed by pooling layers and linear transformation layers to generate one-dimensional style reward scores and one-dimensional quality reward scores, respectively.

[0018] Preferably, the annotation of the question style of the questions to be rated includes at least one of the following methods: large model annotation, manual annotation, and human-computer collaboration.

[0019] Preferably, game theory mechanisms are also used to improve the quality of the question style annotation, including:

[0020] Two independent annotation models, 1 and 2, of different types are set up for question style annotation, and a quality control model is set up for quality inspection. In each cycle, each annotation model annotates multiple sets of questions, and then submits a batch of annotated data to the quality control model. and

[0021] The quality inspection model evaluates the quality of annotations by scoring each problem in the annotation model; for problems with scores below a preset value, [the quality is adjusted accordingly]. or Delete it and return it to the annotation model so that it can be re-annotated in the next cycle, with the rejection reasons provided by the quality inspection model attached;

[0022] Will and The average score of the two annotation models is used to obtain the total score of annotation model 1 and annotation model 2. In order to incentivize the two annotation models to provide higher quality annotation data, the unit price of annotation data in each cycle is jointly determined by the annotation quality of the two annotation models. The profit of annotation model 1 and annotation model 2 is calculated based on the annotation cost of each sample.

[0023] After continuous independent decision-making and game between the two annotation models, the quantity and quality of annotations eventually stabilize at a certain value and reach an equilibrium point E.

[0024] Preferably, the issue quality labeling is used to improve the quality of the issue itself, and the issue quality labeling includes at least one of the following sub-modules in the issue quality assessment:

[0025] The text accuracy submodule and the symbol accuracy submodule are used to evaluate the quality of characters;

[0026] The vocabulary diversity submodule and the vocabulary accuracy submodule are used to evaluate the quality of vocabulary;

[0027] The syntactic complexity submodule and the syntactic correctness submodule are used to evaluate the quality of the syntax;

[0028] The clarity and completeness submodules are used to evaluate the quality of the content;

[0029] An evaluation submodule assesses the logical consistency and smoothness between questions and answers, used to evaluate the quality of the interaction;

[0030] The question-and-answer matching submodule and the practical value submodule are used to evaluate the overall effectiveness;

[0031] For each submodule, binary labeling is performed: if the answer to the question is yes, it is 1; otherwise, it is 0. Then, the labeling results of all submodules are summed to obtain the total question quality score. These questions are then sorted in descending order according to their total question quality scores to generate a priority sequence.

[0032] Preferably, the environment and agent in the reinforcement learning model are combined with a bias-correction reward function to obtain feedback for training the model; the environment includes a question, an answer, and a subgraph; the agent is a cold-start question generation model.

[0033] Based on the data provided by the environment, the agent initializes itself. Then, the agent observes the standard questions, answers, and subgraphs in the environment and generates an initial question, i.e., an action, which is stored in a replay buffer. In order to calculate the reward and optimize the agent, the questions are sampled from the replay buffer and passed to a reward predictor, which calculates the reward score and the bias correction score.

[0034] Preferably, the reward function for bias correction and the entropy regularization that makes policy π correspond to the action distribution in state s are used together as the loss function to avoid overfitting, and the reward function for bias correction includes at least one of the following:

[0035] The semantic distance between two texts is calculated using the Word Mover's Distance model at the word level, which is used to evaluate the semantic similarity between the text generated by the strategy and the target text;

[0036] At the sentence level, the ROUGE-L model is used to calculate the similarity between the automatically generated text and the reference text, which is used to evaluate the similarity between the sentences generated by the strategy and the target sentences.

[0037] The difference is calculated using a character-level similarity metric, wherein the character-level similarity metric includes at least one of the Levenshtein distance or a character-level language model.

[0038] The difference can be calculated using semantic similarity measures between paragraphs at the paragraph level, such as similarity measures based on paragraph vector representations;

[0039] The overall effectiveness of the strategy is comprehensively evaluated based on the differences in the text generated by the strategy at different levels.

[0040] Preferably, the reinforcement learning model is based on data provided by the environment, namely, an annotated question-and-answer dataset and knowledge base. The agent generates questions and stores these questions in an experience pool. The agent is a cold-start question generation model. In order to calculate rewards and optimize the agent, questions are sampled from the experience pool and forwarded to the reward predictor to generate bias correction rewards. The bias correction rewards include question style rewards, question quality rewards, and bias correction. Based on the reward results, the agent is optimized using a proximal policy optimization algorithm.

[0041] Preferably, the deviation correction includes:

[0042] Using the aforementioned questioning style reward and question quality reward, scores are generated based on prior knowledge, logic, and question quality.

[0043] Calculate the deviation correction score to mitigate the baseline problem Q. b Sample Problem Q s and the target question Q t The confirmation bias arises from word-level similarity and sentence-level similarity; among them, the baseline problem Q b It is the model output obtained by selecting the best action using the current policy. It serves as a reference standard during training and is used to evaluate the sample problem Q. s Quality, Sample Issues Q s It is the model output obtained by randomly sampling from the current policy.

[0044] Preferably, the word-level reward in the question-style reward model aims to positively reward the sample question Q. s If Q s Compared to baseline problem Q b Closer to the target question Q t ;

[0045] Sentence-level rewards use ROUGE-L to measure similarity. The reward objective of ROUGE-L is to positively reward the sample problem Q. s If Q s With the target question Q t Compared to the baseline problem Q b More similar;

[0046] The reward r after bias correction total It includes reward for question style, reward for question quality, reward for words, reward for sentences, and a baseline question Q measured according to the BLEU-4 metric. b And sample problem Q s With the target question Qt The sum of the rewards.

[0047] Preferably, based on the reward r total The agent is trained iteratively, and during the training process, an entropy regularization term is used to make the policy distribution more uniform; for the discrete probability distribution on the action space |A|.

[0048] The second aspect of this application discloses a problem generation apparatus based on reinforcement learning, comprising:

[0049] The acquisition module is used to acquire a question-and-answer dataset, which includes multiple data consisting of questions, answers, and subgraphs;

[0050] The initial question generation module, based on supervised fine-tuning training, uses the subgraph to guide the contextual information of the answer to generate the initial question;

[0051] The question reward module outputs questions to be scored based on the answers and subgraphs of the initial questions. It also marks and scores the question style of the questions to be scored, marks the question quality of the questions to be scored, and calculates reward scores for the marked questions to be scored.

[0052] The reinforcement training module uses a reinforcement learning model to train a problem generation model, and the reinforcement learning model corrects the bias of the scored problems based on feedback from prior knowledge.

[0053] The output module, after multiple rounds of training, produces a trained problem generation model. The actual problem is then input into the problem generation model, and the final problem is output.

[0054] A third aspect of this application discloses an electronic device comprising a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0055] The fourth aspect of this application discloses a computer-readable storage medium comprising a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the method.

[0056] A computer program product disclosed in the fifth aspect of this application includes a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the steps of the method.

[0057] The technical solutions provided in this application have at least the following beneficial effects:

[0058] This application trains a preference-based question reward model by labeling an initial question set and using a pre-set reward loss function. The trained model generates relatively fluent questions and scores them, improving scoring accuracy. A reinforcement learning module is then trained using a pre-set reinforcement loss function. This trained module is used to construct a question generation environment. Within this environment, multiple candidate questions are generated using the initial question generation model and standard questions, answers, and subgraphs. The reinforcement learning module is trained again using the pre-set reinforcement loss function, resulting in a more even distribution of policies when constructing the question generation environment, thus improving the accuracy of candidate question generation. The initial question generation model observes the "standard questions," answers, and subgraphs in the environment to generate candidate questions. These candidate questions are generated in conjunction with specific real-world scenarios, demonstrating high applicability. The trained preference-based question reward model then scores these candidate questions, and bias correction is used to adjust the scoring results. However, the trained preference-based question reward model may not function properly when the feedback is inaccurate, resulting in poor performance. This application employs a question quality reward model, mitigating bias by emphasizing question quality, such as punctuation, vocabulary, grammar, and whether the question matches the answer. Furthermore, to alleviate labeling bias in the style question labeling process, this application uses a game theory mechanism to improve the labeling quality of style questions by coordinating two competing labeling models. To mitigate potential confirmation bias, this application proposes a reinforcement learning method for bias correction, learning from feedback to correct confirmation bias. The scoring results are corrected through bias correction to ensure quality. The initial question generation model is trained using the bias-corrected reward and a proximal policy optimization algorithm to iteratively optimize the strategy, resulting in a well-trained question model. Diverse questions are generated based on this trained question generation model, thereby increasing the diversity and quality of question generation. Attached Figure Description

[0059] Figure 1 Problems existing in the knowledge base question generation domain;

[0060] Figure 2 A method flowchart is provided in one embodiment of this application;

[0061] Figure 3 This application provides a diagram of a reinforcement learning-based bias correction framework according to one embodiment;

[0062] Figure 4 This application provides a structural diagram of a knowledge graph problem generation device according to one embodiment;

[0063] Figure 5 This application provides a structural diagram of a knowledge graph problem generation device according to one embodiment. Detailed Implementation

[0064] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0065] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0066] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0067] The technical solutions provided in this application involve technologies such as machine learning and natural language processing in artificial intelligence, which are specifically described and illustrated through the following embodiments.

[0068] Before introducing the embodiments of this application, some terms involved in this application will be explained.

[0069] 1. Knowledge Graph: A series of different graphs that display the development process and structural relationships of knowledge. It uses visualization technology to describe knowledge resources and their carriers, and to mine, analyze, construct, draw and display knowledge and the interrelationships between them.

[0070] A knowledge graph can include multiple nodes and edges connecting them. Nodes in a knowledge graph include different types such as entity nodes and value nodes. Each entity node represents an entity, and each value node represents a value.

[0071] 2. Knowledge Base Question Generation (KBQG): A knowledge base (KB), often referred to as a knowledge graph (KG), is generated by...<s,r,o> This set of triples, where s, r, and o represent the subject, relation, and object, respectively. A subgraph is a subset of a knowledge graph that contains one or more triples and can be represented as G = (V, E), where... This represents the set of entities (s or o) in the knowledge graph. This represents all the relationships that connect these entities. Let ε and ε represent the sets of all entities and relations in the knowledge graph, respectively. The knowledge base question generation task mainly involves taking a subgraph G = (V, E) as input and the answer... Then generate more human-like and realistic natural language questions Q. G The training data for this task consists of a set of triples: <knowledge graph subgraph G, answer V> t Target question Q t >

[0072] Current knowledge base question generation models are mainly divided into template-based and neural network-based methods. Typically, triples in a knowledge graph subgraph are organized into sequences and fed as input to a Seq2Seq neural network (L2A).

[0073] 3. Reinforcement Learning: Reinforcement learning training requires feedback from the environment, along with specific feedback values. Reinforcement learning includes the environment, agent, action, state, and reward. It is not a classification task; its main purpose is to guide the training subject in making decisions at each step, determining which actions will achieve a specific goal or maximize benefits.

[0074] 4. The Proximal Policy Optimization (PRO) algorithm is a model-free reinforcement learning algorithm based on policy gradients. Its main idea comes from the TRPO algorithm, replacing the KL divergence with a Clip function and directly constraining the policy through the ratio rt(θ). In a given state, the agent chooses action A and receives a reward R. The policy is then optimized using policy gradients to maximize the overall reward.

[0075] Please refer to Figure 2 The diagram illustrates a schematic flow chart of an embodiment of this application. The diagram includes four steps:

[0076] Step S1: Obtain the question-and-answer dataset, which includes multiple data consisting of questions, answers, and subgraphs.

[0077] Step S2: Based on supervised fine-tuning training, generate initial questions using the contextual information of the subgraph guiding the answer. In one embodiment, a cold-start question generation model is trained using supervised fine-tuning. This model can generate basic fluent natural language questions, which can be used as initial questions.

[0078] Supervised fine-tuning of cold-start question generation utilizes state-of-the-art (SOTA) models in the current knowledge graph question generation field. These SOTA models can be G2S, G2S-AE, IGND, or G2S-AE-RL models, which leverage subgraph guidance to utilize contextual information from the answers, generating more accurate and fluent natural language questions. For example, training data is provided to the G2S-AE-RL model for fine-tuning, generating initial questions for subsequent training. The G2S-AE-RL model is one of the latest KBQG models, employing a novel BiGraph2Seq model to generate questions and answers from KB subgraphs. This application does not limit the choice of baseline model; many models can be selected, including but not limited to: L2A, RNN, Transformer, zero-shot, mhqg, T5, IGND, G2S, G2S-AE, and G2S-AE-RL.

[0079] In one embodiment, step 2, for the supervised fine-tuning training method, using the subgraph-guided contextual information to generate the initial question, includes:

[0080] This application employs the question generation model G2S-AE-RL proposed in 2023, which uses a novel BiGraph2Seq model to generate questions and answers from KB subgraphs. Specifically, this application provides training data in the format described above to fine-tune G2S-AE-RL, which will generate initial questions.

[0081] In one embodiment, step 2, for the supervised fine-tuning training method, using the subgraph-guided contextual information to generate the initial question, includes:

[0082] The application employs a question generation model, L2A, an early KG subgraph, as input to an LSTM-based Seq2Seq model. This model belongs to the attention-based Seq2Seq category and can generate natural language questions in open-domain dialogue systems using contextual information. Specifically, this application provides training data in the format described above to fine-tune L2A, which will generate initial questions.

[0083] In one embodiment, for the supervised fine-tuning training manner, the step 2 of generating an initial question by using the subgraph to guide the context information of the answer comprises:

[0084] A Zero-shot question generation model is adopted, which is a zero-sample RNN-based knowledge base question generation model based on OVV predicates and entity types. Specifically, the present application provides the training data in the format 1 described above to perform fine-tuning on Zero-shot, which will generate a preliminary question.

[0085] In one embodiment, for the supervised fine-tuning training manner, the step 2 of generating an initial question by using the subgraph to guide the context information of the answer comprises:

[0086] An MHQG question generation model is adopted, which is an end-to-end neural network-based complex multi-hop knowledge base question generation model. Specifically, the present application provides the training data in the format 1 described above to perform fine-tuning on MHQG, which will generate a preliminary question.

[0087] Step S3: Output questions to be scored according to the answers to the initial questions and the subgraph, label and score the questioning style of the questions to be scored; label the question quality of the questions to be scored; and calculate a reward score for the labeled questions to be scored. The style questioning labeling is to assign sequence priority labels to the initial questions according to prior knowledge. In one embodiment, step S3 includes the following 3 sub-steps:

[0088] S31 Question generation: Input each <KG subgraph, answer> pair, and use the QG-SFT model trained in step 2 to generate a set of K questions. Multiple sets of data (K questions per set) will be generated simultaneously. Then, all these questions to be labeled will be labeled by a labeling model, so as to generate the sequence priority of the K questions for training the reward model. An example of the input format of the answer and the subgraph is shown below.

[0089] {"answers":["Computer Vision","Face Recognition"],"answer_ids":["10001082","10001100"],"outSeq":"By the way, what knowledge using probabilistic models have you used?","qId":18,"inGraph":{"g_node_names":{"10001082":"Computer Vision","10001100":"Face Recognition","10001092":"Probabilistic Model"},"g_edge_types":{"Computer / Domain / Topic Learning Dependencies":"Computer / Domain / Topic / Learning Dependencies"},"g_adj":"10001082":{"10001092":"Computer / Domain / Topic / Learning Dependencies"},"10001100":{"10001092":"Computer / Domain / Topic / Learning Dependencies"}}}}

[0090] S32: Sequence Priority Labeling: Using a Question-Style Reward Model Problem quality reward model Based on the semantics, syntax, relevance, and questioning style of the questions in the prior knowledge, the generated questions are labeled from good to bad.

[0091] Relative ranking is used to rank the generated questions because it reduces the difficulty of labeling.

[0092] It is worth noting that this application uses two different annotation models in the style question reward model and uses a game theory mechanism to coordinate the two annotation models and stabilize the annotation quality.

[0093] S33: Train the corresponding question-style reward models according to the label sorting priority. Problem quality reward model Where θ is a parameter. After training, the goal is for these two models to automatically score the generated questions. Specifically, using the answer and subgraph information x as hints, the ERNIE model encodes the previously generated question y into a vector representation, which is then processed by pooling layers and linear transformation layers to produce a one-dimensional reward value r. human and r quality .

[0094] In one embodiment, step 3 includes the following steps:

[0095] In S31, given each <knowledge graph subgraph, answer> pair, this module uses the QG-SFT model trained in step 1 to generate a set of K scored questions. Multiple sets of data (K questions per set) will be generated simultaneously. Then, all these questions to be labeled are labeled by human annotators, resulting in the sequence priority of the K questions for training the reward model.

[0096] In S32, relative ranking is used to rank the generated questions, as it captures human preferences and also reduces the difficulty of labeling.

[0097] S33 Question style annotation: It is expected that the questions generated by the machine follow the principles of human-like style, harmlessness and authenticity. To achieve this, 9 volunteers were selected and trained, who were equally divided into two annotation groups (3 student volunteers in each group) and a quality inspection group (3 teacher volunteers in each group). Members of the annotation team independently assign a priority sequence to each set of k questions based on three principles (prioritizing human-like style > harmlessness > authenticity). In case of inconsistency in annotation, the teams will conduct collective discussion and voting. After the annotation process of each team is completed, the quality inspection group conducts a comprehensive check. They score the quality of the annotated questions from the teams and provide feedback. Unqualified questions (less than ε) will be sent back by the quality inspection group for re-annotation.

[0098] S34 Question quality annotation: Simply pursuing human-like style often leads to sycophantic bias. Therefore, as a supplement, the present application designs a multi-granularity question quality annotation module to highlight the quality of the question itself. The present application divides the question quality assessment into 6 layers and 11 sub-modules:

[0099] · Character layer, including text accuracy (whether the text is accurate) and symbol accuracy (whether the symbols are accurate); · Lexical layer, including lexical diversity (whether there is a rich vocabulary quantity) and lexical precision (whether the word choice is accurate);

[0100] · Syntactic layer, including syntactic complexity (whether the sentence length and structure are complex) and syntactic correctness (whether grammar and syntactic rules are followed);

[0101] · Content layer, including question clarity (whether the question has a clear theme and clear logic) and question completeness (whether the question is sufficiently complete);

[0102] · Overall effectiveness layer, including question-answer matching (whether the answer truly answers the question) and practical value (whether useful information or solutions are provided).

[0103] For each submodule, a binary annotation is created: 1 if the answer to the question in parentheses is yes, and 0 otherwise. Then, the annotation results for all submodules are summarized to obtain a total score for the questions in terms of quality. These questions are then sorted in descending order of their scores to generate a priority sequence. Based on this information, the quality control team will evaluate the annotation quality of the annotation model.

[0104] Step S4: Use a bias-corrected reinforcement learning framework to guide further training of the problem generation model from the prior knowledge feedback framework, through a bias-corrected reward function.

[0105] This feedback framework uses the environment and agent in reinforcement learning, combined with a bias-correcting reward function, to obtain feedback from the trained model.

[0106] refer to Figure 3 In one embodiment, based on data provided by the environment, the agent (initialized as QG-SFT in step 2) generates a question and stores it in a replay buffer. To calculate the reward and optimize the agent, the question is sampled from the replay buffer and passed to a reward predictor, which performs two functions: reward value calculation and bias correction score calculation.

[0107] The environment primarily consists of question-and-answer data, including questions, answers, and subgraph information. This application first initializes the agent using QG-SFT in step S2. Then, the agent observes the "standard questions," answers, and subgraphs in the environment and generates questions, i.e., actions.

[0108] Since existing algorithms learn the reward function solely through preferences, they may fail to function properly and perform poorly when feedback is inaccurate. To mitigate potential confirmation bias, this application further proposes a bias correction RLHF framework. Based on the reward model trained in step S3, the reward value r is calculated according to prior knowledge. human and r quality r human and r quality The higher the level, the more diverse the problems become.

[0109] To avoid overfitting of the trained model, this application proposes a method for calculating a bias correction score to mitigate the impact of the baseline problem Q. b Sample Problem Q s and the target question Q t Confirmation bias from the perspectives of word-level similarity and sentence-level similarity.

[0110] In one embodiment, this application uses issue quality labeling to highlight the quality of the issue itself. Issue quality assessment is divided into 6 levels and 11 sub-modules:

[0111] 1) Character layer, including text accuracy (whether the text is accurate) and symbol accuracy (whether the symbols are accurate);

[0112] 2) Lexical layer, including lexical diversity (whether there is a rich variety and quantity of vocabulary) and lexical precision (whether the vocabulary selection is accurate);

[0113] 3) Syntactic layer, including syntactic complexity (whether the sentence length and structure are complex) and syntactic correctness (whether it follows grammar and grammatical rules);

[0114] 4) Content layer, consisting of the clarity of the question (whether the question has a clear theme and clear logic) and the completeness of the question (whether the question is complete enough);

[0115] 5) The interaction layer consists of interaction consistency (whether there is logical consistency and smoothness between the question and the answer);

[0116] 6) The overall validity layer consists of question-answer matching (whether the answer truly answers the question) and practical value (whether it provides useful information or solutions).

[0117] For each submodule, the annotation model is required to perform binary annotation: a value of 1 if the answer to the question in parentheses is yes, and 0 otherwise. Then, the annotation results for all submodules are summed to obtain a total score for the question quality. These questions are then sorted in descending order of their scores to generate a priority sequence. Based on this information, a quality control model evaluates the annotation quality of the annotation model.

[0118] In one embodiment, calculating a reward score for the labeled question to be scored includes:

[0119] We employ annotation ranking priorities for human-like question annotations and multi-granularity question quality annotations, training corresponding reward models Rθhuman quality and Rθquality, where θ is a parameter. After training, we aim for these two models to automatically score generated questions. Specifically, we use the answer and subgraph information x as hints, employing...

[0120] ELECTRA / ALBERT / RoBERTa / ELMo / Bert / Cluade3 / Cluade2 / Cluade / GPT4 / GPT3.5 / GPT3 / GPT2 / GPT / ERNIE / ERNIE2 / ERNIE3 / UniLM / BART encodes the previously generated problem y into a vector representation, which is then processed by a pooling layer and a linear transformation layer to produce a one-dimensional reward value and quality, respectively.

[0121] Then, based on the reward, a loss function was designed:

[0122]

[0123]

[0124] Where (K, 2) represents a comparison with two questions. D is a labeled dataset of multiple annotated sequence sets. For each sequence set, i.e., K questions, a reward model is used to generate a reward value for each question, where yb represents the question ranked before y. σ is a function that transforms the difference between 0 and 1 into a σ-type function. The objective of this application is to maximize the score difference between good and bad questions, which is achieved by taking the negative of the loss.

[0125] These two reward models were trained to score the questions.

[0126] In some embodiments, the method for calculating the reward score in step S3 includes, but is not limited to, the following methods:

[0127] (1) Manual evaluation: The questions to be scored are evaluated by experts or human annotators, who then award corresponding bonus points based on their experience and judgment. This method can more accurately assess the quality and relevance of the questions.

[0128] (2) Crowdsourced evaluation: The problem to be rated is posted on a crowdsourcing platform for evaluation by multiple participants. Methods such as majority voting and weighted average can be used to calculate the reward score. This method obtains opinions from multiple participants, avoiding the subjective bias of a single evaluator.

[0129] (3) BLEU (Bilingual Evaluation Understudy): The BLEU metric was first used in the field of machine translation. It is an indicator used to evaluate the difference between sentences generated by the model and actual sentences. Its core idea is to compare the degree of overlap of n-grams in the candidate translation and the reference translation. The higher the degree of overlap, the higher the quality of the translation is considered. Unigrams are used to measure the accuracy of word translation, while higher-order n-grams are used to measure the fluency of sentence translation. In practice, N = 1 to 4 is usually used, and then a weighted average is performed.

[0130] (4) ROUGE (Recall-Oriented Understudy for Gisting Evaluation): The ROUGE metric is a common evaluation metric in machine translation, automatic summarization, question answering generation, and other fields. Both the ROUGE and BLEU metrics can be used to measure the degree of matching between the generated results and the standard results. The difference is that ROUGE is based on recall, while BLEU places more emphasis on accuracy.

[0131] (5) METEOR (Metric for Evaluation of Translation with Explicit Ordering): In 2005, Banerjee and Lavie invented the METEOR metric, based on single-precision weighted harmonic mean and single-word recall, to address some inherent shortcomings in the BLEU standard. METEOR extends the BLEU concept of "co-occurrence" by proposing three modules for statistically analyzing co-occurrence: the "exact" module, the "porter stem" module, and the "WN synonymy" module. METEOR also incorporates word order into the evaluation, establishing a penalty mechanism based on word order changes. When the word order of the translation being tested differs from the reference translation, an appropriate penalty is applied. Finally, precision, recall, and F-score are calculated based on the co-occurrence count, and the penalty is considered to obtain the final METEOR value for the translation being tested.

[0132] (6) CIDEr (Consensus-based Image Description Evaluation): CIDEr is a combination of BLEU and vector space model. It treats each sentence as a document and then calculates the cosine angle of the TF-IDF vectors (except that the term is an n-gram instead of a word) to obtain the similarity between the candidate sentence and the reference sentence.

[0133] (7) SARI: SARI focuses on evaluating the lexical, structural and semantic changes between the generated sentence and the reference sentence, taking into account three sub-indicators: Additions (measures the difference between the newly added words or phrases in the generated translation and the reference translation), Deletions (measures the presence of missing words or phrases in the reference translation in the generated translation), and Keepings (measures the proportion of words or phrases in the generated translation that are the same as those in the reference translation).

[0134] (8) LEPOR: LEPOR scores sentences by combining various factors, such as length penalty, precision, n-gram position difference penalty, and recall. The n-gram position difference penalty in LEPOR aims to compare the word order between the reference sentence and the output sentence, and the penalty value is normalized. The calculation of the n-gram position difference penalty involves alignment and calculating the n-gram position difference of the aligned words between the output sentence and the reference sentence.

[0135] (9) BERTScore: This is a paper proposed by Cornell University at ICLR 2020 (paper: BERTSCORE: EVALUATING TEXT GENERATION WITH BERT). It is mainly based on the BERT pre-trained model, uses contextual embedding to describe sentences, and calculates the cosine similarity between two sentences.

[0136] (10) GEMBA: This is a method proposed by Kocmi and Federmann to use an LLM evaluator to assign quality scores (0 to 100) to evaluate generated translations, regardless of whether there is a reference. GEMBA demonstrates the effectiveness of using an LLM of GPT-3.5 or higher for translation quality evaluation.

[0137] In one embodiment, step S1, which involves obtaining a question-and-answer dataset, includes multiple datasets consisting of questions, answers, and subgraphs, including:

[0138] The question-and-answer dataset mentioned is WebQuestions, which was proposed in 2013 and constructed by Stanford University researchers using the Google Suggest API. The dataset itself contains 5810 question-and-answer pairs, with simple questions accounting for 84%, and relatively few complex multi-hop and reasoning questions. According to the proposer's initial data partitioning method, WebQuestions is divided into two sets: a training set containing 3778 data points and a test set containing 2032 data points.

[0139] {"answers":["Portugal national football team"],"answer_ids":[" / m / 02rqxc"],"outSeq":"which of the sports teams for whom cristianoronaldo playedwas founded most recently?","qId":20990,"inGraph":{"g_node_names":{" / m / 02sf29t":"none"," / m / 02rqxc":"Portugal national football team"," / m / 02xt6q":"Cristiano Ronaldo","1914":"1914"},"g_edge_types":{" / sports / sports_team_roster / team": " / sports / sports_team_roster / team"," / sports / pro_athlete / teams":" / sports / pro_athlete / teams"," / sports / sports_team / founded": " / sports / sports_team / founded"}, "g_adj":{" / m / 02sf29t":{" / m / 02rqxc":" / sports / sports_team_roster / team"}," / m / 02xt6q":{" / m / 02sf29t":" / sports / pro_athlete / teams"}," / m / 02rqxc":{"1914":" / sports / sports_team / founded"}}}}

[0140] {"answers":["Burj Khalifa"],"answer_ids":[" / m / 03cn0v"],"outSeq":"which attractions in dubai are greater than 26 stories tall / ?","qId":20991,"inGraph":{"g_node_names":{" / m / 03cn0v":"Burj Khalifa","163":"163"," / m / 01f08r":"Dubai"}, "g_edge_types": {" / architecture / building / floors":" / architecture / building / floors", " / travel / travel_destination / tourist_attractions":" / travel / travel_destination / tourist_attractions"},"g_adj":{" / m / 03cn0v":{"163":" / architecture / building / floors"}," / m / 01f08r":{" / m / 03cn0v":" / travel / travel_destination / tourist_attractions"}}}}

[0141] {"answers":["Hawaii-Aleutian Time Zone"],"answer_ids":[" / m / 02lctm"],"outSeq":"find the state that has a nene as the official symbol,what time zoneis this state in?","qId":20992,"inGraph":{"g_node_names":{" / m / 03gh4":"Hawaii"," / m / 02lctm":"Hawaii-Aleutian Time Zone"," / m / 071qmv":"UTC\u221210:00"," / m / 04st848":"none"," / m / 0139l8":"Nene"},"g_edge_types":{" / location / location / time_zones":" / location / location / time_zones"," / location / location_symbol_relationship / symbol":

[0142] " / location / location_symbol_relationship / symbol",

[0143] " / government / governmental_jurisdiction / official_symbols":

[0144] " / government / governmental_jurisdiction / official_symbols"},"g_adj":{" / m / 03gh4":{" / m / 02lctm": " / location / location / time_zones", " / m / 071qmv":" / location / location / time_zones", " / m / 04st848":" / government / governmental_jurisdiction / official_symbols"}, " / m / 04st848":{" / m / 0139l8":" / location / location_symbol_relationship / symbol"}}}}。

[0145] In one embodiment, step S1, which involves obtaining a question-and-answer dataset, includes multiple datasets consisting of questions, answers, and subgraphs, including:

[0146] The question-and-answer dataset is the EduQA online education question-and-answer dataset. An example of the data format is as follows: Question: Comrades from the fourth northeastern province, do you know what types of cognitive science disciplines exist?

[0147] Answers: ["Cognitive Science: Learning and Developmental Psychology", "Cognitive Science: Perception and Behavior", "Cognitive Science: Artificial Intelligence", "Cognitive Science: Memory", "Cognitive Science: Language", "Cognitive Science: Attention"];

[0148] Subgraphs: ["Cognitive Science Discipline Scope", "Computer / Domain / Topic / Facetation / Hyper-orientation", "Cognitive Science Learning and Developmental Psychology"], ["Cognitive Science Discipline Scope", "Computer / Domain / Topic / Facetation / Hyper-orientation", "Cognitive Science Perception and Behavior"], ["Cognitive Science Discipline Scope", "Computer / Domain / Topic / Facetation / Hyper-orientation", "Cognitive Science Artificial Intelligence"], ["Cognitive Science Discipline Scope", "Computer / Domain / Topic / Facetation / Hyper-orientation", "Cognitive Science Memory"], ["Cognitive Science Discipline Scope", "Computer / Domain / Topic / Facetation / Hyper-orientation", "Cognitive Science Language"], ["Cognitive Science Discipline Scope", "Computer / Domain / Topic / Facetation / Hyper-orientation", "Cognitive Science Attention"].

[0149] In one embodiment, step S1, which involves obtaining a question-and-answer dataset, includes multiple datasets consisting of questions, answers, and subgraphs, including:

[0150] The question-and-answer dataset described is the China Legal Research Cup Question-and-Answer Dataset, which provides dataset support for legal education-related work. Specifically, it consists of user questions and several answers providing legal consultation, serving as a real question-and-answer dataset. The task for participants is to output the corresponding correct, complete, and concise reference answers. The training, validation, and test sets used for the technical evaluation of this task are all from the internet, containing approximately 24,000 questions. Each question includes 2-4 lawyer answers and a corresponding standard summary answer. Question: Which option is a form of state compensation? Answer: Monetary awards. Subgraph text: Monetary awards is a form of state compensation.

[0151] Question: Who is the principal offender according to Criminal Law? Answer: Bob, the leader of a robbery group, who ordered his subordinates to commit robbery on multiple occasions, but was never personally involved.

[0152] Subplot text: The principal ofender is the person in a group of offenderswholeads, and carries out the main part of a criminal act.

[0153] In one embodiment, step S1, which involves obtaining a question-and-answer dataset, includes multiple datasets consisting of questions, answers, and subgraphs, including:

[0154] The question-and-answer dataset mentioned is the Baidu Knows knowledge-based question-and-answer dataset: the Baidu Knows dataset contains 1,470,142 pre-filtered, high-quality questions and answers, with each question belonging to a category. There are a total of 492 categories, of which 434 categories have a frequency of 10 or more. Here, we have modified the format to include questions, answers, and subgraphs.

[0155] {"qid": <qid>,"category": <category>,"title": <title> "desc":< / title> <desc>"answer": }

[0156] {"Question":"Whose fault is more likely in this traffic accident? A car and a motorcycle collided at an intersection without traffic lights.","Sub-image":"<Car, accident, motorcycle>,<Car, speed, 10 km / h>,<Motorcycle, position, left of the car>,<Car, position, driving on the right>","Answer":"When crossing an uncontrolled intersection, you should slow down and yield to vehicles on the right. Based on what you've said, the motorcycle seems to be to the left of the car, so strictly speaking, the motorcycle might be fully responsible. Of course, it also depends on whether the car has all the necessary documents and whether the driver was under the influence of alcohol, etc. The specific determination will be made by the traffic police after investigation."}

[0157] In one embodiment, step S1, which involves obtaining a question-and-answer dataset, includes multiple datasets consisting of questions, answers, and subgraphs, including:

[0158] The question-and-answer dataset is the WebText2019Zh knowledge dataset, containing 4.1 million pre-filtered, high-quality questions and replies. Each question belongs to a topic, totaling 28,000 diverse topics. From 14 million original questions and answers, answers with at least 3 likes were selected, indicating that the replies were of good quality or interesting, thus obtaining a high-quality dataset. In addition to each question corresponding to a topic, a description of the question, and one or more replies, each reply also includes the number of likes, the reply ID, and the replyer's subgraph tag.

[0159] In one embodiment, this application uses game theory to improve the quality of style question annotation. Two independent annotation models, 1 and 2, of different types are set up, along with a quality control model for quality inspection. Assume there are T cycles. In each cycle, given equal time, the annotation models annotate multiple sets of questions (non-overlapping), and then submit a batch of annotated data to the quality control model. and Next, the quality inspection model evaluates the quality of each annotation by scoring a set of questions s∈[0,1]. Questions with s<σ will be used from... or The rejection was deleted and the annotation model was returned for re-annotation in the next cycle, along with the rejection reasons provided by the quality inspection model. and The scores are averaged to produce the total score for each labeled model, i.e., To incentivize the two annotation models to provide higher quality annotation data, the unit price for annotation data in each period is set to... The profit is determined by the annotation quality of both annotation models. Assume the annotation cost for each sample is c∈(0,1). Then the profit of annotation model 1 is... The profit of labeled model 2 is Where || represents the number of samples. The two annotation models know the annotation quality of each model after each period. and quantity

[0160] For example, the annotation model can be Siamese-uniNLU or paddleNLP, and the quality inspection model can be GPT-4.

[0161] In some embodiments, quality standards are applied to the problem quality, including large model annotation, manual annotation, and human-computer collaborative annotation. Large model annotation includes, but is not limited to, large models such as Cluade3 / Cluade2 / Cluade / GPT4 / GPT3.5 / GPT3 / GPT2 / GPT / ERNIE / ERNIE2 / ERNI E3.

[0162] The manual annotation method is crowdsourced, inviting student volunteers, teacher volunteers, and hiring domain experts to perform annotations.

[0163] Human-machine collaboration first uses machine annotation, using the following models: Cluade3 / Cluade2 / Cluade / GPT4 / GPT3.5 / GPT3 / GPT2 / GPT / ERNIE / ERNIE2 / ERNIE3, and then uses human proofreading, such as crowdsourcing, inviting volunteers, or hiring domain experts.

[0164] Based on the fact that annotation quality is often negatively correlated with quantity, we assume... Where r is the influence coefficient of labeled data on quality (preferably, r1 = r2 = r). Through continuous independent decision-making and game-playing between the two parties, the quantity and quality of labeled data will eventually stabilize at a certain value and reach an equilibrium point E, where each data point... Cost and total amount of labeled data

[0165] In one embodiment, in step S33, the loss functions for the question style reward model and the question quality reward model are defined as follows, based on the rewards:

[0166]

[0167] Where (K,2) represents the comparison of two questions, D is a labeled dataset of multiple sets of sequences, and for each set of sequence data, i.e., K questions, a reward model is used to generate the reward value for each question, r. θ (x, y) represents the scalar output of the generated question y calculated by the reward model, corresponding to the model parameters θ, where x is the answer and subgraph information of the corresponding question y, and y is... b This represents all generated problems sorted before y. σ is a sigmoid function that converts real numbers to the range of 0-1. The goal is to maximize the score difference between good and bad problems by taking the negative of the loss.

[0168] For example, suppose we have sequence labeling information D>A>C>B, and we need to train a scoring model. Based on the four sentences output by the question generation model, the scores should satisfy r(D)>r(A)>r(C)>r(B), and the loss function (Loss) is equal to the sum of the rewards for "good questions" and "bad questions". Here, we want the reward model to maximize the difference between the scores for "good questions" and "bad questions," because this means higher scores for "good questions" and lower scores for "bad questions," resulting in stronger predictive ability. Gradient descent performs a minimization operation. Therefore, we take the negative of the loss function, transforming the minimization operation into a maximization operation to achieve the desired effect.

[0169] Question style reward model and question quality reward model were trained to score questions.

[0170] In one embodiment, to avoid overfitting of the reward model, a bias correction score is calculated to mitigate the bias from the baseline problem Q. b Sample Problem Q s And the target question Q t The confirmation bias is viewed from the perspectives of word-level similarity and sentence-level similarity. Among these, the baseline problem Q... b It is the model output obtained by selecting the best action using the current policy. It serves as a reference standard during training and is used to evaluate the sample problem Q. s The quality of the sample. Sample problem Q s It is the model output obtained by randomly sampling from the current policy.

[0171] Word-level rewards are measured as:

[0172]

[0173] Here, WMD refers to Word Mover's Distance. The goal is to positively reward Q. s If it is better than the current best strategy Q b Closer to the target question Q t Regarding sentence-level rewards, this application uses two metrics, ROUGE-L and BLEU-4, to measure similarity.

[0174] Taking the ROUGE-L metric as an example, the reward is:

[0175]

[0176] The goal is to positively reward Q. s If it relates to the target problem Q t Compared to the current best strategy Q b More similar. BLEU-4's reward record is... Please note that for all reward scores, this application limits them to a value range [―γ, γ]. If the score exceeds this range, the hyperparameter (―)γ will be used instead. Finally, the bias-corrected reward r total (deviation correction score) is defined as:

[0177]

[0178] The rewards after bias correction can guide the reinforcement model to accurately generate diversity problems, correct bias errors, and further enhance the effectiveness of the model.

[0179] In particular, while deterministic policy learning reduces the likelihood of choosing very poor actions, it can also lead to agent complacency. To address this issue, this application designs an entropy regularization term. It refers to the entropy regularization of the action distribution corresponding to policy π in state s. The aim is to encourage a more even distribution of strategies.

[0180] Given a discrete probability distribution in the action space |A|, the entropy regularization term is defined as:

[0181]

[0182] Here, λ represents the hyperparameter that determines the weight of the regularization term. This entropy regularization term... The model is optimized by incorporating it into the objective function of reinforcement learning.

[0183] After multiple rounds of training, high-quality questions that meet the requirements of diversity are generated.

[0184] In one embodiment, reference Figure 3 In step S3, the predicted reward value is used for agent training, and the policy and QG-SFT model are iteratively optimized using a near-end policy optimization algorithm based on quadratic exponential decay rollback.

[0185] In one embodiment, considering the limitations of reinforcement learning due to its weak stability, current rollback mechanisms can effectively alleviate the gradient vanishing problem. However, this rollback mechanism also introduces a series of new problems. For example, the surrogate function in the existing PPO-RB algorithm is not differentiable at its maximum point, causing the policy gradient to oscillate near the maximum point. This application uses a near-end policy optimization algorithm based on a quadratic exponential decay rollback mechanism. By using a quadratic exponential decay rollback function, the zero and maximum points of the rollback function coincide, thereby solving the problem of the surrogate function's non-differentiable maximum point. This allows reinforcement learning to better adapt to diverse learning environments and improves learning efficiency.

[0186] To push the policy gradient update to the maximum point of the surrogate function, this application introduces a quadratic exponential decay rollback function, as shown below:

[0187]

[0188] Where, the hyperparameter α = ε ―2 r t Given the ratio, based on the aforementioned quadratic decay rollback function, the surrogate function for the near-end policy optimization algorithm is:

[0189]

[0190] There exist real numbers δ1, δ2 ∈ (1+ε, +∞). When r ∈ (1+ε, δ1), the policy gradient of the near-end policy optimization algorithm based on quadratic exponential decay rollback approaches zero more stably than the existing PPO-RB algorithm, which means that the policy is close to the maximum point of the replacement function. When r ∈ (δ1, δ2), the policy is far from the maximum point, and the near-end policy optimization algorithm will drive the policy gradient to take large steps to approach 0. When r > δ2, the near-end policy optimization algorithm will guide the policy gradient to take small steps to avoid the new policy from going out of the trust region.

[0191] In one embodiment, step S4 uses a reinforcement learning model for training. The reinforcement learning model performs bias correction on the scored questions based on prior knowledge feedback. The reward function of the bias correction regularizes the entropy of the action distribution corresponding to the policy in the state. The reinforcement learning model includes, but is not limited to, policy gradient, A2C / A3C, PPO algorithm, TRPO algorithm, PPO-Clip algorithm, PPO-QED algorithm, PPO-S algorithm, DDPG algorithm, TD3 algorithm, SAC algorithm, DQN algorithm, C51 algorithm, QR-DQN algorithm, HER algorithm, and Q-Learning algorithm.

[0192] Policy gradient optimization directly optimizes policy parameters to maximize long-term cumulative reward. Gradient ascent is typically used to update policy parameters. A2C / A3C algorithms, based on policy gradient, combine value function estimation with advantage function-guided policy updates; A3C is a parallelized version of A2C. The PPO algorithm improves stability and convergence by limiting the step size of policy updates, ensuring each update is not too large. The TRPO algorithm is similar to PPO but uses stricter policy update magnitude constraints to ensure each update is within a reliable region. The PPO-Clip algorithm is a variant of PPO that limits the magnitude of policy updates by comparing policy ratios. The PPO-QED algorithm is an improved version of PPO, introducing a new estimation method for policy updates. The PPO-S algorithm is another improved version of PPO, targeting policy optimization in continuous action spaces. The DDPG algorithm is a policy optimization algorithm for continuous action spaces, combining an actor-critic architecture and deep learning methods. The TD3 algorithm is an improved version of DDPG, improving stability by using two Q-networks and a delayed update target network. SAC (Solution-Action Computation) is another policy optimization algorithm for continuous action spaces, introducing entropy regularization to promote exploration. DQN (Depth-Qinness Classification) is a deep learning-based Q-learning method used to solve reinforcement learning problems in discrete action spaces. C51 is a variant of DQN that uses distribution estimation to represent the action-value function. QR-DQN is another improved DQN algorithm that uses quantile regression to estimate the action-value function. HER (Hyper-Reward) is a technique for solving sparse reward problems, learning failure trajectories by replaying memories. Q-Learning is a classic reinforcement learning algorithm that learns the optimal policy by iteratively updating the action-value function.

[0193] In one embodiment, the model-based reinforcement learning algorithm in step S4 includes, but is not limited to, the WorldModel algorithm, I2A algorithm, MBMF algorithm, MBVF algorithm, and AlphaZero algorithm. The WorldModel algorithm divides the environment into three parts: a perception model, a dynamic model, and a control model. It transforms raw environmental information into a processable state representation through the perception model, then uses the dynamic model to predict state transitions, and finally generates a policy through the control model. The I2A algorithm embeds the environment model into the agent's decision network, allowing the agent to simulate the results of multiple operations in its imagination and make decisions based on these simulations. The MBMF algorithm uses a meta-learning framework to quickly learn the environment model for each task and optimize the policy using the environment model, thus achieving rapid adaptation to new tasks. The MBVF algorithm expands the estimation of the action value function based on the environment model, using the model-simulated future trajectory to expand the estimation range of the value function, thereby improving the accuracy of decision-making. The AlphaZero algorithm uses the Monte Carlo tree search algorithm and deep neural networks to learn the policy and value function, continuously improving the policy through self-play, ultimately achieving game performance surpassing human levels.

[0194] In one embodiment, the reward function for bias correction in step S4 regularizes the entropy of the action distribution corresponding to the policy in the state. The reward function for bias correction includes, but is not limited to, the following:

[0195] The word-level reward function calculates the difference using the Word Mover's Distance (WMD) model. WMD is a method for calculating the semantic distance between two texts, which can be used to evaluate the semantic similarity between the policy-generated text and the target text.

[0196] The sentence-level reward function calculates the difference using the ROUGE-L model. ROUGE-L is a metric commonly used to evaluate the similarity between automatically generated text and reference text, and is particularly suitable for evaluating the similarity between sentences generated by a strategy and the target sentence.

[0197] The character-level reward function uses character-level similarity metrics, such as edit distance (Levenshtein distance), or a character-level language model to calculate the difference.

[0198] The paragraph-level reward function uses a semantic similarity metric between paragraphs to calculate the difference, such as a similarity metric based on paragraph vector representations.

[0199] The reward function at the overall effectiveness level comprehensively considers the differences in text generated by the strategy at different levels, and comprehensively evaluates the overall effect of the strategy generation.

[0200] In one embodiment, the entropy regularization in step S4 includes, but is not limited to, the following methods:

[0201] Entropy Regularization: The simplest entropy regularization method is to add an entropy term to the objective function of policy optimization. This entropy term is defined as the entropy of the action distribution corresponding to a certain state of the policy, i.e., the entropy of the negative action distribution. By maximizing the entropy of the policy, the policy can be made more random, increasing its exploratory nature.

[0202] KL-Divergence Regularization: KL-divergence regularization is another common entropy regularization method. It achieves this by controlling the KL divergence between the distributions before and after the policy update. Specifically, a KL divergence term is added to the objective function, using the KL divergence between the action distributions before and after the policy update as a regularization term. This prevents the policy update from causing excessive changes in the action distribution, thus maintaining a certain degree of exploratory nature.

[0203] Temperature Scaling: Temperature scaling is a method of entropy regularization that adjusts the temperature parameter of action probabilities in a policy. By increasing the temperature parameter, the variance of action probabilities can be increased, thereby increasing the exploratory nature of the policy. The higher the temperature parameter, the stronger the exploratory nature of the policy.

[0204] Action Dropout: Action dropout is a highly stochastic entropy regularization method that randomly discards certain actions with a certain probability each time a policy selects an action, thereby increasing the diversity and exploratory nature of the policy. Action dropout can be viewed as a random sampling of the action space.

[0205] The methods used to calculate reward scores in the embodiments can all be used for bias correction. Furthermore, bias correction can also employ various fold-one reinforcement models (such as policy gradient, A2C / A3C, PPO algorithm, TRPC algorithm, PPO-Clip algorithm, etc.) to utilize prior knowledge from feedback (such as feedback reward score question information, answer information, or subgraph information, or executed actions, etc.) to perform bias correction on the scored questions. Bias correction functions can include one or more of the following: word level, sentence level, character level, paragraph level, and overall effectiveness level. Bias correction functions can also be BLEU, ROUGE, METEOR, CIDEr, BERT scores, etc.

[0206] In one embodiment, the step of outputting the final question after multiple rounds of training includes:

[0207] A single data entry is split into multiple data entries, each including sections for User and Assistant.

[0208] After concatenating a data set, it is input into the model, and the loss of each part is calculated in parallel. Only the loss of the Assistant part is involved in the weight update.

[0209] In one embodiment, the step of outputting the final question after multiple rounds of training includes:

[0210] A learning rate decay strategy is introduced to gradually reduce the learning rate in order to achieve more stable convergence in the later stages of training.

[0211] In each training round, the negative sampling strategy is dynamically adjusted based on the model's performance, and negative samples are selected more selectively to enhance the model's ability to learn and remember key information.

[0212] Data augmentation techniques, such as random perturbation and partial occlusion, are introduced to increase data diversity and improve the model's generalization ability.

[0213] Implement an early stopping strategy: terminate training promptly when the model's performance on the validation set no longer improves to avoid overfitting.

[0214] By using methods such as grid search and Bayesian optimization, the hyperparameters of the model are tuned to find the optimal parameter combination and further improve the model performance.

[0215] In one embodiment, the step of outputting the final question after multiple rounds of training includes:

[0216] For imbalanced datasets, methods such as oversampling or undersampling are used to balance the distribution of samples of each class, thereby improving the model's learning performance on minority class samples.

[0217] By combining the model update process with real-time data streams, continuous online and incremental learning is achieved to adapt to changes in data distribution and the accumulation of new knowledge.

[0218] By integrating multiple models of different types or structures, such as through voting or stacking, the generalization ability and performance of the model can be further improved.

[0219] This application trains a preference-based question reward model by labeling an initial question set and using a pre-set reward loss function. This enables the trained model to generate relatively fluent questions and scores them, improving scoring accuracy. A reinforcement learning module is then trained using a pre-set reinforcement loss function. This trained module is used to construct a question generation environment. Within this environment, multiple candidate questions are generated using the initial question generation model and standard questions, answers, and subgraphs. The reinforcement learning module is trained again using the pre-set reinforcement loss function, resulting in a more even distribution of policies when constructing the question generation environment, thus improving the accuracy of candidate question generation. The initial question generation model generates candidate questions by observing the "standard questions," answers, and subgraphs in the environment. These candidate questions are generated in conjunction with specific real-world scenarios, demonstrating high applicability. The trained preference-based question reward model then scores these candidate questions, and bias correction is used to adjust the scoring results. However, the trained question reward model, by learning the reward function, may malfunction and perform poorly when the feedback is inaccurate. This application employs a question quality reward model, mitigating bias by emphasizing question quality, such as punctuation, vocabulary, grammar, and whether the question matches the answer. Furthermore, to alleviate annotation bias during the annotation process, this application uses a game theory mechanism to improve annotation quality by coordinating two competing question style annotation models. To mitigate potential confirmation bias, this application proposes a reinforcement learning method for bias correction, learning from feedback to correct confirmation bias. The scoring results are corrected through bias correction to ensure quality. The initial question generation model is trained using the bias-corrected reward and a proximal policy optimization algorithm to iteratively optimize the strategy, resulting in a well-trained question model. Based on this trained question generation model, diverse questions are generated, increasing the diversity and quality of question generation.

[0220] refer to Figure 4 In one embodiment, the present invention also discloses a knowledge graph question generation apparatus diagram, comprising:

[0221] The acquisition module 11 is used to acquire a question-and-answer dataset, which includes multiple data consisting of questions, answers, and subgraphs;

[0222] The initial question generation module 12 generates an initial question based on supervised fine-tuning training and using the context information of the subgraph to guide the answer.

[0223] Supervised fine-tuning of cold-start question generation utilizes state-of-the-art (SOTA) models in the current knowledge graph question generation field. These SOTA models can be G2S, G2S-AE, IGND, or G2S-AE-RL models, which leverage subgraph guidance to utilize contextual information about the answers, generating accurate and fluent natural language questions. For example, training data is provided to the G2S-AE-RL model for fine-tuning, generating initial questions for subsequent training. It is worth noting that this application does not limit the choice of baseline model; many models can be selected, including but not limited to: L2A, RNN, Transformer, zero-shot, MHQG, T5, IGND, G2S, G2S-AE, and G2S-AE-RL. For example, the G2S-AE-RL model employs a novel BiGraph2Seq model to generate questions and answers from KB subgraphs. Specifically, this application provides training data to fine-tune the G2S-AE-RL model, which generates initial questions. For the L2A model, an early KG subgraph is used as input to an LSTM-based Seq2Seq model, belonging to an attention-based Seq2Seq model that can generate natural language questions in open-domain dialogue systems using contextual information. Specifically, this application provides training data to fine-tune L2A, which will generate initial questions. For the G2S-AE-RL model, it is a knowledge base question generation model utilizing a two-level RL (reinforcement learning) training strategy, where the objective function combines cross-entropy loss and RL loss. Specifically, this application provides training data to fine-tune G2S-AE-RL, which will generate initial questions.

[0224] The question reward module 13 outputs questions to be scored based on the answers and subgraphs of the initial questions. After prioritizing the questions to be scored, it uses question style labeling and question quality labeling to calculate the reward score for the questions to be scored based on the priority of the labels.

[0225] The issue reward module includes the following three sub-modules:

[0226] The question generation submodule generates four questions by feeding the input answer and subgraph into the QG-SFT model trained in the first step. Below is an example of the input format for the answer and subgraph.

[0227] {"answers":["Computer Vision","Face Recognition"],"answer_ids":["10001082","10001100"],"outSeq":"By the way, what knowledge using probabilistic models have you used?","qId":18,"inGraph":{"g_node_names":{"10001082":"Computer Vision","10001100":"Face Recognition","10001092":"Probabilistic Model"},"g_edge_types":{"Computer / Domain / Topic Learning Dependencies":"Computer / Domain / Topic / Learning Dependencies"},"g_adj":"10001082":{"10001092":"Computer / Domain / Topic / Learning Dependencies"},"10001100":{"10001092":"Computer / Domain / Topic / Learning Dependencies"}}}}

[0228] Sequence priority labeling submodule: using a question-style reward model Problem quality reward model Based on the semantics, syntax, relevance, and questioning style of the questions in the prior knowledge, the generated questions are labeled from good to bad.

[0229] Relative ranking is used to rank the generated questions because it reduces the difficulty of labeling.

[0230] It is worth noting that this application uses two different annotation models in the style question reward model and uses a game theory mechanism to coordinate the two annotation models and stabilize the annotation quality.

[0231] The reward training submodule uses annotation priority based on human-like question style annotation and question quality annotation to train the corresponding reward models. Human quality and Quality, where θ is a parameter. After training, the goal is for these two models to automatically score the generated questions. Specifically, using the answer and subgraph information x as hints, the ERNIE model encodes the previously generated question y into a vector representation, which is then processed by pooling layers and linear transformation layers to produce a one-dimensional reward value r. human and r quality .

[0232] The reinforcement training module 14 uses a reinforcement learning model for training. The reinforcement learning model corrects the bias of the scored question based on prior knowledge feedback. The reward function of the bias correction regularizes the entropy of the action distribution corresponding to policy π in state s. After multiple rounds of training, the final question is output.

[0233] Figure 5 A structural block diagram of a knowledge graph question generation apparatus according to an embodiment of the present invention is shown. Figure 5 As shown, the device includes a memory 910 and a processor 920. The memory 910 stores a computer program that can run on the processor 920. When the processor 920 executes the computer program, it implements the question-and-answer pair generation method in the above embodiments. The number of the memory 910 and the processor 920 can be one or more.

[0234] The device also includes:

[0235] The communication interface 110 is used to communicate with external devices and perform data exchange and transmission.

[0236] The memory 120 may include high-speed RAM memory, and may also include nonvolatile memory, such as at least one disk storage.

[0237] If the memory 120, processor 130, and communication interface 110 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0238] As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. Specifically, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0239] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if it is not structurally equivalent to the disclosed structure that performs the function in the exemplary implementation of this disclosure shown herein. Moreover, although a particular feature of this disclosure has been disclosed with respect to only one of several implementations, such feature may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Moreover, with respect to the use of the terms "comprising," "having," "containing," or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term "comprising."

[0240] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0241] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.< / desc> < / category> < / qid>

Claims

1. A problem generation method based on reinforcement learning, characterized in that, Includes the following steps: Obtain a question-and-answer dataset, which includes multiple datasets consisting of questions, answers, and subgraphs; Based on supervised fine-tuning training, the initial question is generated using the contextual information of the subgraph to guide the answer. Based on the answers to the initial questions and the subgraph, output the questions to be scored; annotate and score the question style of the questions to be scored; annotate the question quality of the questions to be scored; calculate the bonus score for the annotated questions to be scored. A problem generation model is trained using a reinforcement learning model, which corrects for bias in the scored problems based on prior knowledge feedback. After multiple rounds of training, a trained problem generation model is obtained. The actual problem is input into the problem generation model, and the final problem is output. Specifically, question style annotation and question quality annotation are used to automatically assign reward scores to the generated questions. That is, the initial questions generated are encoded into vector representations using the answers and subgraph information as prompts, and then processed by pooling layers and linear transformation layers to generate one-dimensional style reward scores and one-dimensional quality reward scores, respectively. The reinforcement learning model is based on data provided by the environment, namely labeled question-and-answer datasets and knowledge bases. The agent generates questions and stores these questions in an experience pool. The agent is a cold-start question generation model. In order to calculate rewards and optimize the agent, questions are sampled from the experience pool and forwarded to the reward predictor to generate bias correction rewards. The bias correction rewards include question style rewards, question quality rewards, and bias correction. Based on the reward results, the agent is optimized using a proximal policy optimization algorithm.

2. The problem generation method based on reinforcement learning according to claim 1, characterized in that, The annotation of the question style for the questions to be rated shall include at least one of the following methods: large model annotation, manual annotation, and human-computer collaboration.

3. The problem generation method based on reinforcement learning according to claim 1, characterized in that, Game theory mechanisms are also used to improve the quality of the question style annotation, including: Two independent annotation models, 1 and 2, of different types are set up for question style annotation, and a quality control model is set up for quality inspection. In each cycle, each annotation model annotates multiple sets of questions, and then submits a batch of annotated data to the quality control model. and ; The quality inspection model evaluates the quality of annotations by scoring each problem in the annotation model; for problems with scores below a preset value, [the quality is adjusted accordingly]. or Delete it and return it to the annotation model so that it can be re-annotated in the next cycle, with the rejection reasons provided by the quality inspection model attached; Will and The average score of the two annotation models is used to obtain the total score of annotation model 1 and annotation model 2. In order to incentivize the two annotation models to provide higher quality annotation data, the unit price of annotation data in each cycle is jointly determined by the annotation quality of the two annotation models. The profit of annotation model 1 and annotation model 2 is calculated based on the annotation cost of each sample. After continuous independent decision-making and game-playing between the two annotation models, the quantity and quality of annotations eventually stabilized at a certain value and reached an equilibrium point. .

4. The problem generation method based on reinforcement learning according to claim 1, characterized in that, The issue quality labeling is used to improve the quality of the issue itself, and the issue quality labeling will include at least one of the following sub-modules in the issue quality assessment: The text accuracy submodule and the symbol accuracy submodule are used to evaluate the quality of characters; The vocabulary diversity submodule and the vocabulary accuracy submodule are used to evaluate the quality of vocabulary; The syntactic complexity submodule and the syntactic correctness submodule are used to evaluate the quality of the syntax; The clarity and completeness submodules are used to evaluate the quality of the content; An evaluation submodule assesses the logical consistency and smoothness between questions and answers, used to evaluate the quality of the interaction; The question-and-answer matching submodule and the practical value submodule are used to evaluate the overall effectiveness; For each submodule, binary labeling is performed: if the answer to the question is yes, it is 1; otherwise, it is 0. Then, the labeling results of all submodules are summed to obtain the total question quality score. These questions are then sorted in descending order according to their total question quality scores to generate a priority sequence.

5. The problem generation method based on reinforcement learning according to claim 1, characterized in that, The prior knowledge includes at least the known environment, the question generated by the agent, and one of the actions taken. The environment and agent in the reinforcement learning model are combined with a reward function for bias correction to obtain feedback for training the model. The environment includes the question, the answer, and the subgraph. The agent is a cold-start question generation model. Based on the data provided by the environment, the agent initializes itself. Then, the agent observes the standard questions, answers, and subgraphs in the environment and generates an initial question, i.e., an action, which is stored in a replay buffer. In order to calculate the reward and optimize the agent, the questions are sampled from the replay buffer and passed to a reward predictor, which calculates the reward score and the bias correction score.

6. The problem generation method based on reinforcement learning according to claim 1, characterized in that, The reward function for deviation correction and the strategy In state The entropy regularization of the corresponding action distribution is used as the loss function to avoid overfitting, and the reward function for bias correction includes at least one of the following: The semantic distance between two texts is calculated using the Word Mover's Distance model at the word level, which is used to evaluate the semantic similarity between the text generated by the strategy and the target text; At the sentence level, the ROUGE-L model is used to calculate the similarity between the automatically generated text and the reference text, which is used to evaluate the similarity between the sentences generated by the strategy and the target sentences. The difference is calculated using a character-level similarity metric, wherein the character-level similarity metric includes at least one of the Levenshtein distance or a character-level language model. The difference can be calculated using semantic similarity measures between paragraphs at the paragraph level, such as similarity measures based on paragraph vector representations; The overall effectiveness of the strategy is comprehensively evaluated based on the differences in the text generated by the strategy at different levels.

7. The problem generation method based on reinforcement learning according to claim 1, characterized in that, The deviation correction includes: Using the aforementioned questioning style reward and question quality reward, scores are generated based on prior knowledge, logic, and question quality. Calculate deviation correction scores to mitigate baseline problems. Sample problem and target problem Confirmation bias arising from word-level similarity and sentence-level similarity; among which, baseline issues... It is the model output obtained by selecting the best action using the current policy. It serves as a reference standard during training and is used to evaluate sample problems. Quality, sample issues It is the model output obtained by randomly sampling from the current policy.

8. The problem generation method based on reinforcement learning according to claim 7, characterized in that, The goal of the word-level reward in the questioning style reward model is to positively reward sample questions. ,if Baseline problem Getting closer to the target problem ; Sentence-level rewards use ROUGE-L to measure similarity. The reward objective of ROUGE-L is the positive reward sample problem. ,if With the target problem Compared to the baseline problem More similar; Rewards after deviation correction It includes rewards for question style, question quality, word-level rewards, sentence-level rewards, and baseline questions measured according to the BLEU-4 metric. and sample problem With the target problem The sum of the rewards.

9. The problem generation method based on reinforcement learning according to claim 8, characterized in that, Based on the aforementioned rewards The agent is trained iteratively, and during the training process, an entropy regularization term is used to make the policy distribution more uniform; for the action space... Discrete probability distribution on.

10. A problem generation device based on reinforcement learning, characterized in that, include: The acquisition module is used to acquire a question-and-answer dataset, which includes multiple data consisting of questions, answers, and subgraphs; The initial question generation module, based on supervised fine-tuning training, uses the subgraph to guide the contextual information of the answer to generate the initial question; The question reward module outputs questions to be scored based on the answers and subgraphs of the initial questions; it also marks and scores the question style of the questions to be scored; it marks the question quality of the questions to be scored; and it calculates reward scores for the marked questions to be scored. The reinforcement training module uses a reinforcement learning model to train a problem generation model, and the reinforcement learning model corrects the bias of the scored problems based on feedback from prior knowledge. The output module, after multiple rounds of training, produces a trained problem generation model. The actual problem is input into the problem generation model, and the final problem is output. Specifically, question style annotation and question quality annotation are used to automatically assign reward scores to the generated questions. That is, the initial questions generated are encoded into vector representations using the answers and subgraph information as prompts, and then processed by pooling layers and linear transformation layers to generate one-dimensional style reward scores and one-dimensional quality reward scores, respectively. The reinforcement learning model is based on data provided by the environment, namely labeled question-and-answer datasets and knowledge bases. The agent generates questions and stores these questions in an experience pool. The agent is a cold-start question generation model. In order to calculate rewards and optimize the agent, questions are sampled from the experience pool and forwarded to the reward predictor to generate bias correction rewards. The bias correction rewards include question style rewards, question quality rewards, and bias correction. Based on the reward results, the agent is optimized using a proximal policy optimization algorithm.

11. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes a computer program stored in a computer-readable storage medium, and a processor of an electronic device reads from and executes the computer program, causing the electronic device to perform the steps of the method according to any one of claims 1 to 9.

Images