A method and device for training a question generation model

Abstract

The one or more embodiments of the specification disclose a training method of a question generation model. The method first acquires a historical complex question and a corresponding answer, and a historical document type knowledge base corresponding to the historical complex question, and the historical document type knowledge base at least includes a historical document related to the historical complex question. Then, an entity word related to the historical complex question is extracted from the historical document, a simple question is constructed based on the entity word, and a plurality of historical knowledge points are determined based on the simple question and the historical complex question. Finally, the question generation model is trained based on the historical document and the plurality of historical knowledge points through a preset loss function to obtain a trained question generation model, wherein the question generation model is used to generate a historical complex question corresponding to the historical document.

Description

Technical Field

[0001] This document relates to the field of intelligent question answering technology, and in particular to a training method and apparatus for a question generation model. Background Technology

[0002] Intelligent question answering is an important form of human-computer interaction. An intelligent question answering system is a new type of information retrieval system that processes natural language. It adopts a question-and-answer format, receives questions from users, accurately locates relevant knowledge in the question-and-answer database, and performs business navigation according to the set business process flow to realize the interaction between business guidance and users. It then feeds back the relevant knowledge as the answer to the question to the user, thus completing the intelligent question answering process.

[0003] Current intelligent question-answering systems typically only allow for simple questions, such as "What year were you born?" These questions don't require associative reasoning or critical thinking. Because both the question and answer are relatively simple, this approach fails to meet user needs in some areas. For example, in lie detection, users might answer simple questions quickly without any apparent reaction, making it impossible to observe their responses and resulting in inaccurate lie detection. Therefore, there is an urgent need for a method that can generate complex questions that require some associative reasoning and critical thinking. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a technical solution that can intelligently generate complex problems to meet user needs.

[0005] To achieve the above technical solution, the embodiments in this specification are implemented as follows:

[0006] On one hand, one or more embodiments of this specification provide a training method for a question generation model, comprising: obtaining historical complex questions and their corresponding answers, and a historical document-type knowledge base corresponding to the historical complex questions, wherein the historical document-type knowledge base includes at least one historical document related to the historical complex questions; extracting entity words related to the historical complex questions from the historical documents, constructing simple questions based on the entity words, and determining multiple historical knowledge points based on the simple questions and the historical complex questions; training the question generation model based on the historical documents and the multiple historical knowledge points using a preset loss function to obtain a trained question generation model, wherein the question generation model is used to generate historical complex questions corresponding to the historical documents.

[0007] On the other hand, one or more embodiments of this specification provide a training device for a question generation model, comprising: an input module for acquiring historical complex questions and their corresponding answers, and a historical document-type knowledge base corresponding to the historical complex questions, wherein the historical document-type knowledge base includes at least one historical document related to the historical complex questions; a historical knowledge point determination module for extracting entity words related to the historical complex questions from the historical documents, constructing simple questions based on the entity words, and determining multiple historical knowledge points based on the simple questions and the historical complex questions; and a model training module for training the question generation model based on the historical documents and the multiple historical knowledge points using a preset loss function to obtain a trained question generation model, wherein the question generation model is used to generate historical complex questions corresponding to the historical documents.

[0008] Furthermore, one or more embodiments of this specification provide an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, wherein when the executable instructions are executed, the processor is able to: acquire a historical complex question and its corresponding answer, and a historical document-type knowledge base corresponding to the historical complex question, the historical document-type knowledge base including at least one historical document related to the historical complex question; extract entity words related to the historical complex question from the historical document, construct a simple question based on the entity words, and determine multiple historical knowledge points based on the simple question and the historical complex question; and train a question generation model based on the historical document and the multiple historical knowledge points using a preset loss function to obtain a trained question generation model, the question generation model being used to generate the historical complex question corresponding to the historical document.

[0009] On another front, one or more embodiments of this specification provide a storage medium for storing computer-executable instructions. When executed by a processor, these instructions implement the following process: obtaining a historical complex question and its corresponding answer, and a historical document-type knowledge base corresponding to the historical complex question, wherein the historical document-type knowledge base includes at least one historical document related to the historical complex question; extracting entity words related to the historical complex question from the historical document, constructing a simple question based on the entity words, and determining multiple historical knowledge points based on the simple question and the historical complex question; training a question generation model using a preset loss function based on the historical document and the multiple historical knowledge points to obtain a trained question generation model, wherein the question generation model is used to generate the historical complex question corresponding to the historical document. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic flowchart illustrating a training method for a problem generation model as described in this manual;

[0012] Figure 2 This is a schematic diagram illustrating the principle of a training method for a problem generation model as described in this manual;

[0013] Figure 3 This is a schematic flowchart illustrating another training method for the problem generation model in this manual;

[0014] Figure 4 This is a schematic flowchart illustrating another training method for a problem generation model in this manual;

[0015] Figure 5 This is a schematic block diagram of a training device for a problem generation model as described in this specification;

[0016] Figure 6 This is a schematic block diagram of an electronic device described in this specification. Detailed Implementation

[0017] This specification provides a method and apparatus for training a problem generation model through one or more embodiments.

[0018] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this document.

[0019] Currently, in educational exams and risk control exams, questions requiring logical reasoning and critical thinking are often designed to better assess the respondent's true abilities. In dialogue scenarios primarily driven by AI-powered questioners, complex questioning is a significant research area. Common methods for generating complex questions include manually creating question banks, where complex questions are designed and added manually. However, this method requires designers to extensively read relevant knowledge and then write complex questions. As the knowledge content updates, designers need to continuously invest time and effort to update the complex questions. Furthermore, complex questions generated in this way are easily exploited by users through repeated attempts (i.e., brute-force attacks). Additionally, there is the AUTOQA automated question-answering system, which can generate simple questions. This system is primarily trained using a large amount of internet data. However, for the complex questions required, the internet lacks sufficient corpus data, making it unsuitable for training with an automated question-answering system. Based on this, embodiments of this specification provide a training method and apparatus for a problem generation model. This method completes model training and reaches a usable level using a relatively small sample size consisting of complex problems and their corresponding answers, thereby significantly improving the efficiency and effectiveness of generating complex problems. A detailed description follows.

[0020] Example 1

[0021] like Figure 1 As shown in the embodiments of this specification, a training method for a problem generation model is provided. The execution subject of this method can be a terminal device or a server. The terminal device can be a mobile phone, tablet computer, or other similar device, or a laptop computer or desktop computer. The server can be a single server or a server cluster consisting of multiple servers, and can be a backend server for an application. The method may include the following steps:

[0022] In step S102, a historical complex problem and its corresponding answer are obtained, as well as a historical document-type knowledge base corresponding to the historical complex problem. The historical document-type knowledge base includes at least one historical document related to the historical complex problem.

[0023] In the embodiments of this specification, the historical complex questions and their corresponding answers can be data used for model learning. The historical complex questions and their corresponding answers in this embodiment can be a pair (i.e., historical complex question-answer) or multiple pairs. During model training, one or more pairs of historical complex questions and their corresponding answers can be obtained as samples from a specified database, or one or more pairs of historical complex questions and their corresponding answers can be obtained as samples from multiple different users (or data providers).

[0024] The historical complex questions in the embodiments of this specification, and the complex questions mentioned below, refer to questions that require the respondent to make certain connections and reasoning in order to complete the answer, while simple questions may not require the respondent to make connections and reasoning or may only require the respondent to make very simple connections and reasoning to answer directly.

[0025] The historical complex problems (i.e., historical complex problem text) or complex problems (i.e., complex problem text) in the embodiments of this specification can be problems containing multi-hop relationships, problems with constraints, or problems containing both multi-hop relationships and constraints. Complex problems containing multi-hop relationships, such as "Who are the directors of the movie starring A?", require a multi-hop reasoning path formed by multiple triples to answer. For example, using the triples "A, starring, new XX story" and "new XX story, director, B", we can infer that "B" is the answer to this complex problem. Complex problems with constraints, such as "Who won the first Wimbledon men's singles title?", use "first" to represent a constraint on the answer entity. In practical applications, A is replaced with the name of a specific film star, and B is replaced with the name of a specific director.

[0026] Simple questions often contain a single relationship. For example, "What year were you born?" can be a simple question, while "What important thing happened in the year you were born?" can be a complex question.

[0027] The historical document-type knowledge base in the embodiments of this specification corresponds to historically complex problems; that is, the answers to historically complex problems can be found in the historical document-type knowledge base. In implementation, one historically complex problem may correspond to one related historical document, or one historically complex problem may correspond to two or more related historical documents. This specification does not limit this.

[0028] In one possible implementation, the historical document knowledge base is an open-source, updatable knowledge base. Specifically, this knowledge base can be obtained from the internet, and updated versions of the historical document knowledge base can be obtained as internet information is updated.

[0029] In step S104, entity words related to historical complex issues are extracted from historical documents, simple issues are constructed based on the entity words, and multiple historical knowledge points are determined based on the simple issues and historical complex issues.

[0030] Historical documents corresponding to complex historical problems can include various entity words. Entity words related to the complex historical problem can be extracted from these documents, and simpler problems can be constructed based on each entity word. Alternatively, simpler problems can be constructed based on multiple entity words from the historical documents, thus forming one or more simpler problems related to the complex historical problem. The embodiments in this specification extract entity words related to the complex historical problem from historical documents and then construct corresponding simpler problems from them. This significantly reduces the number of simpler problems unrelated to the complex historical problem, which is beneficial for improving model training efficiency and saving resources.

[0031] In practice, simple questions can be manually set based on entity words and expert experience, or a smart question-and-answer system can be used to automatically generate simple questions corresponding to entity words. This specification does not limit this approach in the embodiments.

[0032] As shown in step S104, after constructing simple questions based on entity words, multiple historical knowledge points are then determined based on the simple questions and historical complex questions. In implementation, each simple question can be compared with a historical complex question to determine multiple historical knowledge points. Alternatively, based on preset keywords, each historical complex question can be compared with each simple question one by one to determine multiple historical knowledge points. Another approach is to compare the similarity between each simple question and a historical complex question, and select one or more entity words with high similarity as historical knowledge points.

[0033] In the embodiments of this specification, the number of historical knowledge points determined based on simple and historically complex questions is specifically set according to the complexity of the user's needs. In one possible implementation, the number of historical knowledge points is 2-3. Correspondingly, after the question generation model is trained, when using the question generation model to generate complex questions, the number of knowledge points can also be 2-3. This number of knowledge points ensures that the generated complex or historically complex questions can meet the user's questioning needs without being overly complex and consuming excessive resources.

[0034] Figure 2 This is a schematic diagram illustrating the principle of a training method for a problem generation model as described in this specification. Figure 2 Taking a historical document-based knowledge base containing two historical documents as an example, the specific historical documents, historical complex questions, and their corresponding answers are as follows:

[0035] Historical document 1: The Geneva Conference in 1954.

[0036] From April 26 to July 21, 1954, a diplomatic conference of the Soviet Union, the United States, the United Kingdom, France and China was held at the League of Nations building in Geneva.

[0037] Historical document 2:A.

[0038] Born in Hong Kong, China in April 1954, he is a Hong Kong film and television actor, director, producer, screenwriter, and singer. He entered the film industry as a stuntman in 1971.

[0039] The corresponding complex historical question: What important events happened in the year A was born?

[0040] The answer to this complex historical question: the Geneva Conference.

[0041] For the complex historical question "What important events happened in the year A was born?", the two corresponding knowledge points, "What year was A born?" and "What important events happened in 1954?", need to be found in the corresponding historical documents 1 and 2. To determine these two knowledge points, we first construct simple questions for the entities related to the complex historical question in the two historical documents. These entities include: 1954, Hong Kong, China, and A. The constructed simple questions can be as follows: for the entity "1954", the simple question is "What year was A born?"; for the entity "Hong Kong, China", the simple question is "Where was A born?"; and for the entity "A", the simple question is "Who was born in Hong Kong on April 7, 1954?". Finally, by comparing the above simple questions with the complex historical question, we determine that the entity "1954" is most likely the test point of the complex historical question and can be used as a historical knowledge point; the entity "A" has already appeared in the complex historical question and can be excluded as a historical knowledge point; the entity "Hong Kong, China" has low relevance to the complex historical question and can also be excluded as a historical knowledge point.

[0042] In step S106, based on historical documents and multiple historical knowledge points, the question generation model is trained using a preset loss function to obtain the trained question generation model. The question generation model is used to generate historical complex questions corresponding to historical documents.

[0043] In one possible implementation, after determining multiple historical knowledge points corresponding to each historical complex problem, a controllable generation prompt learning mechanism can be used. The historical documents corresponding to the historical complex problem and multiple historical knowledge points are used as input data, and the historical complex problem is used as the output result. Controllable generation training is performed based on a preset loss function until the preset loss function converges or the set number of iterations is reached, thus obtaining the problem generation model.

[0044] The controllable generation prompt learning mechanism employs a text-controlled generation method. This method refers to generating corresponding text from certain prompting and controllable elements in the input data through a model. In this embodiment, the prompting and controllable elements are: historical documents corresponding to historical complex problems and multiple historical knowledge points; the output text is: historical complex problems.

[0045] The question generation model generated using a text-controlled generation method can be a pre-trained language model. This question generation model can be any one of GPT (Generative Pre-Training), BART (Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehendment, used to pre-train seq-to-seq models), or T5 (TransferText-to-text Transformer, a unified text-to-text task). The embodiments in this specification do not limit this.

[0046] The preset loss function can be the cross-entropy loss function (CE-loss), the focal loss function, etc. The goal of the preset loss function is to make the historical complex problem generated by the problem generation model infinitely close to the historical complex problem labeled.

[0047] In one possible implementation, the preset loss function can be the cross-entropy loss function, which is easier to apply and helps improve the accuracy and efficiency of model training.

[0048] This specification provides a training method for a question generation model. It involves acquiring historical complex questions and their corresponding answers, as well as a historical document-based knowledge base. Entity words related to the historical complex questions are extracted from these historical documents. Simple questions are constructed based on these entity words, and multiple historical knowledge points are determined based on both the simple and historical complex questions. Finally, the question generation model is trained using a preset loss function based on the historical documents and these multiple historical knowledge points, resulting in a trained model. This method of automatically generating historical complex questions by extracting multiple historical knowledge points from historical documents, based on entity words related to the historical complex questions, and using the simple questions constructed from these entity words and the historical complex questions to ultimately determine the historical knowledge points, makes the generated historical complex questions more targeted and the assessment of knowledge points more precise. This improves the efficiency of model training and the accuracy of the model in generating complex questions. Furthermore, the trained question generation model can intelligently generate complex questions and can specify multiple knowledge points as test points, effectively saving manpower and improving question generation efficiency. Furthermore, since the generated complex questions are not simple questions that can be directly found on the internet, the risk of credential stuffing attacks is greatly reduced, and the generated complex questions are more stable. Because each complex question corresponds to at least two knowledge points, it also corresponds to at least two simple questions, essentially decomposing a complex question into two simple questions, making the model training method more interpretable. In addition, compared to the AUTOQA training method, the question generation model training method in the embodiments of this specification only requires 10% of the complex questions and their corresponding sample size for training. Typically, a model trained with around 100 complex questions and their corresponding answers is sufficient to reach a usable level, eliminating the need for a large number of samples, saving resources, and improving model training efficiency.

[0049] Example 2

[0050] like Figure 3 As shown in the embodiments of this specification, another method for training a problem generation model is provided. This method may include:

[0051] S202: Obtain historical complex problems and their corresponding answers, as well as a historical document-based knowledge base corresponding to the historical complex problems. The historical document-based knowledge base shall include at least one historical document related to the historical complex problems.

[0052] The method for obtaining samples for model training in this embodiment can be found in Embodiment 1 above, and will not be repeated here.

[0053] S204: Identify an entity in a historical document that is related to a historical complex issue, but which does not appear in the historical complex issue.

[0054] If an entity word has already appeared in a historical complex problem, it can be excluded from being considered a knowledge point (or test point). Therefore, this entity word can be removed from the historical documents containing entities related to historical complex problems. Removing this entity word in step S204 helps prevent the generation of corresponding simpler questions in subsequent step S206 and reduces the number of simpler questions compared to historical complex problems in step S208, thereby saving resources and improving model training efficiency.

[0055] S206: For each entity word, determine the corresponding simple question, where a simple question is a question containing a single relation.

[0056] One possible implementation uses AUTOQA to generate simple questions. Specifically, for each entity word, a simple question is generated based on the automatic question answering system.

[0057] S208: By comparing the similarity between each simple question and a historically complex question, the historical knowledge points corresponding to the historically complex question are determined from all entity words.

[0058] As can be seen from steps S204-S208 above, when extracting entity words, constructing simple questions based on entity words, and determining multiple historical knowledge points based on simple questions and historical complex questions, the embodiments in this specification use a similarity comparison method to extract historical knowledge points. This method of comparing the overlap between simple and complex questions uses unsupervised comparison to retrieve historical knowledge points, which is beneficial for more accurately comparing the similarity between each simple question and historical complex questions, thereby making the determined historical knowledge points more accurate, and thus improving the accuracy of the trained model in generating complex questions.

[0059] In one possible implementation, by comparing the similarity between each simple question and the historically complex question, the historical knowledge points corresponding to the historically complex question can be determined from all entity words. This can be achieved through the following steps A1-A2:

[0060] Step A1: The similarity between the simple problem and the historical complex problem is obtained by calculating the BLEU (Bilinggual Evaluation Understudy) value for each simple problem and the historical complex problem respectively.

[0061] Step A2: Based on the calculated similarity and according to the set similarity threshold, determine multiple historical knowledge points corresponding to the historical complex issues from the determined entity words.

[0062] In implementation, answers to simple questions with similarity scores higher than the set similarity threshold can be used as historical knowledge points. For example, when the BLEU value exceeds the preset BLEU threshold, the entity word corresponding to the simple question is determined to be a knowledge point corresponding to a historical complex question; otherwise, the entity word corresponding to the simple question is determined not to be a knowledge point corresponding to a historical complex question.

[0063] Alternatively, the similarity scores can be sorted from highest to lowest. The answers to the first n (n is a positive integer greater than or equal to 1) simple questions in the sorted sequence can be used as historical knowledge points.

[0064] In one possible implementation, by comparing the similarity between simple problems and historically complex problems, the historical knowledge points corresponding to the historically complex problems can be determined from the given entity words. This can be achieved by performing the following steps B1-B2:

[0065] Step B1: Calculate the similarity between each simple problem and the historical complex problem by calculating the maximum number of identical substring characters between each simple problem and the historical complex problem.

[0066] Step B2: Based on the calculated similarity and according to the set similarity threshold, determine multiple historical knowledge points corresponding to complex historical issues from all entity words.

[0067] The principle of steps B1-B2 is the same as that of steps A1-A2, and will not be repeated here.

[0068] S210: Based on historical documents and multiple historical knowledge points, the question generation model is trained using a preset loss function to obtain the trained question generation model. The question generation model is used to generate historical complex questions corresponding to historical documents.

[0069] The process of step S210 can be referred to the content of the above embodiment one, and will not be repeated here.

[0070] This specification provides a training method for a question generation model. After obtaining historical complex questions and their corresponding answers, as well as a historical document-based knowledge base corresponding to the historical complex questions, all entity words related to the historical complex questions in the historical documents are identified. These entity words do not appear in the historical complex questions. Then, based on each entity word, a corresponding simple question is determined. By comparing the similarity between each simple question and the historical complex question, historical knowledge points corresponding to the historical complex questions are determined from all entity words. Finally, based on the historical documents and multiple historical knowledge points, the question generation model is trained using a preset loss function to obtain the trained question generation model. By setting the condition that entity words must meet the condition of "not appearing in historical complex questions," unnecessary entity words can be eliminated, reducing unnecessary simple question generation and comparison operations, saving resources, and improving model training efficiency. This specification's embodiment determines historical knowledge points by comparing the similarity between each simple question and the historical complex question, using an unsupervised comparison method to retrieve historical knowledge points. This facilitates a more accurate comparison of the similarity between each simple question and the historical complex question, making the identified historical knowledge points more precise, thereby improving the accuracy of the trained model in generating complex questions.

[0071] Example 3

[0072] like Figure 4 As shown in the embodiments of this specification, another method for training a problem generation model is provided. This method may include:

[0073] S302: Obtain historical complex problems and their corresponding answers, as well as a historical document-based knowledge base corresponding to the historical complex problems. The historical document-based knowledge base shall include at least one historical document related to the historical complex problems.

[0074] S304: Extract entity words related to historical complex issues from historical documents, construct simple issues based on entity words, and identify multiple historical knowledge points based on simple issues and historical complex issues.

[0075] Specifically, after extracting entity words and constructing simple questions based on them, each simple question can be manually compared with a historical complex question to identify multiple historical knowledge points. Alternatively, historical complex questions can be compared with each simple question one by one based on preset keywords to identify multiple historical knowledge points. Furthermore, by comparing the similarity between each simple question and a historical complex question, one or more entity words with high similarity can be selected as historical knowledge points.

[0076] In one possible implementation, entity words related to historically complex issues are extracted from historical documents, simpler questions are constructed based on these entity words, and multiple historical knowledge points are identified based on the simpler and historically complex issues. This can be achieved by performing the following steps C1-C3:

[0077] Step C1: Identify all entity words in the historical documents that are related to the historical complex issue, and that the entity word does not appear in the historical complex issue.

[0078] Step C2: For each entity word, determine the corresponding simple question. A simple question is one that contains a single relation.

[0079] Step C3: By comparing the similarity between each simple question and the historically complex question, the historical knowledge points corresponding to the historically complex question are determined from all entity words.

[0080] S306: Based on historical documents and multiple historical knowledge points, the question generation model is trained using a preset loss function to obtain the trained question generation model. The question generation model is used to generate historical complex questions corresponding to historical documents.

[0081] S308: Retrieve multiple specified knowledge points and their corresponding document-based knowledge bases.

[0082] After training the problem generation model, when applying this model to generate complex problems, the input data consists of knowledge points and a related document-based knowledge base. This document-based knowledge base must include at least one document related to the knowledge point to be examined; this can be an open-source document or a user-provided document. The knowledge points can be specified by the user according to their needs. In one possible implementation, 2-3 knowledge points can be specified for a historical complex problem.

[0083] S310: Input multiple knowledge points and document-based knowledge bases into the question generation model to obtain complex questions with multiple knowledge points as test points.

[0084] The processing of steps 302-306 in this embodiment can be referred to the relevant content of Embodiment 1 or Embodiment 2 above, and will not be repeated here.

[0085] This specification provides a method for training a question generation model. It involves acquiring historical complex questions and their corresponding answers, as well as a historical document-based knowledge base. Entity words related to the historical complex questions are extracted from the historical documents. Simple questions are constructed based on these entity words. Multiple historical knowledge points are determined based on the simple questions and the historical complex questions. Then, the question generation model is trained using a preset loss function based on the historical documents and these knowledge points. After training, multiple currently specified knowledge points and their corresponding document-based knowledge bases are acquired. These knowledge points and the document-based knowledge bases are input into the question generation model to generate complex questions that test these multiple knowledge points, thus completing the generation of complex questions. This specification embodiment trains a question generation model and then uses this model to generate complex questions. When training the question generation model, complex historical questions are automatically generated by extracting multiple historical knowledge points from historical documents. The extracted knowledge points are based on entity words related to these complex historical questions, and the historical knowledge points are ultimately determined by combining simple questions constructed from these entity words with the complex historical questions. This method of extracting historical knowledge points makes the generated complex historical questions more targeted and the assessment of knowledge points more precise, thus improving the efficiency of model training and the accuracy of the model in generating complex questions. Compared to the AUTOQA training method, the question generation model training method in this specification only requires 10% of the complex questions and their corresponding sample size for training. Typically, around 100 complex questions and their corresponding answers are needed to train a usable model, eliminating the need for a large number of samples, saving resources, and improving model training efficiency. Furthermore, the trained question generation model can intelligently generate complex questions and can specify multiple knowledge points as test points, effectively saving manpower and improving question generation efficiency. Moreover, since the generated complex questions are not simple questions that can be directly found on the internet, the risk of credential stuffing attacks is greatly reduced, resulting in higher stability of the generated complex questions. Since each complex problem corresponds to at least two knowledge points, it also corresponds to at least two simple problems. This is equivalent to breaking down a complex problem into two simple problems, making the model training method more interpretable.

[0086] Example 4

[0087] The above describes the training method for a problem generation model provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a training device for a problem generation model, such as... Figure 5 As shown.

[0088] The training device for the problem generation model includes: an input module 410, a historical knowledge point determination module 420, and a model training module 430, wherein:

[0089] Input module 410 obtains historical complex problems and their corresponding answers, as well as a historical document-type knowledge base corresponding to the historical complex problems. The historical document-type knowledge base includes at least one historical document related to the historical complex problems.

[0090] The historical knowledge point identification module 420 extracts entity words related to complex historical issues from historical documents, constructs simple questions based on entity words, and identifies multiple historical knowledge points based on simple questions and complex historical issues;

[0091] The model training module 430 trains the question generation model based on historical documents and multiple historical knowledge points using a preset loss function to obtain the trained question generation model. The question generation model is used to generate historical complex questions corresponding to historical documents.

[0092] In the embodiments of this specification, the historical knowledge point determination module 420 includes:

[0093] The entity word determination unit identifies all entity words in historical documents that are related to historical complex issues, and which do not appear in the historical complex issues.

[0094] The simple question building unit determines the corresponding simple question based on each entity word. A simple question is a question containing a single relation.

[0095] The similarity comparison unit determines the historical knowledge points corresponding to the historical complex issues from all entity words by comparing the similarity between each simple question and the historical complex issues.

[0096] In one possible implementation, as described in this embodiment, the simple question construction unit generates a simple question corresponding to each entity word based on an automatic question-answering system, using each entity word as the answer.

[0097] In one possible implementation, the similarity comparison unit includes:

[0098] The similarity calculation subunit obtains the similarity between each simple problem and the historical complex problem by calculating the BLEU value between each simple problem and the historical complex problem, or by calculating the maximum number of characters in the same substring between each simple problem and the historical complex problem.

[0099] Based on similarity, the historical knowledge point extraction sub-unit is used to determine multiple historical knowledge points corresponding to complex historical issues from all entity words according to the calculated similarity and the set similarity threshold.

[0100] The historical complex problems in the embodiments of this specification include: problems involving multi-hop relationships, problems with constraints, or problems involving both multi-hop relationships and constraints.

[0101] In one possible implementation, the historical document-type knowledge base in the input module 410 is an open-source, updatable knowledge base.

[0102] In one possible implementation, the number of historical knowledge points corresponding to the complex historical problem in the historical knowledge point determination module 420 is 2-3.

[0103] In one possible implementation, the loss function preset in the model training module 430 is the cross-entropy loss function.

[0104] In the embodiments described in this specification, the training device for the problem generation model further includes:

[0105] The information acquisition module retrieves multiple specified knowledge points and their corresponding document-based knowledge bases.

[0106] The complex question generation module takes multiple knowledge points and a document-based knowledge base as inputs into the question generation model to generate complex questions that test multiple knowledge points.

[0107] Those skilled in the art will understand that the training apparatus for the above-described problem generation model can be used to implement the training method for the problem generation model described above. The detailed description therein should be similar to that described in the method section above, and will not be repeated here to avoid being cumbersome.

[0108] This specification provides a training device for a question generation model. An input module acquires historical complex questions and their corresponding answers, as well as a historical document-based knowledge base corresponding to the historical complex questions. A historical knowledge point determination module extracts entity words related to the historical complex questions from the historical documents, constructs simple questions based on these entity words, and determines multiple historical knowledge points based on the simple questions and the historical complex questions. Finally, a model training module trains the question generation model using the historical documents and multiple historical knowledge points, employing a preset loss function to obtain the trained question generation model. By extracting multiple historical knowledge points from historical documents, the automatic generation of historical complex questions is achieved. The extracted knowledge points are based on entity words related to the historical complex questions, and the historical knowledge points are ultimately determined based on the simple questions constructed from these entity words and the historical complex questions. This method of extracting historical knowledge points makes the generated historical complex questions more targeted, the assessment of knowledge points more precise, and improves the efficiency of model training and the accuracy of the model in generating complex questions. Furthermore, the trained question generation model can intelligently generate complex questions and can specify multiple knowledge points as test points, effectively saving manpower and improving question generation efficiency. Furthermore, since the generated complex questions are not simple questions that can be directly found on the internet, the risk of credential stuffing attacks is greatly reduced, and the generated complex questions are more stable. Because each complex question corresponds to at least two knowledge points, it also corresponds to at least two simple questions, essentially decomposing a complex question into two simple questions, making the model training method more interpretable. In addition, compared to the AUTOQA training method, the question generation model training method in the embodiments of this specification only requires 10% of the complex questions and their corresponding sample size for training. Typically, a model trained with around 100 complex questions and their corresponding answers is sufficient to reach a usable level, eliminating the need for a large number of samples, saving resources, and improving model training efficiency.

[0109] Example 5

[0110] The above describes the training apparatus for the problem generation model provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide an electronic device, such as... Figure 6 As shown.

[0111] Electronic devices can vary considerably due to differences in configuration or performance, and may include one or more processors 501 and memory 502. Memory 502 may store one or more application programs or data. Memory 502 may be temporary or persistent storage. The application programs stored in memory 502 may include one or more modules (not shown), each module including a series of computer-executable instructions for the electronic device. Furthermore, processor 501 may be configured to communicate with memory 502 and execute the series of computer-executable instructions stored in memory 502 on the electronic device. The electronic device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input / output interfaces 505, and one or more keyboards 506.

[0112] Specifically, in this embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0113] Obtain historical complex problems and their corresponding answers, as well as a historical document-based knowledge base corresponding to the historical complex problems. The historical document-based knowledge base shall include at least one historical document related to the historical complex problems.

[0114] Extract entity words related to complex historical issues from historical documents, construct simple questions based on entity words, and identify multiple historical knowledge points based on simple questions and complex historical issues;

[0115] Based on historical documents and multiple historical knowledge points, the question generation model is trained using a preset loss function to obtain the trained question generation model. The question generation model is used to generate historical complex questions corresponding to historical documents.

[0116] This specification provides an electronic device that acquires historical complex questions and their corresponding answers, as well as a historical document-based knowledge base corresponding to the historical complex questions. It extracts entity words related to the historical complex questions from the historical documents, constructs simple questions based on these entity words, and determines multiple historical knowledge points based on the simple questions and the historical complex questions. Finally, it trains a question generation model using a preset loss function based on the historical documents and the multiple historical knowledge points, resulting in a trained question generation model. By extracting multiple historical knowledge points from historical documents, the device automatically generates historical complex questions. The extracted knowledge points are based on entity words related to the historical complex questions, and the historical knowledge points are ultimately determined based on the simple questions constructed from these entity words and the historical complex questions. This method of extracting historical knowledge points makes the generated historical complex questions more targeted, the assessment of knowledge points more precise, and improves the efficiency of model training and the accuracy of the model in generating complex questions. Furthermore, the trained question generation model can intelligently generate complex questions and can specify multiple knowledge points as test points, effectively saving manpower and improving question generation efficiency. Moreover, since the generated complex questions are not simple questions that can be directly found on the internet, the risk of credential stuffing attacks is greatly reduced, and the generated complex questions are more stable. Since each complex problem corresponds to at least two knowledge points, it also corresponds to at least two simple problems. This is equivalent to decomposing a complex problem into two simple problems, making the model training method more interpretable. Furthermore, compared to the AUTOQA training method, the question generation model training method in this specification only requires 10% of the complex problems and their corresponding sample sizes for training. Typically, around 100 complex problems and their corresponding answers are needed to train a usable model. This eliminates the need for a large number of samples, saving resources and improving model training efficiency.

[0117] Example 6

[0118] Furthermore, based on the above Figures 1-4 The method shown in this specification, along with one or more embodiments, also provides a storage medium for storing computer-executable instructions. In one specific embodiment, the storage medium may be a USB flash drive, optical disc, hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, it can achieve the following process:

[0119] Obtain historical complex problems and their corresponding answers, as well as a historical document-based knowledge base corresponding to the historical complex problems. The historical document-based knowledge base shall include at least one historical document related to the historical complex problems.

[0120] Extract entity words related to complex historical issues from historical documents, construct simple questions based on entity words, and identify multiple historical knowledge points based on simple questions and complex historical issues;

[0121] Based on historical documents and multiple historical knowledge points, the question generation model is trained using a preset loss function to obtain the trained question generation model. The question generation model is used to generate historical complex questions corresponding to historical documents.

[0122] This specification provides a storage medium that acquires historical complex questions and their corresponding answers, as well as a historical document-based knowledge base corresponding to the historical complex questions. It extracts entity words related to the historical complex questions from the historical documents, constructs simple questions based on these entity words, and determines multiple historical knowledge points based on the simple questions and the historical complex questions. Finally, it trains a question generation model using a preset loss function based on the historical documents and the multiple historical knowledge points, resulting in a trained question generation model. By extracting multiple historical knowledge points from historical documents, the system automatically generates historical complex questions. The extracted knowledge points are based on entity words related to the historical complex questions, and the historical knowledge points are ultimately determined based on the simple questions constructed from these entity words and the historical complex questions. This method of extracting historical knowledge points makes the generated historical complex questions more targeted, the assessment of knowledge points more precise, and improves the efficiency of model training and the accuracy of the model in generating complex questions. Furthermore, the trained question generation model can intelligently generate complex questions and can specify multiple knowledge points as test points, effectively saving manpower and improving question generation efficiency. Moreover, since the generated complex questions are not simple questions that can be directly found on the internet, the risk of credential stuffing attacks is greatly reduced, and the generated complex questions are more stable. Since each complex problem corresponds to at least two knowledge points, it also corresponds to at least two simple problems. This is equivalent to decomposing a complex problem into two simple problems, making the model training method more interpretable. Furthermore, compared to the AUTOQA training method, the question generation model training method in this specification only requires 10% of the complex problems and their corresponding sample sizes for training. Typically, around 100 complex problems and their corresponding answers are needed to train a usable model. This eliminates the need for a large number of samples, saving resources and improving model training efficiency.

[0123] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0124] The apparatus, module, or unit described in the above embodiments can be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0125] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0126] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] This specification describes one or more embodiments of methods, apparatus (systems), and computer program products according to embodiments of this specification with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0128] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0130] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0131] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0132] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0133] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. This specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0135] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0136] The above description is merely one or more embodiments of this specification and is not intended to limit this application. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A method for training a problem generation model, comprising: Obtain historical complex questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the historical complex questions, wherein the historical document-type knowledge base includes at least one historical document related to the historical complex questions; Extract entity words related to the historical complex issues from the historical documents, construct simple questions based on the entity words, and identify multiple historical knowledge points by comparing the overlap between the simple questions and the historical complex issues; Using the historical documents and multiple historical knowledge points as input data, and the historical complex questions as output results, the question generation model is trained using a preset loss function to obtain a trained question generation model, which is used to generate the historical complex questions corresponding to the historical documents.

2. The method according to claim 1, wherein extracting entity words related to the historical complex issue from the historical documents, constructing a simple issue based on the entity words, and determining multiple historical knowledge points by comparing the overlap between the simple issue and the historical complex issue, includes: Identify all entity words in the historical documents that are related to the historical complex issue, and that the entity words do not appear in the historical complex issue; For each entity word, a corresponding simple question is determined, wherein the simple question is a question containing a single relation; By comparing the similarity between each simple question and the historical complex question, the historical knowledge points corresponding to the historical complex question are determined from all entity words.

3. The method according to claim 2, wherein determining the corresponding simple question based on each entity word includes: Using each entity word as the answer, a simple question corresponding to each entity word is generated based on the automatic question answering system.

4. The method according to claim 2, wherein determining the historical knowledge point corresponding to the historical complex problem from all entity words by comparing the similarity between each simple problem and the historical complex problem includes: The similarity between each simple problem and the historical complex problem is obtained by calculating the BLEU value of each simple problem and the historical complex problem, or by calculating the maximum number of characters in the same substring of each simple problem and the historical complex problem. Based on the calculated similarity and according to the set similarity threshold, multiple historical knowledge points corresponding to the historical complex issue are determined from all entity words.

5. According to the method of claim 1 or 4, the number of historical knowledge points corresponding to the complex historical problem is 2-3.

6. The method according to claim 1, wherein the historically complex problem includes: Problems involving multi-hop relationships and / or problems with constraints.

7. The method according to claim 1, wherein the preset loss function is the cross-entropy loss function.

8. The method according to claim 1, further comprising: Retrieves multiple specified knowledge points and their corresponding document-based knowledge bases; By inputting the multiple knowledge points and the document-based knowledge base into the question generation model, a complex question is obtained that uses the multiple knowledge points as test points.

9. A training device for a problem generation model, comprising: The input module obtains historical complex questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the historical complex questions. The historical document-type knowledge base includes at least one historical document related to the historical complex questions. The historical knowledge point identification module extracts entity words related to the historical complex issues from the historical documents, constructs simple questions based on the entity words, and identifies multiple historical knowledge points by comparing the overlap between the simple questions and the historical complex issues. The model training module takes the historical documents and multiple historical knowledge points as input data and the historical complex questions as output results. It trains the question generation model through a preset loss function to obtain the trained question generation model. The question generation model is used to generate the historical complex questions corresponding to the historical documents.

10. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, enable the processor to: Obtain historical complex questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the historical complex questions, wherein the historical document-type knowledge base includes at least one historical document related to the historical complex questions; Extract entity words related to the historical complex issues from the historical documents, construct simple questions based on the entity words, and identify multiple historical knowledge points by comparing the overlap between the simple questions and the historical complex issues; Using the historical documents and multiple historical knowledge points as input data, and the historical complex questions as output results, the question generation model is trained using a preset loss function to obtain a trained question generation model, which is used to generate the historical complex questions corresponding to the historical documents.

Images