Training of question generation model

Abstract

Disclosed in one or more embodiments of the present description is a method for training a question generation model. The method comprises: firstly, acquiring a historical complex question and an answer corresponding thereto, and a historical document-based knowledge base corresponding to the historical complex question, wherein the historical document-based knowledge base at least comprises one historical document related to the historical complex question; then extracting entity words related to the historical complex question from the historical document, constructing simple questions on the basis of the entity words, and determining a plurality of historical knowledge points on the basis of the simple questions and the historical complex question; and finally, on the basis of the historical document and the plurality of historical knowledge points, training a question generation model by means of a preset loss function to obtain a trained question generation model, wherein the question generation model is used for generating a historical complex question corresponding to a historical document.

Description

Training the Question Generation Model Technical Field

[0001] This document relates to the field of intelligent question answering technology, and in particular to a training method and device for a question generation model. Background Art

[0002] Intelligent question-answering is an important form of human-computer interaction. The intelligent question-answering system is a new type of information retrieval system that processes natural language. It adopts a question-and-answer format, receives questions from users, and then accurately locates the relevant knowledge in the question-answer database involved in the question. It then conducts business navigation according to the set business processing flow, realizes the interaction between business guidance and users, and feeds back relevant knowledge to users as the answer to the question, completing intelligent question-answering.

[0003] Current intelligent question-answering systems typically only ask simple questions, such as "What year were you born?" These questions don't require associative reasoning or reflection. Because both the question and the answer are relatively simple, these simple questions fail to meet user needs in some areas. For example, in lie detection, users may respond quickly to simple questions without reacting, making it impossible to observe their reactions and resulting in inaccurate lie detection. Therefore, there is an urgent need for a method that can generate complex questions that require associative reasoning and reflection.

[0004] Summary of the Invention

[0005] The purpose of the embodiments of this specification is to provide a technical solution that can intelligently generate complex questions to meet user needs.

[0006] In order to implement the above technical solution, the embodiments of this specification are implemented as follows.

[0007] On the one hand, one or more embodiments of the present specification provide a method for training a question generation model, including: obtaining a complex historical question and its corresponding answer, and a historical document-type knowledge base corresponding to the complex historical question, wherein the historical document-type knowledge base includes at least one historical document related to the complex historical question; extracting entity words related to the complex historical question from the historical document, constructing a simple question based on the entity words, and determining multiple historical knowledge points based on the simple question and the complex historical question; based on the historical document and the multiple historical knowledge points, training the question generation model through a preset loss function to obtain a trained question generation model, and the question generation model is used to generate the complex historical question corresponding to the historical document.

[0008] On the other hand, one or more embodiments of the present specification provide a training device for a question generation model, comprising: an input module, which obtains a complex historical question and its corresponding answer, and a historical document-type knowledge base corresponding to the complex historical question, wherein the historical document-type knowledge base includes at least one historical document related to the complex historical question; a historical knowledge point determination module, which extracts entity words related to the complex historical question from the historical document, constructs a simple question based on the entity words, and determines multiple historical knowledge points based on the simple question and the complex historical question; a model training module, which trains the question generation model based on the historical document and multiple historical knowledge points through a preset loss function to obtain a trained question generation model, and the question generation model is used to generate the complex historical question corresponding to the historical document.

[0009] On the other hand, one or more embodiments of the present specification provide an electronic device, comprising: a processor; and a memory arranged to store computer-executable instructions, which, when the executable instructions are executed, enable the processor to: obtain a complex historical question and its corresponding answer, and a historical document-type knowledge base corresponding to the complex historical question, wherein the historical document-type knowledge base includes at least one historical document related to the complex historical question; extract entity words related to the complex historical question from the historical document, construct a simple question based on the entity words, and determine multiple historical knowledge points based on the simple question and the complex historical question; based on the historical document and the multiple historical knowledge points, train a question generation model through a preset loss function to obtain a trained question generation model, and the question generation model is used to generate the complex historical question corresponding to the historical document.

[0010] On another aspect, one or more embodiments of the present specification provide a storage medium for storing computer-executable instructions, which implement the following process when executed by a processor: obtaining a complex historical question and its corresponding answer, and a historical document-type knowledge base corresponding to the complex historical question, wherein the historical document-type knowledge base includes at least one historical document related to the complex historical question; extracting entity words related to the complex historical question from the historical document, constructing a simple question based on the entity words, and determining multiple historical knowledge points based on the simple question and the complex historical question; based on the historical document and the multiple historical knowledge points, training a question generation model through a preset loss function to obtain a trained question generation model, and the question generation model is used to generate the complex historical question corresponding to the historical document. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] FIG1 is a schematic flow chart of a method for training a question generation model in this specification;

[0013] FIG2 is a schematic diagram of a training method for a question generation model in this specification;

[0014] FIG3 is a schematic flow chart of another method for training a question generation model in this specification;

[0015] FIG4 is a schematic flow chart of another method for training a question generation model in this specification;

[0016] FIG5 is a schematic block diagram of a training device for a question generation model in this specification;

[0017] FIG6 is a schematic block diagram of an electronic device in this specification. DETAILED DESCRIPTION

[0018] One or more embodiments of this specification provide a method and apparatus for training a question generation model.

[0019] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0020] Currently, when setting questions for educational exams and risk control exams, in order to better assess the respondent's true level, it is often necessary to design questions that require the respondent to engage in certain associative reasoning and thinking before answering. In conversational scenarios where intelligent robots primarily ask questions, asking complex questions is an important research direction. Common methods for generating complex questions include manually creating a question bank, which involves manually designing and adding new complex questions. However, this method requires designers to read extensively about relevant knowledge before writing complex questions. When the knowledge content is updated, the designer needs to continue to invest in updating the complex questions. Complex questions generated by this method are easily overcome by users' repeated attempts (i.e., database collision). In addition, there is an automatic question-answering system called AUTOQA that can generate simple questions. This automatic question-answering system is primarily trained using a large amount of internet information. However, due to the lack of sufficient internet corpus for the required complex questions, it is not possible to train an automatic question-answering system for complex questions. Based on this, the embodiments of this specification provide a method and apparatus for training a question generation model. This method completes model training and reaches a usable level using a relatively small sample size of complex questions and their corresponding answers, thereby significantly improving the efficiency and effectiveness of complex question generation. This is described in detail below.

[0021] Example 1

[0022] As shown in Figure 1, the embodiments of this specification provide a method for training a question generation model. The method can be performed by a terminal device or a server. The terminal device can be a mobile phone, tablet computer, or a computer such as a laptop or desktop computer. The server can be a standalone server or a server cluster consisting of multiple servers. The server can be a backend server for an application. The method can include the following steps.

[0023] In step S102, a complex historical question and its corresponding answer, as well as a historical document-type knowledge base corresponding to the complex historical question are obtained, where the historical document-type knowledge base includes at least one historical document related to the complex historical question.

[0024] In the embodiments of this specification, complex historical questions and their corresponding answers may be data used for model learning. In this embodiment, the complex historical questions and their corresponding answers may be a pair (i.e., complex historical question - answer) or multiple pairs. When training the model, one or more pairs of complex historical questions and their corresponding answers may be obtained from a designated database as samples, or one or more pairs of complex historical questions and their corresponding answers may be obtained from multiple different users (or data providers) as samples.

[0025] The historical complex questions in the embodiments of this specification, as well as the complex questions mentioned later, refer to questions that require the respondent to perform certain associative reasoning and thinking in order to complete the answer, while simple questions do not require the respondent to perform associative reasoning and thinking or the respondent only needs to perform very simple associative reasoning and thinking to answer directly.

[0026] The complex historical questions (i.e., complex historical question texts) or complex questions (i.e., complex question texts) in the embodiments of this specification may be questions involving multi-hop relationships, questions with constraints, or questions involving both multi-hop relationships and constraints. A complex question involving multi-hop relationships, for example: "Who are the directors of the movie starring A?", requires a multi-hop reasoning path formed by multiple triples to be answered, for example: through the triples "A, starring, new XX story", "new XX story, director, B", it can be inferred that "B" is the answer to the complex question. A complex question with constraints, for example: "Who won the first Wimbledon men's singles?", the "first" in this question represents a constraint on the answer entity. In actual applications, A is replaced with the name of a specific film and television star, and B is replaced with the name of a specific director.

[0027] Simple questions can often contain a single relationship. For example, "What year were you born?" can be a simple question, while "What important events happened in the year you were born?" can be a complex question.

[0028] In the embodiments of this specification, the historical document-based knowledge base corresponds to complex historical questions. That is, the answers to complex historical questions can be found in the historical document-based knowledge base. In practice, a complex historical question may correspond to one related historical document, or to two or more related historical documents. This embodiment of this specification does not limit this.

[0029] In a possible implementation, the historical document type knowledge base is an open source and updateable knowledge base, which can be obtained through the Internet, and the updated historical document type knowledge base can be obtained as the Internet information is updated.

[0030] In step S104, entity words related to historical complex questions are extracted from historical documents, simple questions are constructed based on the entity words, and multiple historical knowledge points are determined based on the simple questions and historical complex questions.

[0031] The historical documents corresponding to the complex historical issues may include a variety of entity words, from which entity words related to the complex historical issues can be extracted, and corresponding simple questions can be constructed based on each entity word. Alternatively, corresponding simple questions can be constructed based on multiple entity words in the historical documents, thereby forming one or more simple questions related to the complex historical issues. The embodiments of this specification extract entity words related to the complex historical issues from the historical documents and then construct corresponding simple questions for them, which can significantly reduce the number of simple questions unrelated to the complex historical issues, thereby improving model training efficiency and saving resources.

[0032] In implementation, simple questions can be manually set based on entity words through expert experience, or an intelligent question-answering system can be used to automatically generate simple questions corresponding to entity words. This embodiment of the present specification does not limit this.

[0033] As can be seen from step S104, after constructing a simple question based on the entity words, multiple historical knowledge points are determined based on the simple question and the complex historical question. In implementation, each simple question can be compared with the complex historical question to determine multiple historical knowledge points. Alternatively, based on preset keywords, the complex historical question can be compared with each simple question one by one to determine multiple historical knowledge points. Alternatively, the similarity between each simple question and the complex historical question can be compared to select one or more entity words with higher similarity as historical knowledge points.

[0034] In the embodiments of this specification, the number of multiple historical knowledge points determined based on simple questions and historical complex questions is specifically set according to the complex questions required by the user. In one possible implementation, the number of historical knowledge points is 2-3. Accordingly, after the question generation model is trained, when the question generation model is used to generate complex questions, the number of knowledge points can be 2-3. This number of knowledge points can ensure that the generated complex questions or historical complex questions can meet the user's questioning needs without being too complex and taking up too many resources.

[0035] Figure 2 is a schematic diagram of a training method for a question generation model in this specification. Figure 2 takes a historical document knowledge base containing two historical documents as an example. The specific historical documents, historical complex questions and their corresponding answers are as follows.

[0036] Historical Document 1: Geneva Conference of 1954.

[0037] From April 26 to July 21, 1954, a diplomatic conference involving the Soviet Union, the United States, Britain, France and China was held at the League of Nations building in Geneva.

[0038] Historical Document 2:A.

[0039] Born in April 1954 in Hong Kong, China, he is a Hong Kong film and television actor, director, producer, screenwriter, and singer. He entered the film industry in 1971 as a martial artist.

[0040] Corresponding complex historical question: What important events happened in the year A was born?

[0041] The answer to the complex historical question: the Geneva Conference.

[0042] For the complex historical question "What important events occurred in the year A was born?", its corresponding two knowledge points, "What year was A born?" and "What important events occurred in 1954?", require relevant information to be found in the corresponding historical documents 1 and 2. To determine these two knowledge points, we first construct corresponding simple questions for entity words related to the complex historical question in the two historical documents. These entity words include: 1954, Hong Kong, China, and A. Simple questions constructed include: "What year was A born?" for the entity word "1954," "Where was A born?" for the entity word "Hong Kong, China," and "Who was born in Hong Kong on April 7, 1954?" for the entity word "A." Finally, by comparing these simple questions with the complex historical question, we determine that the entity word "1954" is most likely the test point for the complex historical question and can be used as a historical knowledge point. Since the entity word "A" already appears in the complex historical question, it can be omitted as a historical knowledge point. The entity word "Hong Kong, China" is not highly relevant to the complex historical question and can be omitted as a historical knowledge point.

[0043] In step S106, based on the historical documents and multiple historical knowledge points, the question generation model is trained by a preset loss function to obtain a trained question generation model, which is used to generate complex historical questions corresponding to the historical documents.

[0044] In one possible implementation, after determining the corresponding multiple historical knowledge points for each complex historical problem, a controllable generation prompt learning mechanism can be used, with the historical documents and multiple historical knowledge points corresponding to the complex historical problem as input data, and the complex historical problem as output result. Controllable generation training can be performed based on a preset loss function until the preset loss function converges or the set number of iterations is reached, to obtain a problem generation model.

[0045] The controllable generation of prompt learning mechanisms uses a text-based controllable generation approach. This refers to using a model to generate corresponding text for certain controllable suggestive elements in the input data. In the examples of this specification, the controllable suggestive elements are historical documents and multiple historical knowledge points corresponding to complex historical issues, and the output text is: Complex Historical Issues.

[0046] The question generation model generated in a controllable text generation manner can be a pre-trained language model. The question generation model can adopt any one of the GPT (Generative Pre-Training), BART (Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, used to pre-train the seq-to-seq model denoising autoencoder), and T5 (Transfer Text-to-text Transformer, unified text to text task) models, and the embodiments of this specification are not limited to this.

[0047] The preset loss function can adopt the cross entropy loss function (CE-loss), focal loss function (focal-loss), etc. The goal to be achieved by the preset loss function can be to make the historical complex questions generated by the question generation model as close as possible to the historical complex questions used as labels.

[0048] In one possible implementation, the preset loss function may adopt a cross-entropy loss function, which is more convenient to apply and helps to improve the accuracy and efficiency of model training.

[0049] The embodiment of this specification provides a training method for a question generation model, which obtains historical complex questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the historical complex questions, extracts entity words related to the historical complex questions from the historical documents, constructs simple questions based on the entity words, and determines multiple historical knowledge points based on the simple questions and the historical complex questions. Finally, based on the historical documents and the multiple historical knowledge points, the question generation model is trained by a preset loss function to obtain a trained question generation model. The automatic generation of historical complex questions is completed by extracting multiple historical knowledge points from the historical documents. The extracted knowledge points are based on entity words related to the historical complex questions, and the historical knowledge points are finally determined based on the simple questions constructed by the entity words and the historical complex questions. This method of extracting historical knowledge points makes the generated historical complex questions more targeted, and the assessment of the knowledge points is more accurate, which is conducive to improving the efficiency of model training and improving the accuracy of the model generation of complex questions. In addition, through the trained question generation model, complex questions can be intelligently generated, and multiple knowledge points can be specified as test points for assessment, which can effectively save manpower and improve the efficiency of question generation. Moreover, since the generated complex questions are not simple questions that can be directly queried on the Internet, the risk of database collision breakthrough can be greatly reduced, and the generated complex questions are more stable. Since each complex question can correspond to at least two knowledge points, it also corresponds to at least two simple questions accordingly, which is equivalent to decomposing a complex question into two simple questions, making the model training method more explanatory. In addition, the question generation model training method in the embodiment of this specification, compared with the training method of AUTOQA, only 10% of the sample size of complex questions and their corresponding questions is needed as a reference for training. Usually, a model trained with about 100 complex questions and their corresponding answers can reach a usable level. It does not require a large number of samples, can save resources, and is also beneficial to improving the efficiency of model training.

[0050] Example 2

[0051] As shown in Figure 3, another method for training a question generation model provided in an embodiment of this specification may include the following steps.

[0052] S202: Obtaining a complex historical question and its corresponding answer, as well as a historical document-type knowledge base corresponding to the complex historical question, wherein the historical document-type knowledge base includes at least one historical document related to the complex historical question.

[0053] The method for obtaining samples for model training in this embodiment can be found in the content of the above-mentioned embodiment 1 and will not be repeated here.

[0054] S204: Determine entity words in the historical document that are related to the historical complex issue, and the entity words do not appear in the historical complex issue.

[0055] If an entity word already appears in a complex historical question, then the entity word may not be used as a knowledge point (or test point). Based on this, the entity words can be removed from the entity words related to the complex historical question in the historical document. Removing the entity words in step S204 is beneficial for not generating the corresponding simple questions in the subsequent step S206, and reducing the number of simple questions compared with the complex historical question in S208, thereby saving resources and improving model training efficiency.

[0056] S206: Determine a corresponding simple question based on each entity word, wherein a simple question is a question containing a single relationship.

[0057] In one possible implementation, simple questions are generated using AUTOQA. Specifically, each entity word is used as an answer, and a simple question corresponding to each entity word is generated based on an automatic question-answering system.

[0058] S208: By comparing the similarity between each simple question and the complex historical question, the historical knowledge points corresponding to the complex historical question are determined from all entity words.

[0059] As can be seen from the above steps S204-S208, when extracting entity words, constructing simple questions based on the entity words, and determining multiple historical knowledge points based on the simple questions and historical complex questions, the embodiment of this specification uses a similarity comparison method to extract historical knowledge points. This method of comparing the overlap between simple questions and complex questions uses an unsupervised comparison method to retrieve historical knowledge points, which is conducive to more accurately comparing the similarity between each simple question and the historical complex question, thereby making the determined historical knowledge points more accurate, and further improving the accuracy of the trained model in generating complex questions.

[0060] In one possible implementation, by comparing the similarity between each simple question and the complex historical question, the historical knowledge points corresponding to the complex historical question are determined from all entity words, and the following steps A1-A2 can be performed.

[0061] In step A1, the similarity between the simple question and the historically complex question is obtained by calculating the BLEU (Bilinggual Evaluation Understudy, a bilingual translation quality assessment auxiliary tool, an automatic evaluation method for machine translation / text summarization) value of each simple question and the historically complex question.

[0062] In step A2, based on the calculated similarity and a set similarity threshold, multiple historical knowledge points corresponding to the complex historical issues are determined from the determined entity words.

[0063] In implementation, according to the set similarity threshold, the answers to simple questions corresponding to the similarities higher than the set similarity threshold can be used as historical knowledge points. For example, when the BLEU value exceeds the preset BLEU threshold, it is determined that the entity word corresponding to the simple question is the knowledge point corresponding to the complex historical question; otherwise, it is determined that the entity word corresponding to the simple question is not the knowledge point corresponding to the complex historical question.

[0064] The obtained similarities may also be sorted in descending order, and the answers to the first n (n is a positive integer greater than or equal to 1) simple questions in the sorted sequence may be used as historical knowledge points.

[0065] In a possible implementation, by comparing the similarity between the simple question and the complex historical question, the historical knowledge point corresponding to the complex historical question is determined from the determined entity words, and the following steps B1-B2 can be performed.

[0066] Step B1: Calculate the maximum number of identical substrings between each simple question and the historical complex question to obtain the similarity between each simple question and the historical complex question.

[0067] Step B2: Based on the calculated similarity and the set similarity threshold, multiple historical knowledge points corresponding to the complex historical issues are determined from all entity words.

[0068] The principles of steps B1-B2 are the same as those of steps A1-A2 and will not be repeated here.

[0069] S210: Based on the historical documents and multiple historical knowledge points, the question generation model is trained using a preset loss function to obtain a trained question generation model, which is used to generate complex historical questions corresponding to the historical documents.

[0070] The processing of step S210 can refer to the content of the above embodiment 1 and will not be repeated here.

[0071] The embodiment of this specification provides a training method for a question generation model. After obtaining a complex historical question and its corresponding answer, as well as a historical document-type knowledge base corresponding to the complex historical question, all entity words related to the complex historical question in the historical document are determined. The entity word does not appear in the complex historical question. Then, based on each entity word, the corresponding simple question is determined. By comparing the similarity of each simple question with the complex historical question, the historical knowledge point corresponding to the complex historical question is determined from all entity words. Finally, based on the historical document and multiple historical knowledge points, the question generation model is trained by a preset loss function to obtain a trained question generation model. Since the entity word is set to meet the condition of "not appearing in the complex historical question", unnecessary entity words can be eliminated, unnecessary simple question generation and comparison operations can be reduced, resources can be saved, and the efficiency of model training can be improved. The embodiment of this specification determines the historical knowledge point by comparing the similarity of each simple question with the complex historical question, and retrieves the historical knowledge point by unsupervised comparison, which is conducive to more accurately comparing the similarity of each simple question with the complex historical question, thereby making the determined historical knowledge point more accurate, thereby improving the accuracy of the trained model in generating complex questions.

[0072] Example 3

[0073] As shown in Figure 4, another method for training a question generation model provided in an embodiment of this specification may include the following steps.

[0074] S302: Obtain a complex historical question and its corresponding answer, as well as a historical document-type knowledge base corresponding to the complex historical question, wherein the historical document-type knowledge base includes at least one historical document related to the complex historical question.

[0075] S304: extracting entity words related to complex historical issues from historical documents, constructing simple questions based on the entity words, and determining multiple historical knowledge points based on the simple questions and complex historical issues.

[0076] Specifically, after extracting entity words and constructing simple questions based on them, each simple question can be manually compared with a complex historical question to determine multiple historical knowledge points. Alternatively, based on preset keywords, the complex historical question can be compared with each simple question one by one to determine multiple historical knowledge points. Alternatively, by comparing the similarity between each simple question and the complex historical question, one or more entity words with higher similarity can be selected as historical knowledge points.

[0077] In one possible implementation, entity words related to complex historical issues are extracted from historical documents, simple questions are constructed based on the entity words, and multiple historical knowledge points are determined based on the simple questions and complex historical issues. The following steps C1-C3 can be performed.

[0078] Step C1: determine all entity words related to the historical complex question in the historical document, and the entity words do not appear in the historical complex question.

[0079] In step C2, a corresponding simple question is determined based on each entity word. A simple question is a question containing a single relationship.

[0080] In step C3, by comparing the similarity between each simple question and the complex historical question, the historical knowledge points corresponding to the complex historical question are determined from all entity words.

[0081] S306: Based on the historical documents and multiple historical knowledge points, the question generation model is trained using a preset loss function to obtain a trained question generation model, which is used to generate complex historical questions corresponding to the historical documents.

[0082] S308: Acquire the currently specified multiple knowledge points and their corresponding document-based knowledge bases.

[0083] After training the model to generate a question generation model, the input data for applying this model to complex question generation is knowledge points and a related document-based knowledge base. This document-based knowledge base includes at least one document related to the knowledge point to be examined. This can be open source or user-provided. Knowledge points can be specified by the user based on their needs. In one possible implementation, two to three knowledge points can be specified for a complex historical question.

[0084] S310: Input multiple knowledge points and a document-based knowledge base into a question generation model to obtain a complex question with multiple knowledge points as test points.

[0085] For the processing of steps 302 to 306 in this embodiment, reference may be made to the relevant contents of the above-mentioned embodiment 1 or embodiment 2, and no further details will be given here.

[0086] The embodiment of this specification provides a method for training a question generation model, by obtaining complex historical questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the complex historical questions, extracting entity words related to the complex historical questions from the historical documents, constructing simple questions based on the entity words, and determining multiple historical knowledge points based on the simple questions and the complex historical questions, and then training the question generation model based on the historical documents and the multiple historical knowledge points through a preset loss function to obtain a trained question generation model. After the model training is completed, the currently specified multiple knowledge points and their corresponding document-type knowledge bases are obtained, and the multiple knowledge points and the document-type knowledge base are input into the question generation model to obtain a complex question with multiple knowledge points as test points, thereby completing the generation of complex questions. The embodiment of this specification trains a question generation model and then uses the question generation model to generate complex questions. When training the question generation model, the automatic generation of historical complex questions is completed by extracting multiple historical knowledge points from historical documents. The extracted knowledge points are based on entity words related to the historical complex questions, and the historical knowledge points are finally determined based on simple questions constructed by entity words and historical complex questions. This method of extracting historical knowledge points makes the generated historical complex questions more targeted, and the assessment of knowledge points is more accurate, which is conducive to improving the efficiency of model training and improving the accuracy of model generation of complex questions. Using the question generation model training method in the embodiment of this specification, compared with the training method of AUTOQA, only 10% of the sample size of complex questions and their corresponding questions is needed as a reference for training. Usually, a model trained with about 100 complex questions and their corresponding answers can reach a usable level. It does not require a large number of samples, can save resources, and is also conducive to improving model training efficiency. In addition, through the trained question generation model, complex questions can be intelligently generated, and multiple knowledge points can be specified as test points for assessment, which can effectively save manpower and improve question generation efficiency. Moreover, since the generated complex questions are not simple questions that can be directly queried on the Internet, the risk of database collision breakthrough can be greatly reduced, and the generated complex questions are more stable. Since each complex problem can correspond to at least two knowledge points, it also corresponds to at least two simple problems. This is equivalent to decomposing a complex problem into two simple problems, making the model training method more interpretable.

[0087] Example 4

[0088] The above is a training method for a question generation model provided in an embodiment of this specification. Based on the same idea, an embodiment of this specification provides a training device for a question generation model, as shown in FIG5 .

[0089] The training device of the question generation model includes: an input module 410, a historical knowledge point determination module 420 and a model training module 430, wherein: the input module 410 obtains historical complex questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the historical complex questions, and the historical document-type knowledge base includes at least one historical document related to the historical complex questions; the historical knowledge point determination module 420 extracts entity words related to the historical complex questions from the historical documents, constructs simple questions based on the entity words, and determines multiple historical knowledge points based on the simple questions and the historical complex questions; the model training module 430 trains the question generation model based on the historical documents and multiple historical knowledge points through a preset loss function to obtain a trained question generation model, and the question generation model is used to generate historical complex questions corresponding to the historical documents.

[0090] In the embodiment of this specification, the historical knowledge point determination module 420 includes: an entity word determination unit, which determines all entity words related to historical complex issues in historical documents, and the entity words do not appear in the historical complex issues; a simple question construction unit, which determines the corresponding simple question based on each entity word, and a simple question is a question containing a single relationship; a similarity comparison unit, which determines the historical knowledge points corresponding to the historical complex issues from all entity words by comparing the similarity of each simple question with the historical complex issues.

[0091] In a possible implementation, in an embodiment of this specification, the simple question construction unit uses each entity word as an answer and generates a simple question corresponding to each entity word based on an automatic question-answering system.

[0092] In one possible implementation, the similarity comparison unit includes: a similarity calculation subunit, which obtains the similarity between each simple question and the historical complex question by calculating the BLEU value of each simple question and the historical complex question respectively or calculating the maximum number of identical substring words of each simple question and the historical complex question respectively; a historical knowledge point extraction subunit based on similarity, which determines multiple historical knowledge points corresponding to the historical complex question from all entity words according to the calculated similarity and a set similarity threshold.

[0093] The historically complex problems in the embodiments of this specification include: problems involving multi-hop relationships, problems with constraints, or problems involving multi-hop relationships and problems with constraints.

[0094] In a possible implementation, the historical document knowledge base in the input module 410 is an open source and updateable knowledge base.

[0095] In a possible implementation, the number of the multiple historical knowledge points corresponding to the complex historical question in the historical knowledge point determination module 420 is 2-3.

[0096] In one possible implementation, the loss function preset in the model training module 430 adopts a cross entropy loss function.

[0097] In an embodiment of the present specification, the training device of the question generation model also includes: an information acquisition module, which obtains multiple currently specified knowledge points and their corresponding document-type knowledge bases; a complex question generation module, which inputs multiple knowledge points and document-type knowledge bases into the question generation model to obtain complex questions with multiple knowledge points as test points.

[0098] Those skilled in the art should understand that the training device for the above-mentioned question generation model can be used to implement the training method for the question generation model described above, and the detailed description should be similar to that described in the above method part. To avoid tediousness, it will not be repeated here.

[0099] The embodiment of this specification provides a training device for a question generation model, which obtains historical complex questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the historical complex questions, through an input module, extracts entity words related to the historical complex questions from the historical documents through a historical knowledge point determination module, constructs simple questions based on the entity words, and determines multiple historical knowledge points based on the simple questions and the historical complex questions. Finally, the model training module trains the question generation model based on the historical documents and the multiple historical knowledge points through a preset loss function to obtain a trained question generation model. The automatic generation of historical complex questions is completed by extracting multiple historical knowledge points from the historical documents. The extracted knowledge points are based on the entity words related to the historical complex questions, and the historical knowledge points are finally determined based on the simple questions constructed by the entity words and the historical complex questions. This method of extracting historical knowledge points makes the generated historical complex questions more targeted and the assessment of the knowledge points more accurate, which is conducive to improving the efficiency of model training and improving the accuracy of the model generation of complex questions. In addition, the trained question generation model can intelligently generate complex questions, and multiple knowledge points can be specified as test points for assessment, which can effectively save manpower and improve the efficiency of question generation. Moreover, since the generated complex questions are not simple questions that can be directly queried on the Internet, the risk of database collision breakthrough can be greatly reduced, and the generated complex questions are more stable. Since each complex question can correspond to at least two knowledge points, it also corresponds to at least two simple questions accordingly, which is equivalent to decomposing a complex question into two simple questions, making the model training method more explanatory. In addition, the question generation model training method in the embodiment of this specification, compared with the training method of AUTOQA, only 10% of the sample size of complex questions and their corresponding questions is needed as a reference for training. Usually, a model trained with about 100 complex questions and their corresponding answers can reach a usable level. It does not require a large number of samples, can save resources, and is also beneficial to improving the efficiency of model training.

[0100] Example 5

[0101] The above is a training device for a question generation model provided in an embodiment of this specification. Based on the same idea, an embodiment of this specification provides an electronic device, as shown in FIG6 .

[0102] Electronic devices may have relatively large differences due to different configurations or performances, and may include one or more processors 501 and memory 502, and the memory 502 may store one or more storage applications or data. Among them, the memory 502 can be a temporary storage or a persistent storage. The application stored in the memory 502 may include one or more modules (not shown in the figure), each module may include a series of computer-executable instructions for the electronic device. Furthermore, the processor 501 can be configured to communicate with the memory 502 to execute a series of computer-executable instructions in the memory 502 on the electronic device. The electronic device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input and output interfaces 505, and one or more keyboards 506.

[0103] Specifically in this embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the electronic device, and is configured to be executed by one or more processors. The one or more programs include instructions for performing the following computer-executable instructions: obtaining complex historical questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the complex historical questions, the historical document-type knowledge base including at least one historical document related to the complex historical questions; extracting entity words related to the complex historical questions from the historical documents, constructing simple questions based on the entity words, and determining multiple historical knowledge points based on the simple questions and the complex historical questions; based on the historical documents and the multiple historical knowledge points, training the question generation model through a preset loss function to obtain the trained question generation model, and the question generation model is used to generate complex historical questions corresponding to the historical documents.

[0104] The embodiment of this specification provides an electronic device, which obtains complex historical questions and their corresponding answers, as well as a historical document-based knowledge base corresponding to the complex historical questions, extracts entity words related to the complex historical questions from the historical documents, constructs simple questions based on the entity words, and determines multiple historical knowledge points based on the simple questions and the complex historical questions. Finally, based on the historical documents and the multiple historical knowledge points, a question generation model is trained through a preset loss function to obtain a trained question generation model. The automatic generation of complex historical questions is completed by extracting multiple historical knowledge points from the historical documents. The extracted knowledge points are based on entity words related to the complex historical questions, and the historical knowledge points are finally determined based on the simple questions constructed by the entity words and the complex historical questions. This method of extracting historical knowledge points makes the generated complex historical questions more targeted and the assessment of knowledge points more accurate, which is conducive to improving the efficiency of model training and improving the accuracy of model generation of complex questions. In addition, through the trained question generation model, complex questions can be intelligently generated, and multiple knowledge points can be specified as test points for assessment, which can effectively save manpower and improve the efficiency of question generation. Moreover, since the generated complex questions are not simple questions that can be directly queried on the Internet, the risk of database collision breakthrough can be greatly reduced, and the generated complex questions are more stable. Since each complex question can correspond to at least two knowledge points, it also corresponds to at least two simple questions, which is equivalent to decomposing a complex question into two simple questions, making the model training method more explanatory. In addition, the question generation model training method in the embodiment of this specification only requires 10% of the sample size of complex questions and their corresponding questions as a reference for training compared to the training method of AUTOQA. Usually, a model trained with about 100 complex questions and their corresponding answers can reach a usable level. It does not require a large number of samples, can save resources, and is also conducive to improving the efficiency of model training.

[0105] Example 6

[0106] Furthermore, based on the method shown in Figures 1 to 4 above, one or more embodiments of the present specification also provide a storage medium for storing computer-executable instructions. In a specific embodiment, the storage medium may be a U disk, an optical disk, a hard disk, etc. The computer-executable instruction information stored in the storage medium can implement the following process when executed by the processor: obtaining complex historical questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the complex historical questions, the historical document-type knowledge base including at least one historical document related to the complex historical questions; extracting entity words related to the complex historical questions from the historical documents, constructing simple questions based on the entity words, and determining multiple historical knowledge points based on the simple questions and the complex historical questions; based on the historical documents and multiple historical knowledge points, training the question generation model through a preset loss function to obtain the trained question generation model, and the question generation model is used to generate complex historical questions corresponding to the historical documents.

[0107] The embodiment of this specification provides a storage medium, which obtains historical complex questions and their corresponding answers, as well as a historical document-type knowledge base corresponding to the historical complex questions, extracts entity words related to the historical complex questions from the historical documents, constructs simple questions based on the entity words, and determines multiple historical knowledge points based on the simple questions and the historical complex questions. Finally, based on the historical documents and the multiple historical knowledge points, the question generation model is trained by a preset loss function to obtain a trained question generation model. The automatic generation of historical complex questions is completed by extracting multiple historical knowledge points from the historical documents. The extracted knowledge points are based on the entity words related to the historical complex questions, and the historical knowledge points are finally determined based on the simple questions constructed by the entity words and the historical complex questions. This method of extracting historical knowledge points makes the generated historical complex questions more targeted, the assessment of knowledge points is more accurate, which is conducive to improving the efficiency of model training and improving the accuracy of model generation of complex questions. In addition, through the trained question generation model, complex questions can be intelligently generated, and multiple knowledge points can be specified as test points for assessment, which can effectively save manpower and improve the efficiency of question generation. Moreover, since the generated complex questions are not simple questions that can be directly queried on the Internet, the risk of database collision breakthrough can be greatly reduced, and the generated complex questions are more stable. Since each complex question can correspond to at least two knowledge points, it also corresponds to at least two simple questions, which is equivalent to decomposing a complex question into two simple questions, making the model training method more explanatory. In addition, the question generation model training method in the embodiment of this specification only requires 10% of the sample size of complex questions and their corresponding questions as a reference for training compared to the training method of AUTOQA. Usually, a model trained with about 100 complex questions and their corresponding answers can reach a usable level. It does not require a large number of samples, can save resources, and is also conducive to improving the efficiency of model training.

[0108] The foregoing description of specific embodiments of this specification is intended to be a description of other embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0109] The devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0110] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0111] It will be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] One or more embodiments of this specification are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0113] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0115] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0116] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0117] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0118] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0119] One or more embodiments of this specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In distributed computing environments, program modules may be located in local and remote computer storage media, including storage devices.

[0120] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0121] The foregoing is merely one or more embodiments of this specification and is not intended to limit this application. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A method for training a question generation model, comprising: Acquire a complex historical question and its corresponding answer, and a historical document-type knowledge base corresponding to the complex historical question, wherein the historical document-type knowledge base includes at least one historical document related to the complex historical question; Extracting entity words related to the complex historical question from the historical document, constructing simple questions based on the entity words, and determining a plurality of historical knowledge points based on the simple questions and the complex historical question; Based on the historical document and multiple historical knowledge points, the question generation model is trained through a preset loss function to obtain a trained question generation model, and the question generation model is used to generate complex historical questions corresponding to the historical document.

2. According to the method of claim 1, extracting entity words related to the complex historical question from the historical document, constructing a simple question based on the entity words, and determining a plurality of historical knowledge points based on the simple question and the complex historical question, comprises: Determining all entity words in the historical document that are related to the complex historical issue, and the entity words do not appear in the complex historical issue; Determine a corresponding simple question according to each entity word, wherein the simple question is a question containing a single relationship; By comparing the similarity between each simple question and the complex historical question respectively, the historical knowledge points corresponding to the complex historical question are determined from all entity words.

3. According to the method of claim 2, determining a corresponding simple question according to each entity word comprises: Take each entity word as the answer and generate simple questions corresponding to each entity word based on the automatic question-answering system.

4. According to the method of claim 2, the step of comparing the similarity between each simple question and the complex historical question to determine the historical knowledge points corresponding to the complex historical question from all entity words comprises: By respectively calculating the BLEU value of each simple question and the complex historical question or respectively calculating the maximum number of identical substrings of each simple question and the complex historical question, the similarity between each simple question and the complex historical question is obtained; According to the calculated similarity and the set similarity threshold, multiple historical knowledge points corresponding to the complex historical issue are determined from all entity words.

5. According to the method of claim 1 or 4, the number of multiple historical knowledge points corresponding to the complex historical question is 2-3.

6. The method according to claim 1, wherein the historical complex issues include: Problems involving multi-hop relationships and / or problems with constraints.

7. According to the method of claim 1, the preset loss function adopts a cross entropy loss function.

8. The method according to claim 1, further comprising: Get the currently specified multiple knowledge points and their corresponding document-based knowledge base; The multiple knowledge points and the document-based knowledge base are input into the question generation model to obtain complex questions with the multiple knowledge points as test points.

9. A training device for a question generation model, comprising: An input module is used to obtain a complex historical question and its corresponding answer, and a historical document-type knowledge base corresponding to the complex historical question, wherein the historical document-type knowledge base includes at least one historical document related to the complex historical question; A historical knowledge point determination module extracts entity words related to the historical complex question from the historical document, constructs a simple question based on the entity words, and determines a plurality of historical knowledge points based on the simple question and the historical complex question; The model training module trains the question generation model based on the historical document and multiple historical knowledge points through a preset loss function to obtain a trained question generation model, and the question generation model is used to generate complex historical questions corresponding to the historical document.

10. An electronic device comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: Acquire a complex historical question and its corresponding answer, and a historical document-type knowledge base corresponding to the complex historical question, wherein the historical document-type knowledge base includes at least one historical document related to the complex historical question; Extracting entity words related to the complex historical question from the historical document, constructing simple questions based on the entity words, and determining a plurality of historical knowledge points based on the simple questions and the complex historical question; Based on the historical document and multiple historical knowledge points, the question generation model is trained through a preset loss function to obtain a trained question generation model, and the question generation model is used to generate complex historical questions corresponding to the historical document.

Images