A training method and device for a text question answering model and a text question answering method

Abstract

The present application discloses a training method and device of a text question-answering model and a text question-answering method, wherein the training method comprises: obtaining a sample data set; the sample data set comprises sample data of a main task and sample data of an auxiliary task, the sample data of the main task comprises text, questions and answers, and the sample data of the auxiliary task comprises text, questions and answer evidence; based on each sample data, the text question-answering model is used to perform a corresponding task, and the parameters of the text question-answering model are optimized and adjusted using the task execution results; wherein, when the sample data is sample data of the main task, the text question-answering model is used to predict answers to questions in the sample data using an answer evidence perception method; when the sample data is sample data of an auxiliary task, the text question-answering model is used to predict answer evidence to questions in the sample data. By adopting the present application, the accuracy of answer prediction can be improved, and it has interpretability.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and in particular to a training method and device for a text question-answering model and a text question-answering method. Background Art

[0002] Text-based Question Answering can also be called Machine Reading Comprehension. Specifically, given one or more text paragraphs, the machine is required to read the content of the paragraphs and answer corresponding questions based on the content of the paragraphs.

[0003] At present, for text question answering tasks, text question answering models are usually built in an end-to-end manner, that is, given information such as paragraphs and questions are input into the model for processing, and the answer to the question is directly obtained. When training the model, the answer is used as a supervisory signal to guide the training of the entire model.

[0004] In the process of implementing the present invention, the inventors found that the above existing text question-answering solutions have problems of poor answer accuracy and unexplainable answers. Through research and analysis, the specific reasons for these problems are as follows:

[0005] In existing text question-answering solutions, when using a text question-answering model to generate answers, the text paragraphs, questions, and candidate answers are first concatenated into a sequence input into an encoder for encoding, and then the distributed representation obtained after encoding is input into a decision module for processing to obtain the final answer. Specifically, for "yes / no" type questions (i.e., questions with yes or no answers), the decision module is generally a binary classifier. When the predicted category is 0, it means the answer is no, and when the predicted category is 1, it means the answer is yes. For multiple-choice questions (i.e., questions with multiple options as candidate answers), the decision module is a simple nonlinear layer. The representation of each option can be mapped to a score through the decision module, and the final model selects the option with the highest score as the final answer. In the training process of the above-mentioned text question-answering model, only the answer is used as a supervisory signal, so the supervisory signal used for model training is only a simple category information. Because the information of the supervisory signal is too simple, the model cannot learn the semantic understanding of the text content and the ability to make logical inferences about the answer to the question based on the understanding of the text content through training. Through training, the model often learns some features of the question or text paragraph itself, but cannot accurately learn the correlation between the question and the text content, that is, the same answer is often given to different questions for the same text paragraph. Therefore, the existing end-to-end method of modeling directly on this data is very easy to introduce data set bias, thereby reducing the accuracy of the answer. In addition, the answer predicted by the model is only a simple category, which cannot provide the basis for the corresponding answer, resulting in the inability to know the reason for the answer, and thus the problem of unexplainable answers. Summary of the invention

[0006] In view of this, the main purpose of the present invention is to provide a training method and device for a text question and answer model and a text question and answer method, which can improve the accuracy of answers and have explainability.

[0007] In order to achieve the above object, the technical solution proposed in the embodiment of the present invention is:

[0008] A training method for a text question answering model, comprising:

[0009] Acquire a sample data set; the sample data set includes sample data of a main task and sample data of a secondary task, the sample data of the main task includes text, questions and answers, and the sample data of the secondary task includes text, questions and answer evidence;

[0010] Based on each sample data in the sample data set, a text question and answer model is used to perform a corresponding task, and the parameters of the text question and answer model are optimized and adjusted using the task execution results; wherein, when the sample data is the sample data of the main task, the text question and answer model is used to predict answers to questions in the sample data using an answer evidence perception method; when the sample data is the sample data of the auxiliary task, the text question and answer model is used to predict answer evidence to questions in the sample data.

[0011] The embodiment of the present invention further provides a text question answering method, comprising:

[0012] Get the target text and target question;

[0013] Based on the target text and the target question, a text question answering model is used to predict answer evidence and an answer to the target question, and the predicted result is output; wherein the text question answering model is obtained based on the above-mentioned training method.

[0014] An embodiment of the present invention also proposes a training device for a text question and answer model, comprising a processor and a memory; the memory stores an application program executable by the processor, which is used to enable the processor to execute the training method for the text question and answer model as described above.

[0015] In summary, the above technical solution proposed in the embodiment of the present invention trains the text question answering model based on sample data of two tasks, namely, the main task for predicting the answer to the question and the auxiliary task for predicting the evidence of the answer to the question, and adopts the answer evidence perception method to predict the answer to the question in the sample data. In this way, the supervision signal used for model training is no longer just a simple category signal for characterizing whether the answer is correct or not, but also includes an answer evidence related signal. In this way, on the one hand, the text question answering model can learn the ability to understand the semantics of the text content, and the ability to make logical inferences about the answer to the question based on the understanding of the text content, thereby improving the prediction ability of the text question answering model for the answer to the question; on the other hand, when using the trained text question answering model to predict the answer to the target question based on the target text, the evidence used to predict the answer in the target text can be provided, so that the predicted answer is interpretable. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a training method for a text question answering model according to an embodiment of the present invention;

[0017] Figure 2 Schematic diagram of the text question answering method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Figure 1 FIG. 1 is a flow chart of a training method for a text question answering model according to an embodiment of the present invention. Figure 1 As shown, the training method of the text question answering model implemented in this embodiment mainly includes:

[0020] Step 101, obtain a sample data set; the sample data set includes sample data of a main task and sample data of a secondary task, the sample data of the main task includes text, questions and answers, and the sample data of the secondary task includes text, questions and answer evidence.

[0021] In this step, unlike the existing solutions, it is necessary to obtain sample data of two tasks, namely, sample data of the main task and sample data of the auxiliary task. The main task is used to predict the answer to the question, and the auxiliary task is used to predict the evidence of the answer to the question (referred to as answer evidence in this article), that is, to extract text fragments from the text for inferring the answer to the question. In this way, when training based on sample data later, it is possible not only to train the text question answering model's ability to predict the answer to the question, but also to train the text question answering model's ability to predict the answer evidence, making the answer prediction interpretable.

[0022] In one implementation, the questions in the sample data may be yes / no questions or multiple choice questions.

[0023] Step 102: Based on each sample data in the sample data set, a text question-answering model is used to perform a corresponding task, and the parameters of the text question-answering model are optimized and adjusted using the task execution results; wherein, when the sample data is sample data of a main task, the text question-answering model is used to predict answers to questions in the sample data using an answer evidence-aware method; and when the sample data is sample data of a secondary task, the text question-answering model is used to predict answer evidence to questions in the sample data.

[0024] This step is used to train the text question answering model based on the sample data of the two tasks obtained. It should be noted here that when training the text question answering model based on the sample data of the main task, it is necessary to adopt the answer evidence perception method to predict the answers to the questions in the sample data. In this way, the text question answering model can learn the ability to understand the semantics of the text content, and the ability to make logical inferences about the answers to the questions based on the understanding of the text content, avoiding the introduction of data set bias, thereby improving the ability of the text question answering model to accurately predict the answers to the questions.

[0025] In one implementation, when the sample data is sample data of a main task, the following steps a1 to a2 may be specifically adopted to perform the corresponding task using the text question answering model:

[0026] Step a1: Based on the sample data, a text question and answer sequence is constructed according to the question type to obtain a text question and answer sequence.

[0027] In one implementation, the following method may be used to distinguish different question types (i.e., yes or no questions and multiple choice questions) to construct a text question-answering sequence:

[0028] If the question in the sample data is a yes-or-no question, the text in the sample data and the question are concatenated to obtain a text question-answer sequence.

[0029] For example, suppose the text in the sample data is a sequence of length l The problem is a sequence of length k

[0030] For "yes / no" tasks, the input is a sequence of text and question concatenated:

[0031]

[0032] Among them, [CLS] is a special character, indicating the starting position of the sequence; [SEP] is a separator, used to separate different input items. Since the input sequence includes text, questions and three special characters, the length of the text question and answer sequence is m = l + k + 3.

[0033] If the question in the sample data is a multiple-choice question, then for each candidate answer in the sample data, based on the candidate answer, the question in the sample data is converted into a corresponding yes-or-no question, and the text in the sample data and the converted question are concatenated to obtain a text question-answer sequence corresponding to the candidate answer.

[0034] It should be noted that multiple-choice questions can be regarded as special yes-no questions, that is, to judge whether each candidate answer is correct. Therefore, it is necessary to convert the task of multiple-choice questions into the task form of yes-no questions. For each candidate answer C, the original question word in the question can be replaced with the candidate answer according to the preset conversion rules, and the question words such as "yes" and "?" can be introduced to convert the original question into a yes-no question. For example, for the question "Where does Beijing University of Posts and Telecommunications belong to", the candidate answer set: {Ministry of Posts and Telecommunications, Ministry of Education} can be converted into two yes-no questions: "Is Beijing University of Posts and Telecommunications now affiliated with the Ministry of Posts and Telecommunications?" and "Is Beijing University of Posts and Telecommunications now affiliated with the Ministry of Education?" For each new question, it can be concatenated with the text into an input text question-answering sequence I. In the subsequent steps, each text question-answering sequence is input into the text question-answering model for processing, and the model will give a "yes" or "no" answer.

[0035] Step a2: for each of the text question and answer sequences, the text question and answer sequence is input into the text question and answer model, answer evidence is predicted, and the corresponding answer is predicted based on the prediction result of the answer evidence.

[0036] In this step, before predicting the answer, it is necessary to predict the evidence of the answer to the question first, and then predict the corresponding answer based on the prediction result of the answer evidence, so that the text question answering model can learn the ability to understand the semantics of the text content, and the ability to make logical inferences about the answer to the question based on the understanding of the text content.

[0037] In one implementation, the following steps a21 to a25 may be used to predict the answer evidence, and the corresponding answer may be predicted based on the prediction result of the answer evidence:

[0038] Step a21: Encode the text question-answer sequence using an encoder, and extract the encoded representation of the corresponding text from the encoded result.

[0039] It should be noted here that the text question and answer sequence contains text and questions. In this way, when encoding the text question and answer sequence, the question information will also be encoded into the text, so that the encoded representation of the text also contains the question information.

[0040] Specifically, the existing pre-trained BERT model based on the bidirectional Transformer encoder can be used to encode the text question-answer sequence. That is, the text question-answer sequence I is input into the BERT model to obtain the encoded representation H:

[0041] h=BERT(I)

[0042] The specific implementation of the BERT(·) function is the same as the existing solution. The text question answering model can directly call the existing BERT interface. Based on the text question answering sequence representation above: H is a matrix of dimension m×d, where d is the size of BERT’s hidden layer. Each row in H represents the representation of the word at the corresponding position, which can be written as:

[0043]

[0044] in, It is the encoding of the text part in the text question and answer sequence, which can be expressed as P ∈l×d to represent it.

[0045] Step a22: Based on the encoded representation of the text, predict the starting point and the end point of the answer evidence in the text to obtain the corresponding starting point probability distribution and the end point probability distribution.

[0046] In one implementation, the text obtained in step a21 may be represented by H P , respectively input two single-layer perceptrons (starting point perceptron and ending point perceptron) to predict the starting and ending position probabilities of the answer evidence in the text. The details are as follows:

[0047] α s =softmax(H P ·W s +b s )

[0048] α e =softmax(H P ·W e +b e )

[0049] Among them, α s is the starting probability distribution, α e is the endpoint probability distribution, W s is the weight parameter of the starting perceptron, W s ∈R d ,b s is the bias parameter of the starting point perceptron, b s ∈R;W e is the weight parameter of the endpoint perceptron, W s ∈R d ,b e is the bias parameter of the endpoint sensor, b e ∈R; these parameters are all trainable parameters.

[0050] Step a23: taking the starting point probability distribution as the weight distribution, performing weighted summation on the encoded representation of the text, and obtaining a text representation perceived at the starting point of the evidence; taking the endpoint probability distribution as the weight distribution, performing weighted summation on the encoded representation of the text, and obtaining a text representation perceived at the endpoint of the evidence.

[0051] This step is used to use the starting point probability distribution and the end point probability distribution obtained in the previous step to perform weighted processing on the encoded representation of the text, so that the weighted text representation can reflect the importance of each word in the text as evidence, that is, it can reflect the correlation between the text content and the answer. In this way, the model predicts the answer based on the weighted text representation, which can enable the model to learn the semantic understanding ability of the text content and the ability to perform logical reasoning on the answer to the question based on the understanding of the text content, thereby improving the accuracy of answer prediction. Specifically, the text representation of evidence starting point perception and the text representation of evidence end point perception can be obtained according to the following formulas:

[0052] h s =α s ·H P

[0053] h e =α e ·H P

[0054] Among them, h s is the evidence-based text representation, h e Textual representations for evidence endpoint awareness.

[0055] Step a24: Fuse the text representation perceived at the starting point of the evidence and the text representation perceived at the end point of the evidence to obtain an evidence-perceived text representation.

[0056] In one implementation, the following methods may be used to fuse the text representation perceived at the starting point of the evidence and the text representation perceived at the end point of the evidence:

[0057] Method 1: Represent the text perceived by the starting point of the evidence as h s and the textual representation of the evidence endpoint awareness h e Series connection, that is, according to h p =[h s ;h e ], and obtain the evidence-aware text representation h p .

[0058] Method 2: Use the average calculation method, that is, according to Get the evidence-aware text representation h p .

[0059] Method 3: Use the maximum value calculation method, that is, according to h p =max(h s ,h e ), and obtain the evidence-aware text representation h p ; max(,) represents the function of finding the maximum value, that is, h s and h e Take the maximum value of the elements in the same position in the vector h p .

[0060] Step a25: predict the answer to the question based on the evidence-aware text representation.

[0061] In one implementation, a binary classifier may be used to predict the answer. The classifier is defined as follows:

[0062] p c =softmax(W·h p +b)

[0063] Among them, W is the weight parameter of the classifier and b is the bias parameter of the classifier, both of which are trainable parameters, b∈R 2 When the cascade gating mechanism is used, the dimension of W is 2d×2, and when the average or maximum gating mechanism is used, the dimension of W is d×2.

[0064] In one implementation, when the sample data is sample data of an auxiliary task, the following steps b1 to b2 may be specifically adopted to perform the corresponding task using the text question answering model:

[0065] Step b1: Based on the sample data, a text question and answer sequence is constructed according to the question type to obtain a text question and answer sequence.

[0066] The specific implementation of this step is the same as step a1, and will not be repeated here.

[0067] Step b2: For each of the text question and answer sequences, the text question and answer sequence is input into a text question and answer model to predict answer evidence.

[0068] In one implementation, the following method may be used to predict the answer evidence:

[0069] Step b21: Encode the text question-answer sequence using an encoder, and extract the encoded representation of the corresponding text from the encoded result.

[0070] The specific implementation of this step is the same as step a21, which will not be repeated here.

[0071] Step b22: Based on the encoded representation of the text, predict the starting point and the end point of the answer evidence in the text to obtain the corresponding starting point probability distribution and the end point probability distribution.

[0072] The specific implementation of this step is the same as step a22, which will not be repeated here.

[0073] Step b23: Select the location with the maximum probability from the starting point probability distribution as the starting point of the answer evidence; select the location with the maximum probability from the ending point probability distribution as the ending point of the answer evidence; and use the text segment within the range of the starting point and the ending point as the answer evidence.

[0074] In this step, the segment with the highest joint probability of the start and end positions is directly used as the final answer evidence.

[0075] In step 102, the specific method of optimizing and adjusting the parameters of the text question answering model using the task execution result (predicted answer or evidence) is known to those skilled in the art. That is, based on the predicted answer or evidence, and the answer or evidence in the sample data, the loss function value of the task can be calculated, and based on the loss function value, the model parameters of the text question answering model are optimized. The loss function can specifically be cross entropy.

[0076] Based on the above model training method embodiment, the present invention also provides a text question answering method, such as Figure 2 As shown, the method includes:

[0077] Step 201: Obtain target text and target question.

[0078] Step 202: Based on the target text and the target question, a text question answering model is used to predict answer evidence and an answer to the target question, and the prediction result is output.

[0079] The text question answering model is pre-trained based on the above-mentioned text question answering model training method embodiment.

[0080] In one implementation, when outputting the predicted answer evidence, the predicted answer evidence may be marked in the target text by highlighting the evidence.

[0081] Based on the above training method embodiment, the embodiment of the present invention also provides a training device for a text question answering model, including a processor and a memory; the memory stores an application program that can be executed by the processor, which is used to enable the processor to execute the training method of the text question answering model as described above. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium. In addition, the operating system operating on the computer can also be used to complete part or all of the actual operations based on the instructions of the program code. The program code read from the storage medium can also be written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is made to perform part and all of the actual operations, thereby realizing the functions of any of the above text question answering model training method implementation methods.

[0082] The memory may be implemented as various storage media such as an electrically erasable programmable read-only memory (EEPROM), a flash memory (Flash memory), and a programmable program read-only memory (PROM). The processor may be implemented as including one or more central processing units or one or more field programmable gate arrays, wherein the field programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or the central processing unit core may be implemented as a CPU or an MCU.

[0083] It should be noted that not all steps and modules in the above processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only for the convenience of describing the functional division adopted. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices.

[0084] The hardware modules in each embodiment can be implemented mechanically or electronically. For example, a hardware module may include a specially designed permanent circuit or logic device (such as a dedicated processor, such as an FPGA or ASIC) for performing a specific operation. The hardware module may also include a programmable logic device or circuit (such as a general-purpose processor or other programmable processor) temporarily configured by software to perform a specific operation. As for whether to implement the hardware module mechanically, or using a dedicated permanent circuit, or using a temporarily configured circuit (such as configured by software), it can be decided based on cost and time considerations.

[0085] In this article, "schematic" means "serving as an example, instance or explanation", and any diagram or implementation method described as "schematic" in this article should not be interpreted as a more preferred or more advantageous technical solution. In order to make the drawings concise, only the parts related to the present invention are schematically shown in each figure, and do not represent the actual structure of the product. In addition, in order to make the drawings concise and easy to understand, in some figures, only one of the parts with the same structure or function is schematically drawn, or only one of them is marked. In this article, "one" does not mean that the number of the relevant parts of the present invention is limited to "only one", and "one" does not mean that the number of the relevant parts of the present invention is "more than one". In this article, "upper", "lower", "front", "back", "left", "right", "inside", "outside", etc. are only used to indicate the relative position relationship between the relevant parts, rather than to limit the absolute position of these relevant parts.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A training method for a text question answering model, characterized in that: include: Acquire a sample data set; the sample data set includes sample data of a main task and sample data of a secondary task, the sample data of the main task includes text, questions and answers, and the sample data of the secondary task includes text, questions and answer evidence; Based on each sample data in the sample data set, using the text question answering model, executing a corresponding task, and optimizing and adjusting the parameters of the text question answering model using the task execution result; in, When the sample data is sample data of the main task, using the text question answering model, adopting an answer evidence perception method, to predict answers to questions in the sample data; When the sample data is sample data of an auxiliary task, using the text question-answering model to predict answer evidence for questions in the sample data; When the sample data is sample data of the main task, using the text question answering model to perform the corresponding task includes: Based on the sample data, a text question-answer sequence is constructed according to the question type to obtain a text question-answer sequence. For each of the text question-answering sequences, the text question-answering sequence is input into a text question-answering model, answer evidence is predicted, and a corresponding answer is predicted based on the prediction result of the answer evidence; When the sample data is sample data of an auxiliary task, using the text question answering model to perform the corresponding task includes: Based on the sample data, a text question-answer sequence is constructed according to the question type to obtain a text question-answer sequence. For each of the text question-answering sequences, input the text question-answering sequence into a text question-answering model to predict answer evidence; The predicting of the answer evidence and predicting the corresponding answer based on the prediction result of the answer evidence includes: Encoding the text question-answer sequence using an encoder, and extracting an encoded representation of the corresponding text from the encoding result; Based on the encoded representation of the text, predict the starting point and the ending point of the answer evidence in the text to obtain the corresponding starting point probability distribution and the ending point probability distribution; Taking the starting point probability distribution as the weight distribution, performing weighted summation on the encoded representation of the text, and obtaining the text representation of the evidence starting point perception; taking the endpoint probability distribution as the weight distribution, performing weighted summation on the encoded representation of the text, and obtaining the text representation of the evidence endpoint perception; fusing the evidence start point perceived text representation and the evidence end point perceived text representation to obtain an evidence perceived text representation; An answer to the question is predicted based on the evidence-aware text representation.

2. The method according to claim 1, characterized in that The questions in the sample data are yes-or-no questions or multiple-choice questions.

3. The method according to claim 1, characterized in that The constructing of a text question-answering sequence according to question types includes: If the question in the sample data is a yes-or-no question, concatenating the text in the sample data with the question to obtain a text question-answer sequence; If the question in the sample data is a multiple-choice question, then for each candidate answer in the sample data, based on the candidate answer, the question in the sample data is converted into a corresponding yes-or-no question, and the text in the sample data and the converted question are concatenated to obtain a text question-answer sequence corresponding to the candidate answer.

4. The method according to claim 1, characterized in that: The fusion is as follows: The text representation of the evidence starting point perception and a textual representation of the evidence endpoint awareness Concatenate to obtain the evidence-aware text representation ; Or, follow , and obtain the evidence-aware text representation ; Or, follow , and obtain the evidence-aware text representation ; The max( , ) represents a function for finding the maximum value.

5. The method according to claim 1, characterized in that The prediction of answer evidence includes: Encoding the text question-answer sequence using an encoder, and extracting an encoded representation of the corresponding text from the encoding result; Based on the encoded representation of the text, predict the starting point and the ending point of the answer evidence in the text to obtain the corresponding starting point probability distribution and the ending point probability distribution; From the starting point probability distribution, select the location with the maximum probability as the starting point of the answer evidence; from the ending point probability distribution, select the location with the maximum probability as the ending point of the answer evidence; The text segment within the range of the starting point and the end point is used as the answer evidence.

6. A text question answering method, characterized in that: include: Get the target text and target question; Based on the target text and the target question, a text question answering model is used to predict answer evidence and an answer to the target question, and the predicted result is output; wherein the text question answering model is obtained based on any one of the training methods described in claims 1 to 5.

7. A training device for a text question answering model, characterized in that: including a processor and a memory; The memory stores an application program executable by the processor, which is used to enable the processor to execute the training method of the text question answering model as described in any one of claims 1 to 5.

Images