A multi-task learning based generative reading comprehension method

Abstract

The application provides a generative reading comprehension method based on multi-task learning, and the method comprises the following steps: obtaining a target question and a target paragraph, obtaining corresponding feature vectors of the target question and the target paragraph, inputting the feature vectors into a target reading comprehension model, and obtaining a target answer, so that the accuracy and reliability of the target answer are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information processing, and particularly relates to a generative reading comprehension method based on multi-task learning. BACKGROUND

[0002] Machine reading comprehension technology, that is, a technology of automatically answering a question raised by a user based on a given text, is one of the research focuses in the current automatic question answering field. In recent years, with the construction of large-scale reading comprehension data sets and the proposal of pre-training models, the machine reading comprehension technology has made great progress.

[0003] Currently, the mainstream machine reading comprehension model usually sets the answer as a continuous segment in a paragraph. This kind of extractive reading comprehension model has certain limitations, and can only directly take the segment in the paragraph as the answer, which leads to the fact that a natural and fluent answer cannot be given when a certain question is asked. In addition, if the question and the answer are separated, complete and clear information cannot be obtained only according to the answer. Compared with the extractive reading comprehension, the generative reading comprehension is no longer limited to extracting the answer directly from the paragraph segment, but refers to the paragraph, the question and the glossary to generate a more natural and complete expression as the answer. Therefore, how to design a better generative framework is an important challenge faced by the machine reading comprehension task. SUMMARY

[0004] In view of the above technical problems, the technical scheme adopted by the present application is as follows: a generative reading comprehension method based on multi-task learning, the method comprising the following steps:

[0005] S100, acquiring a target question and a target paragraph.

[0006] S200, acquiring corresponding feature vectors of the target question and the target paragraph.

[0007] S300, inputting the feature vectors into a target reading comprehension model to acquire a target answer.

[0008] The present application has at least the following beneficial effects:

[0009] The present application combines the extractive reading comprehension and the generative reading comprehension through the target reading comprehension model, introduces a copy mechanism in the generation process, generates a natural and fluent answer, and improves the accuracy and reliability of the target answer. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to make the technical solutions in the embodiments of the present application clearer, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0011] Figure 1 A flowchart of a generative reading comprehension method based on multi-task learning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0012] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0013] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0014] An embodiment of the present application provides a generative reading comprehension method based on multi-task learning, as shown in the figure, the method comprises the following steps: Figure 1

[0015] S100, obtaining a target question and a target paragraph.

[0016] Specifically, the target question and the target paragraph can be input by a user.

[0017] S200, obtaining a corresponding feature vector of the target question and the target paragraph.

[0018] Specifically, the target question and the target paragraph are spliced with a given identifier CLS and SEP to obtain a target sequence, wherein the CLS is placed at the beginning to identify the semantics of the target question and the target paragraph, and the SEP is placed in the middle of the target question and the target paragraph to separate the character input of the target question and the target paragraph.​

[0019] Furthermore, the target sequence is mapped to character embedding, position embedding, and sentence type embedding, and the three embedding representations are added together to obtain the feature vectors corresponding to the target question and the target paragraph.

[0020] S300 inputs the feature vector into the target reading comprehension model to obtain the target answer.

[0021] Specifically, obtaining the target reading comprehension model includes the following steps:

[0022] S10, Obtain the training dataset X = {X1, X2, ..., X...} j , ..., X n}, where X j ={X j1 X j2}, X j1 This is the j-th training question and training paragraph, X. j2 Is with X j1 The corresponding j-th training data is the true answer, where j ranges from 1 to n, and n refers to the number of training data.

[0023] S20, Based on the training dataset X, obtain the first training input dataset E = {E1, E2, ..., E...} j , ..., E n} and the first training dataset using real answers F = {F1, F2, ..., F j F n}, where E j It is X j1 The corresponding feature vectors, and the first training uses the real answer data F. j =X j2 .

[0024] S30, the constructed reading comprehension model is trained based on the first training input dataset E and the first training real answer dataset F, and the trained reading comprehension model is determined as the target reading comprehension model.

[0025] Furthermore, S30 specifically includes the following steps:

[0026] S31, divide the first training input dataset E into s1 training sets, s2 validation sets, and s3 test sets, where s1+s2+s3=n.

[0027] Optionally, s1:s2:s3 = 8:1:1.

[0028] S33. Construct a reading comprehension model and set parameter values. Use s1 training sets and their corresponding s1 first training real answer data to train the constructed reading comprehension model to obtain the trained reading comprehension model.

[0029] Specifically, the constructed reading comprehension model includes an extracted reading comprehension model, a UniLM one-way decoder, and a copy mechanism.

[0030] Furthermore, in S33, training the reading comprehension model using s1 training sets and their corresponding s1 first training real answer data includes the following steps:

[0031] S331, Obtain the second training input dataset E′={E′1,E′2,…,E′} corresponding to the s1 training sets. i , ..., E′ s1}, E′ i It is the i-th second training input data in the s1 training sets, where the value of i ranges from 1 to s1.

[0032] S332, input E′ into the constructed extractive reading comprehension model, and obtain the training semantic vector set A and the training extractive answer set Y for E′;

[0033] Specifically, the constructed extractive reading comprehension model includes a bidirectional encoder layer, a fully connected layer, and a softmax layer of the UniLM model.

[0034] Furthermore, the acquisition of the training extraction answer set includes the following steps:

[0035] S3321, input E′ into the bidirectional encoder of the UniLM model to obtain the corresponding list of encoded vectors.

[0036] Specifically, E′ is input into the bidirectional encoder of the UniLM model to obtain the encoding vector corresponding to each text, forming a list of encoding vectors.

[0037] S3322, Input the list of encoded vectors into the fully connected layer and obtain the first probability value of each encoded vector as the beginning and end of the target answer. The first probability value refers to the probability value that has not been normalized.

[0038] S3323, input the first probability value into the Softmax layer for normalization processing to obtain the second probability value.

[0039] S3324, based on the second probability value, obtain the extracted answer set for training.

[0040] S333: Input the training semantic vector set into the UniLM unidirectional decoder to obtain the training decoded semantic vector set.

[0041] S334, input the set of decoding semantic vectors for training into the copy mechanism and use BIO tagging to obtain a set of label sequences for training, the label for training being a combination of O or B, I and a, p, where a is an identifier of a common string of the extracted answer for training and the first real answer data for training, p is an identifier of a common string of the training paragraph corresponding to the first real answer data for training and the first input data for training, B is an identifier of a string in the target paragraph, I is an identifier of a string that can form a continuous segment in the target paragraph, and O is an identifier of a predicted string.

[0042] Specifically, when the label for training is O, it indicates that the value corresponding to the label for training is a predicted content, when the label for training is B, the strings not belonging to the training paragraph corresponding to the second input data for training are masked, and when the label for training is I, all strings that cannot form a continuous segment in the training paragraph corresponding to the second input data for training are masked. That is, during decoding, by means of masking, it is ensured that when the label for training is B or I, the value corresponding to the label for training is part of the training paragraph corresponding to the second input data for training.

[0043] Specifically, the label for training in S334 includes the following steps:

[0044] S3341, when the label for training is O, the copy mechanism is used to predict the value corresponding to the label for training.

[0045] S3342, when the label for training is B-a, the string is extracted from the extracted answer for training as the target answer corresponding to the label for training.

[0046] S3343, when the label for training is I-a, the string that can form a continuous segment is extracted from the extracted answer for training as the target answer corresponding to the label for training.

[0047] S335, based on the set of extracted answers for training and the set of label sequences for training, a set of target answers for training is predicted.

[0048] S336, by minimizing the loss function, a trained reading comprehension model is obtained, the loss function being the sum of a first training loss function and a second training loss function, the first training loss function being the loss of the extracted answer for training and the real answer for training, and the second training loss function being the loss of the target answer for training and the real answer for training.

[0049] S35, the trained reading comprehension model is evaluated, and the model whose evaluation result meets the first preset condition is determined as a reading comprehension model to be tested.

[0050] Specifically, the evaluation condition satisfies a first preset condition that the accuracy of the s1 training sets is greater than a first accuracy threshold, wherein the first accuracy threshold can be determined according to actual requirements.

[0051] S37, verifying the to-be-tested reading comprehension model by using the s2 verification sets and the corresponding s2 first training real answer data, and obtaining a verification result.

[0052] S39, determining the to-be-tested reading comprehension model as a target reading comprehension model when the verification result satisfies a second preset condition.

[0053] Specifically, the test result satisfies the second preset condition that the accuracy of the s2 training sets is greater than a second accuracy threshold, wherein the second accuracy threshold can be determined according to actual requirements.

[0054] Specifically, after S39, there is further included: testing the target reading comprehension model by using the s3 test sets and the corresponding s3 first training real answer data, and obtaining a corresponding evaluation index score.

[0055] Further, the present application uses ROUGE-N as an evaluation index to judge the generalization ability of the target copy model.

[0056] Specifically, it is known to those skilled in the art that any method for obtaining a ROUGE-N score in the prior art belongs to the protection scope of the present application, and will not be repeated here.

[0057] Based on S10-S30, the training data set is obtained and processed to obtain the first training input data set and the first training real answer data set, the first training input data set and the first training real answer data set are used to train the constructed reading comprehension model, the second training input data set corresponding to the training set is input into the constructed extractive reading comprehension model, the training semantic vector set and the training extraction answer set are obtained, the training semantic vector set is input into the UniLM unidirectional decoder, the training decoding semantic vector set is obtained, and the BIO label is used to input the copy mechanism to obtain the training label sequence, the training target answer set is predicted based on the training label sequence, the trained reading comprehension model is obtained by minimizing the loss function, and the training target answer is closer to the first training real answer data by minimizing the loss function.

[0058] To sum up, the application extracts the target answer by inputting the target question and the target paragraph into the target reading comprehension model at the same time, and the target answer is natural and fluent, the application combines the extractive reading comprehension model and the generative reading comprehension model, and introduces the copy mechanism in the generation process, so that the accuracy and reliability of the target answer are improved.

[0059] The embodiment of the application further provides a non-transitory computer readable storage medium, which can be arranged in an electronic device to save at least one instruction or at least one program related to a method in the method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to realize the method provided by the above embodiment.

[0060] The embodiment of the application further provides an electronic device, comprising a processor and the aforementioned non-transitory computer readable storage medium.

[0061] The embodiment of the application further provides a computer program product, which comprises program code, and when the program product runs on the electronic device, the program code is used to make the electronic device execute the steps in the method according to various exemplary embodiments of the application described in the specification.

[0062] Although some specific embodiments of the application have been described in detail by examples, those skilled in the art should understand that the above examples are only for illustration, not for limiting the scope of the application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the application. The scope of the application is defined by the appended claims.

Claims

1. A generative reading comprehension method based on multi-task learning, characterized in that, The method includes the following steps: S100, obtain the target question and target paragraph; S200, obtain the feature vectors corresponding to the target question and the target paragraph; S300, Input the feature vector into the target reading comprehension model to obtain the target answer. The acquisition of the target reading comprehension model includes the following steps: S10, Obtain the training dataset X = {X1, X2, ..., X...} j , ..., X n }, where X j ={X j1 X j2 }, X j1 This is the j-th training question and training paragraph, X. j2 Is with X j1 The corresponding j-th training data is the true answer, where j ranges from 1 to n, and n refers to the number of training data. S20, Based on the training dataset X, obtain the first training input dataset E={E1, E2, ..., E...} j , ..., E n } and the first training dataset using real answers F={F1, F2, ..., F j F n }, where E j It is X j1 The corresponding feature vectors, the first training uses the real answer data F j =X j2 ; S30, the constructed reading comprehension model is trained based on the first training input dataset E and the first training real answer dataset F, and the trained reading comprehension model is determined as the target reading comprehension model; wherein, S30 specifically includes the following steps: S31, divide the first training input dataset E into s1 training sets, s2 validation sets, and s3 test sets, where s1+s2+s3=n; S33, construct a reading comprehension model and set parameter values, and train the constructed reading comprehension model using s1 training sets and their corresponding s1 first training real answer data to obtain the trained reading comprehension model. The constructed reading comprehension model includes a constructed extractive reading comprehension model, a UniLM one-way decoder and a copy mechanism. In S33, training the reading comprehension model using s1 training sets and their corresponding s1 sets of first training real answer data includes the following steps: S331, Obtain the second training input dataset Eʹ={Eʹ1, Eʹ2, ..., Eʹ} corresponding to the s1 training sets. i ,…,Eʹ s1 }, Eʹ i It is the i-th feature vector in s1 training sets, where the value of i ranges from 1 to s1; S332, Input Eʹ into the constructed extractive reading comprehension model, and obtain the training semantic vector set and the training extractive answer set of Eʹ; S333, input the training semantic vector set into the UniLM unidirectional decoder to obtain the training decoding semantic vector set; S334, input the training decoded semantic vector set into the copy mechanism and use BIO annotation to obtain the training label sequence set. The training labels are O or a combination of B, I and a, p, where a is the identifier of the common string between the training extracted answer and the first training real answer data, p is the identifier of the common string between the training paragraph corresponding to the first training real answer data and the first training input data, B is the identifier of the string in the target paragraph, I is the identifier of the string that makes up the continuous segment in the target paragraph, and O is the identifier of the predicted string. S335, based on the training extracted answer set and the training label sequence set, predicts the training target answer set; S336, By minimizing the loss function, the trained reading comprehension model is obtained. The loss function is the sum of the first training loss function and the second training loss function. The first training loss function is the loss between the extracted answers and the actual answers during training, and the second training loss function is the loss between the target answers and the actual answers during training. S35, evaluate the trained reading comprehension model, and determine the model whose evaluation results meet the first preset condition as the reading comprehension model to be tested; S37. Use s2 validation sets and their corresponding s2 first training real answer data to validate the reading comprehension model to be tested and obtain the validation results. S39. When the verification result meets the second preset condition, the reading comprehension model to be tested is determined as the target reading comprehension model.

2. The method according to claim 1, characterized in that, Following S39, the following is also included: using s3 test sets and their corresponding s3 first training real answer data to test the target reading comprehension model and obtain the corresponding evaluation index scores.

3. The method according to claim 1, characterized in that, The constructed extractive reading comprehension model includes the bidirectional encoder layer, fully connected layer, and Softmax layer of the UniLM model.

4. The method according to claim 3, characterized in that, The acquisition of the extraction answer set for training in S332 includes the following steps: S3321, Input Eʹ into the bidirectional encoder of the UniLM model to obtain the corresponding list of encoded vectors; S3322, Input the list of encoded vectors into the fully connected layer and obtain the first probability value of each encoded vector as the beginning and end of the target answer. The first probability value refers to the probability value that has not been normalized. S3323, Input the first probability value into the Softmax layer for normalization to obtain the second probability value; S3324, based on the second probability value, obtain the extracted answer set for training.

5. The method according to claim 1, characterized in that, The training labels described in S334 include the following steps: S3341, When the training label is 0, the copy mechanism is used to predict the value corresponding to the training label; S3342, When the training label is Ba, extract a string from the training answer as the target answer corresponding to the training label; S3343, when the training label is Ia, extract the string that forms a continuous segment from the training answer as the target answer corresponding to the training label.

6. The method according to claim 1, characterized in that, s1:s2:s3 = 8:1:1.

Images