A reading comprehension data enhancement method and device based on back translation

Abstract

The present invention relates to a reading comprehension data enhancement method based on back translation, comprising: training a bidirectional neural machine translation model through Chinese-English bilingual parallel corpus; expanding reading comprehension document pre-annotated data through the neural machine translation model; and constructing answers to the pre-annotated data based on answers in the reading comprehension training data and the pre-annotated data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for enhancing reading comprehension data. Background Art

[0002] The current machine reading comprehension system mainly consists of the following steps: first, annotate a given passage and the corresponding question, annotate a fragment in the passage as the answer; then build a neural network model, input the passage and the question, and the annotated correct answer fragment is used as the model output. This method proposes a method for reading comprehension data enhancement based on back translation, which provides more available training data for building a reliable reading comprehension system in a low-resource domain, thereby alleviating the problem of scarce training data in the domain.

[0003] Existing more mature machine reading comprehension models are often trained with large-scale annotated news corpora. However, other professional fields (such as medical and military) lack sufficient annotated data, which makes it difficult to obtain reliable reading comprehension models. Summary of the invention

[0004] The purpose of the present invention is to provide a reading comprehension data enhancement method and device based on back translation, so as to solve the problem in the prior art that it is difficult to obtain a reliable reading comprehension model.

[0005] A first aspect of the present invention provides a reading comprehension data enhancement method based on back translation, the method comprising:

[0006] Train a bidirectional neural machine translation model using parallel corpora in Chinese and English;

[0007] Expand the pre-annotated data of reading comprehension documents through the neural machine translation model;

[0008] An answer to the pre-annotated data is constructed according to the answer in the reading comprehension training data and the pre-annotated data.

[0009] Preferably, the training of a bidirectional neural machine translation model using Chinese-English bilingual parallel corpora specifically includes:

[0010] Obtaining a Chinese-English bilingual parallel corpus pair; the Chinese-English bilingual parallel corpus pair includes a first matrix of Chinese word sequences preprocessed by word vectors and a second matrix of English word sequences preprocessed by word vectors;

[0011] Encoding the Chinese corpus in the document in the training data after the sub-word segmentation to obtain a first encoding representation of the input word sequence information;

[0012] Encoding the Chinese corpus in the Chinese-English bilingual parallel corpus pair to obtain a first encoded representation of the input word sequence information;

[0013] Encoding the input word sequence to obtain an encoded representation of each word;

[0014] Decoding the second matrix to obtain a decoded representation of the second matrix;

[0015] Substituting the first matrix and the second matrix to obtain an output hidden state of a decoder;

[0016] Determine the probability distribution of the decoder's output hidden state;

[0017] The word corresponding to the maximum probability in the probability distribution is selected as the output result of the neural machine translation model.

[0018] Preferably, the step of expanding the pre-annotated data of the reading comprehension document by using the neural machine translation model specifically includes:

[0019] Translating the first Chinese in the document in the reading comprehension training data into English through the Chinese-English translation model in the trained bidirectional neural machine translation model;

[0020] The English is translated into a second Chinese by using an English translation model in a trained bidirectional neural machine translation model; the first Chinese and the second Chinese constitute pre-labeled data.

[0021] Preferably, constructing the answer to the pre-annotated data according to the answer in the reading comprehension training data and the pre-annotated data specifically includes:

[0022] Obtaining the first and last characters of the answer in the reading comprehension training data, and obtaining the vector representation of the first and last characters through the pre-trained word vector;

[0023] Traversing multiple characters in the pre-labeled data, obtaining a first score of the first character of the first and last characters for the word vector of each character, and a second score of the last character of the first and last characters for the word vector of the character; the first score of each character and the second score constitute a score set of the multiple characters;

[0024] The word with the highest score is obtained from the score set to obtain the answer of the pre-labeled data.

[0025] Preferably, the bidirectional neural machine translation model includes a Chinese-English model and an English-Chinese model.

[0026] A second aspect of the present invention provides a reading comprehension data enhancement device based on back translation, the device comprising:

[0027] A training module, which is used to train a bidirectional neural machine translation model using Chinese and English bilingual parallel corpora;

[0028] An expansion module, the expansion module is used to expand the pre-annotated data of reading comprehension documents through the neural machine translation model;

[0029] A construction module is used to construct an answer to the pre-labeled data according to the answer in the reading comprehension training data and the pre-labeled data.

[0030] Preferably, the training module is specifically used for:

[0031] Obtaining a Chinese-English bilingual parallel corpus pair; the Chinese-English bilingual parallel corpus pair includes a first matrix of Chinese word sequences preprocessed by word vectors and a second matrix of English word sequences preprocessed by word vectors;

[0032] The sentences in the reading comprehension training data are segmented into sub-words by the word segmentation method to obtain sub-word sequences;

[0033] Encoding the Chinese corpus in the Chinese-English bilingual parallel corpus pair to obtain a first encoded representation of the input word sequence information;

[0034] Encoding the input word sequence to obtain an encoded representation of each word;

[0035] Decoding the second matrix to obtain a decoded representation of the second matrix;

[0036] Substituting the first matrix and the second matrix to obtain an output hidden state of a decoder;

[0037] Determine the probability distribution of the decoder's output hidden state;

[0038] The word corresponding to the maximum probability in the probability distribution is selected as the output result of the neural machine translation model.

[0039] Preferably, the expansion module is specifically used for:

[0040] Translating the first Chinese in the document in the reading comprehension training data into English through the Chinese-English translation model in the trained bidirectional neural machine translation model;

[0041] The English is translated into a second Chinese by using an English translation model in a trained bidirectional neural machine translation model; the first Chinese and the second Chinese constitute pre-labeled data.

[0042] Preferably, the building blocks are specifically used for:

[0043] Obtaining the first and last characters of the answer in the reading comprehension training data, and obtaining the vector representation of the first and last characters through the pre-trained word vector;

[0044] Traversing multiple characters in the pre-labeled data, obtaining a first score of the first character of the first and last characters for the word vector of each character, and a second score of the last character of the first and last characters for the word vector of the character; the first score of each character and the second score constitute a score set of the multiple characters;

[0045] The word with the highest score is obtained from the score set to obtain the answer of the pre-labeled data.

[0046] Preferably, the bidirectional neural machine translation model includes a Chinese-English model and an English-Chinese model.

[0047] By applying the back-translation-based reading comprehension data enhancement method provided by the present invention, the data scale and data diversity can be effectively increased, and to a certain extent, the problem of poor model performance caused by data scarcity in the field can be alleviated, thereby obtaining a robust and reliable reading comprehension model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a reading comprehension data enhancement method based on back translation provided in Embodiment 1 of the present invention;

[0049] Figure 2 for Figure 1 A specific embodiment of step 110;

[0050] Figure 3 for Figure 1 A specific embodiment of step 120;

[0051] Figure 4 for Figure 1 A specific embodiment of step 130;

[0052] Figure 5 A flowchart of a reading comprehension data enhancement method based on back translation provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0055] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0056] Embodiment 1

[0057] This method proposes a reading comprehension data enhancement method based on back translation, which can effectively expand the scale of training data, so that a more reliable reading comprehension model can be obtained based on a limited data scale. Usually, the reading comprehension training data consists of three parts (document d, question q, answer a), and the training goal is to select the answer to the question from the document. The present application provides a data enhancement method based on back translation. First, open source Chinese-English bilingual parallel corpus is used to train a bidirectional neural machine translation model (English->Chinese, Chinese->English), and then the translation model is used to translate the documents in the reading comprehension training data from Chinese to English, and then the English translation results are translated back into Chinese. Then, based on the results of the back translation, the answer to the question is constructed. Since the translation results may have a certain gap with the original data, this can increase the diversity of the training data and enrich the expression ability of the model, thereby alleviating the problem of scarce training data resources in the field.

[0058] Figure 1 The flowchart of the reading comprehension data enhancement method based on back translation provided in the first embodiment of the present invention is shown in FIG. The execution subject of the method is a device with computing function such as a terminal and a server. Figure 1 The technical solution of the present invention is described in detail.

[0059] Step 110, training a bidirectional neural machine translation model using Chinese and English bilingual parallel corpora;

[0060] Here, taking the training of a Chinese-English machine translation model as an example, step 110 is specifically described.

[0061] like Figure 2 As shown, step 110 includes the following steps 1101-1108:

[0062] Step 1101, obtaining a Chinese-English bilingual parallel corpus pair; the Chinese-English bilingual parallel corpus pair includes a first matrix of Chinese word sequences preprocessed with word vectors and a second matrix of English word sequences preprocessed with word vectors;

[0063] Collect Chinese-English parallel corpus pairs {X,Y}. Where X = [v1,…,v n ] represents the first matrix obtained by preprocessing the input word sequence with word vectors, where v i Represents the vector of the ith subword. Y=[u1,…,u n ] represents the second matrix obtained by preprocessing the sub-word sequence of the target language input with word vectors, where u i Represents the word vector of the i-th subword in the target language.

[0064] Step 1102, segmenting the sentences in the reading comprehension training data into sub-words by using a word segmentation method to obtain a sub-word sequence;

[0065] Specifically, in order to reduce the impact of out-of-set words on generation performance, the BPE method is used to segment all documents in the reading comprehension training data into sub-words, so the input units of the encoder and decoder networks are sub-word sequences.

[0066] Step 1103, encoding the Chinese corpus in the document in the training data after the sub-word segmentation to obtain a first encoding representation of the input word sequence information;

[0067] Specifically, the preprocessed source document is input into the encoder network, and the encoder network encodes the Chinese corpus to obtain a first encoded representation of the input word sequence information.

[0068] Step 1104, encoding the input word sequence to obtain an encoded representation of each word;

[0069] In which, define x=[x1,…,x n ] represents the input subword sequence, and defines Y = [y1…,y n ] represents the subword sequence of the target language English, and defines f enc () is the encoder calculation unit. The encoding representation of each word after the encoder can be calculated by the following formula:

[0070]

[0071] in Represents the encoding representation of the i-th word sequence. Using the encoder, we can get the top-level encoding representation h N .

[0072] Step 1105, decoding the second matrix to obtain a decoded representation of the second matrix;

[0073] Step 1106, the first matrix and the second matrix are combined to obtain the output hidden state of the decoder;

[0074] In which, define f dec() is the computational unit of the target language decoder, and the output hidden state of the decoder at time i is Calculated by the following formula:

[0075]

[0076] Among them, u i represents the input of the decoder at time i, h N represents the hidden state obtained by the encoder, is the hidden state of the decoder network before the i-th moment in the n-th layer. Represents the output state of the top layer of the decoder at the i-th moment.

[0077] Step 1107, determining the probability distribution of the output hidden state of the decoder;

[0078] Specifically, the decoder output is Softmax is used to output the probability distribution of i in all vocabulary lists at each moment.

[0079]

[0080] Among them, W and b are the training parameters of the Chinese-English model, and the dimension of W is the same as the vocabulary dimension.

[0081] Step 1108, selecting the word corresponding to the maximum probability in the probability distribution as the output result of the neural machine translation model.

[0082] Specifically, the word corresponding to the maximum probability is selected as the generated result at time i:

[0083]

[0084] According to the above steps 1101-1108, the final generated result Y=[y1…,y n ].

[0085] Exchange the input and output of the translation model and repeat steps 1101-1108 to complete the training of the English-Chinese model.

[0086] Step 120, expanding the pre-annotated data of reading comprehension documents through a neural machine translation model;

[0087] See also Figure 2 , step 120 specifically includes the following steps 1201-1202:

[0088] Step 1201, translating the first Chinese in the document in the reading comprehension training data into English through the Chinese-English translation model in the trained bidirectional neural machine translation model;

[0089] Specifically, the Chinese-English translation model is used to translate each Chinese document in the reading comprehension training data into an English document. When the translation model is used, BeamSize=k, and a total of k English documents are constructed.

[0090] Step 1202: Translate English into a second Chinese language through the English translation model in the trained bidirectional neural machine translation model; the first Chinese language and the second Chinese language constitute pre-labeled data.

[0091] Specifically, the English-Chinese translation model is used to translate English documents back into Chinese documents, and BeamSize = k is taken. In this way, k original Chinese documents can be constructed. 2 The pre-annotated document data generated by the article construction.

[0092] Step 130 , constructing answers to the pre-annotated data based on the answers in the reading comprehension training data and the pre-annotated data.

[0093] Specifically, the answer a in the original text in the reading comprehension training data and the pre-annotated document d' are constructed to construct the answer a' in the pre-annotated document. Figure 4 , step 130 includes the following steps 1301-1303:

[0094] Step 1301, obtaining the first and last characters of the answer in the reading comprehension training data, and obtaining the vector representation of the first and last characters through the pre-trained word vector;

[0095] Specifically, obtain the first and last characters {s, t} of the answer a in the original text, and obtain the vector representation of the first and last characters {v s ,v t}.

[0096] Step 1302, traverse multiple characters in the pre-labeled data, obtain the first score of the first character in the word vector of each character, and the second score of the last character in the word vector of the character; the first score and the second score of each character constitute a score set of multiple characters;

[0097] Specifically, traverse the character c in the pre-annotated document d' i , calculate the character c respectively i The word vector v ci With v s ,v t The cosine similarity score {S sci ,S tci}. In this way, we get the score set {S sc ,S tc}

[0098] Step 1303, obtain the word with the highest score from the score set to obtain the answer of the pre-labeled data.

[0099] From the score set S sc and S tc Get the words s' and t' with the highest scores, and the fragment c s′ c t′ This is the constructed answer a'

[0100] Finally, the constructed reading comprehension training data (d', q', a) is obtained.

[0101] By applying the back-translation-based reading comprehension data enhancement method provided by the present invention, the data scale and data diversity can be effectively increased, and to a certain extent, the problem of poor model performance caused by data scarcity in the field can be alleviated, thereby obtaining a robust and reliable reading comprehension model.

[0102] Embodiment 2

[0103] Figure 5 The flowchart of the back-translation-based reading comprehension data enhancement method provided in the second embodiment of the present invention is as follows. The back-translation-based reading comprehension data enhancement device includes: a training module 510 , an expansion module 520 and a construction module 530 .

[0104] The training module 510 is used to train a bidirectional neural machine translation model using Chinese and English bilingual parallel corpora;

[0105] The expansion module 520 is used to expand the pre-annotated data of the reading comprehension document through the neural machine translation model;

[0106] The construction module 530 is used to construct answers to the pre-labeled data based on the answers in the reading comprehension training data and the pre-labeled data.

[0107] The training module 510 is specifically used for:

[0108] Obtaining a Chinese-English bilingual parallel corpus pair; the Chinese-English bilingual parallel corpus pair includes a first matrix of Chinese word sequences preprocessed by word vectors and a second matrix of English word sequences preprocessed by word vectors;

[0109] The sentences in the reading comprehension training data are segmented into sub-words by the word segmentation method to obtain sub-word sequences;

[0110] Encoding the Chinese corpus in the Chinese-English bilingual parallel corpus pair to obtain a first encoding representation of the input word sequence information;

[0111] Encode the input word sequence to obtain the encoded representation of each word;

[0112] Decoding the second matrix to obtain a decoded representation of the second matrix;

[0113] Substituting the first matrix and the second matrix, we get the output hidden state of the decoder;

[0114] Determine the probability distribution of the decoder's output hidden state;

[0115] The word corresponding to the maximum probability in the probability distribution is selected as the output result of the neural machine translation model.

[0116] The expansion module 520 is specifically used for:

[0117] Translate the first Chinese in the document in the reading comprehension training data into English through the Chinese-English translation model in the trained bidirectional neural machine translation model;

[0118] The English translation model in the trained bidirectional neural machine translation model is used to translate English into the second Chinese language; the first Chinese language and the second Chinese language constitute the pre-labeled data.

[0119] The construction module 530 is specifically used for:

[0120] Get the first and last characters of the answer in the reading comprehension training data, and obtain the vector representation of the first and last characters through the pre-trained word vector;

[0121] Traverse multiple characters in the pre-labeled data to obtain the first score of the first character in the first and last characters of the word vector of each character, and the second score of the last character in the first and last characters of the word vector of the character; the first score and the second score of each character constitute a score set of multiple characters;

[0122] Get the word with the highest score from the score set to get the answer for the pre-labeled data.

[0123] Among them, the bidirectional neural machine translation model includes Chinese-English model and English-Chinese model.

[0124] Embodiment 3

[0125] Embodiment 3 of the invention provides a device, including a memory and a processor, wherein the memory is used to store a program, and the memory can be connected to the processor via a bus. The memory can be a non-volatile memory, such as a hard disk drive and a flash memory, and the memory stores a software program and a device driver. The software program can perform various functions of the above method provided in the embodiment of the present invention; the device driver can be a network and interface driver. The processor is used to execute the software program, and when the software program is executed, the method provided in embodiment 1 of the present invention can be implemented.

[0126] Embodiment 4

[0127] Embodiment 4 of the present invention provides a computer program product including instructions. When the computer program product is run on a computer, the computer executes the method provided in Embodiment 1 of the present invention.

[0128] Embodiment 5

[0129] Embodiment 5 of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method provided in Embodiment 1 of the present invention is implemented.

[0130] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0131] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0132] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A reading comprehension data enhancement method based on back translation, characterized in that: The method comprises: Training a bidirectional neural machine translation model using Chinese-English bilingual parallel corpus, including: obtaining a Chinese-English bilingual parallel corpus pair; the Chinese-English bilingual parallel corpus pair includes a first matrix of Chinese word sequences preprocessed by word vectors and a second matrix of English word sequences preprocessed by word vectors; The sentences in the reading comprehension training data are segmented into sub-words by the word segmentation method to obtain sub-word sequences; Encoding the Chinese corpus in the document in the training data after the sub-word segmentation to obtain a first encoding representation of the input word sequence information; Encoding the input word sequence to obtain an encoded representation of each word; Decoding the second matrix to obtain a decoded representation of the second matrix; Substituting the first matrix and the second matrix to obtain an output hidden state of a decoder; Determine the probability distribution of the decoder's output hidden state; Selecting the word corresponding to the maximum probability in the probability distribution as the output result of the neural machine translation model; Expand the pre-annotated data of reading comprehension documents through the neural machine translation model; According to the answer in the reading comprehension training data and the pre-annotated data, constructing the answer to the pre-annotated data, including: obtaining the first and last characters of the answer in the reading comprehension training data, and obtaining the vector representation of the first and last characters through the pre-trained word vector; Traversing multiple characters in the pre-labeled data, obtaining a first score of the first character of the first and last characters for the word vector of each character, and a second score of the last character of the first and last characters for the word vector of the character; the first score and the second score of each character constitute a score set of the multiple characters; The word with the highest score is obtained from the score set to obtain the answer of the pre-labeled data.

2. The method according to claim 1, characterized in that: The method of expanding the pre-annotated data of reading comprehension documents through the neural machine translation model specifically includes: Translating the first Chinese in the document in the reading comprehension training data into English through the Chinese-English translation model in the trained bidirectional neural machine translation model; The English is translated into a second Chinese by using an English translation model in a trained bidirectional neural machine translation model; the first Chinese and the second Chinese constitute pre-labeled data.

3. The method according to claim 1, characterized in that: The bidirectional neural machine translation model includes a Chinese-English model and an English-Chinese model.

4. A reading comprehension data enhancement device based on back translation, characterized in that: The device comprises: A training module, the training module is used to train a bidirectional neural machine translation model; obtain a Chinese-English bilingual parallel corpus pair; the Chinese-English bilingual parallel corpus pair includes a first matrix of Chinese word sequences preprocessed by word vectors and a second matrix of English word sequences preprocessed by word vectors; divide sentences in reading comprehension training data into sub-words by a word segmentation method to obtain sub-word sequences; encode the Chinese corpus in the documents in the training data after sub-word segmentation to obtain a first encoded representation of input word sequence information; encode the input word sequence to obtain an encoded representation of each word; decode the second matrix to obtain a decoded representation of the second matrix; combine the first matrix and the second matrix to obtain an output hidden state of the decoder; determine the probability distribution of the output hidden state of the decoder; select the word corresponding to the maximum probability in the probability distribution as the output result of the neural machine translation model; An expansion module, the expansion module is used to expand the pre-annotated data of reading comprehension documents through the neural machine translation model; A construction module is provided, wherein the construction module is used to obtain the first and last characters of the answer in the reading comprehension training data, and obtain the vector representation of the first and last characters through the pre-trained word vector; traverse multiple characters in the pre-labeled data, and obtain the word vector of each character, which is the first score of the first character of the first and last characters, and the word vector of the character, which is the second score of the last character of the first and last characters; the first score and the second score of each character constitute a score set of the multiple characters; obtain the word with the highest score from the score set to obtain the answer of the pre-labeled data.

5. The device according to claim 4, characterized in that The expansion module is specifically used for: Translating the first Chinese in the document in the reading comprehension training data into English through the Chinese-English translation model in the trained bidirectional neural machine translation model; The English is translated into a second Chinese by using an English translation model in a trained bidirectional neural machine translation model; the first Chinese and the second Chinese constitute pre-labeled data; The word with the highest score is obtained from the score set to obtain the answer of the pre-labeled data.

6. The device according to claim 4, characterized in that The bidirectional neural machine translation model includes a Chinese-English model and an English-Chinese model.

Images