Method for expanding corpus and training method and product for translation model

Abstract

The embodiments of the present invention provide a method for expanding corpus and a training method and product for a translation model. The method comprises: connecting the source text in the parallel corpus with the corresponding target text; before or after the connection, masking the source text and / or the target text to generate a connected text containing masked words; and predicting the connected text based on a mask prediction model to obtain a predicted distribution of the masked words, wherein the predicted distribution is used to expand the parallel corpus. The method according to the embodiments of the present invention can not only expand the quantity of parallel corpus, but also ensure the quality of the expanded corpus, thereby facilitating improving the training quality of the machine translation model.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology. More specifically, the embodiments of the present invention relate to a method and device for expanding corpus, and a training method, training device, electronic device and storage medium for a machine translation model. Background Art

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The descriptions herein may include concepts that may be explored, but not necessarily concepts that have been previously thought of or explored. Therefore, unless otherwise noted herein, the content described in this section is not prior art with respect to the specification and claims of this application, and is not admitted to be prior art by inclusion in this section.

[0003] With the continuous development of artificial intelligence and machine language learning technology, the demand and application scope of machine translation technology are constantly expanding. In related technical fields, machine translation models can usually be used for machine translation. However, using machine translation models for machine translation requires a large amount of training corpus for translation training. A sufficient amount of high-quality training corpus will help improve the translation effect of machine translation models.

[0004] Furthermore, data augmentation is a relatively important method for improving the performance of deep learning models. In machine translation tasks, commonly used corpus augmentation methods include randomly selecting two words in the source sentence of the translation corpus for exchange, or randomly selecting two words in the target sentence of the translation corpus for exchange, or dropping some words, or randomly replacing some words with other words. After obtaining the augmented corpus, the augmented corpus can be used to train the translation model to improve the performance of the translation model. Summary of the invention

[0005] However, since randomly exchanging two words, dropping some words, or randomly replacing some words with other words usually changes the structure of the original sentence, the grammatical structure or semantics of the expanded sentence will change compared with the original sentence. Although such corpus expansion methods can improve the generalization of machine translation models to a certain extent, they are prone to introduce large noise and may damage the performance of the model or have limited performance improvement, which is very annoying.

[0006] Therefore, an improved corpus expansion method is needed to ensure that the expanded sentences can maintain the grammatical and semantic consistency with the original sentences.

[0007] In this context, embodiments of the present invention are intended to provide a method for expanding corpus and a training method and product for a translation model.

[0008] In a first aspect of an embodiment of the present invention, a method for expanding a corpus is provided, comprising: connecting a source text in a parallel corpus with a corresponding target text; before or after the connection, performing masking processing on the source text and / or the target text to generate a connected text containing masked words; and predicting the connected text based on a mask prediction model to obtain a predicted distribution of the masked words, wherein the predicted distribution is used to expand the parallel corpus.

[0009] In one embodiment of the present invention, connecting the source-end text with the corresponding target-end text includes: connecting the source-end text with the corresponding target-end text using a preset marker.

[0010] In another embodiment of the present invention, when masking is performed on the source text and / or the target text, the selected number of masked words satisfies at least one of the following conditions: the ratio of the number of masked words in the source text to the number of all words in the source text is less than a first preset threshold; the ratio of the number of masked words in the target text to the number of all words in the target text is less than a second preset threshold; and the ratio of the number of masked words in the connection text to the number of all words in the connection text is less than a third preset threshold.

[0011] In another embodiment of the present invention, before predicting the connected text based on the mask prediction model, the method also includes: obtaining original training corpus for translation training, wherein the original training corpus includes a source-side training sample and a target-side training sample; connecting the source-side training sample and the target-side training sample; before or after the connection, masking part of the original words in the source-side training sample and / or the target-side training sample to generate a connected training sample containing the masked sample word; and training a first prediction model using the connected training sample and the original word corresponding to the masked sample word to obtain the mask prediction model.

[0012] In a second aspect of an embodiment of the present invention, a device for expanding a corpus is provided, comprising: a connection device configured to connect a source text in a parallel corpus with a corresponding target text to generate a connected text containing a masked word; a masking device configured to perform masking processing on the source text and / or the target text before or after the connection; and a prediction device configured to predict the connected text based on a mask prediction model to obtain a predicted distribution of the masked word, wherein the predicted distribution is used to expand the parallel corpus.

[0013] In one embodiment of the present invention, the connection device is further configured to: connect the source-end text with the corresponding target-end text using a preset mark.

[0014] In another embodiment of the present invention, the masking device is further configured to select a number of masked words that satisfies at least one of the following conditions: the ratio of the number of masked words in the source text to the number of all words in the source text is less than a first preset threshold; the ratio of the number of masked words in the target text to the number of all words in the target text is less than a second preset threshold; and the ratio of the number of masked words in the connection text to the number of all words in the connection text is less than a third preset threshold.

[0015] In another embodiment of the present invention, the apparatus further includes a first training device configured to: obtain original training corpus for translation training, wherein the original training corpus includes a source-end training sample and a target-end training sample; connect the source-end training sample and the target-end training sample; before or after the connection, mask part of the original words in the source-end training sample and / or the target-end training sample to generate a connected training sample containing the masked sample word; and train a first prediction model using the connected training sample and the original word corresponding to the masked sample word to obtain the mask prediction model.

[0016] In a third aspect of the embodiments of the present invention, a training method for a machine translation model is provided, comprising: determining one or more sample words corresponding to the masked word based on the predicted distribution of the masked word obtained by using any method described in the first aspect of the embodiments of the present invention; replacing the position of the masked word in a parallel corpus with the one or more sample words to form a parallel training corpus; and training the machine translation model using the parallel training corpus.

[0017] In one embodiment of the present invention, replacing the position of the masked word in the parallel corpus with the one or more sampled words includes: replacing the position of the masked word in the parallel corpus with a word vector corresponding to a sampled word; or replacing the position of the masked word in the parallel corpus with a weighted average of word vectors corresponding to multiple sampled words.

[0018] In a fourth aspect of the embodiments of the present invention, a training device for a machine translation model is provided, comprising a sampling device configured to determine one or more sampled words corresponding to the masked word based on the predicted distribution of the masked word obtained by using any method described in the first aspect of the embodiments of the present invention; a corpus forming device configured to replace the position of the masked word in a parallel corpus with the one or more sampled words to form a parallel training corpus; and a second training device configured to train the machine translation model using the parallel training corpus.

[0019] In one embodiment of the present invention, the corpus forming device is further configured to: replace the position of the masked word in the parallel corpus with a word vector corresponding to a sampled word; or replace the position of the masked word in the parallel corpus with a weighted average of word vectors corresponding to multiple sampled words.

[0020] In a fifth aspect of an embodiment of the present invention, an electronic device is provided, comprising at least one processor; and a memory storing program instructions, wherein when the program instructions are executed by the at least one processor, the electronic device executes a method according to any one of the first aspect of the embodiment of the present invention and / or any one of the third aspect of the embodiment of the present invention.

[0021] In a sixth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, which stores a program for expanding a corpus and / or a program for training a machine translation model. When the program is executed by a processor, the method described in any one of the first aspect of the embodiments of the present invention and / or any one of the third aspect of the embodiments of the present invention is executed.

[0022] According to the method for expanding corpus in the embodiment of the present invention, the source text in the parallel corpus can be connected with the corresponding target text, so that when predicting the connected text based on the mask prediction model, the source text and the corresponding target text can be taken into account to obtain the predicted distribution of masked words that can keep the source text and the corresponding target text fluent and coherent, and consistent in grammar and semantics. The parallel corpus expanded in this way can not only expand the number of parallel corpora, but also ensure the quality of the expanded corpus, which is conducive to improving the training quality of the machine translation model.

[0023] In some embodiments, by using a preset mark for connection, it can help the mask prediction model to distinguish and identify the source text and the corresponding target text. In other embodiments, by setting the first preset threshold, the second preset threshold, and the third preset threshold, the number of masked words can be limited, which is conducive to ensuring the prediction quality of masked words. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0025] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0026] Figure 2 The flowchart of the method for expanding corpus according to an embodiment of the present invention is schematically shown;

[0027] Figure 3 A schematic diagram of a process of predicting a connection text based on a mask prediction model according to an embodiment of the present invention is schematically shown;

[0028] Figure 4 A flowchart of a method for obtaining a mask prediction model according to an embodiment of the present invention is schematically shown;

[0029] Figure 5 The following schematically shows a schematic diagram of a device for expanding corpus according to an embodiment of the present invention;

[0030] Figure 6 The flowchart of the training method for a machine translation model according to an embodiment of the present invention is schematically shown;

[0031] Figure 7 A schematic diagram schematically shows a process diagram of a training method for a machine translation model according to an embodiment of the present invention; and

[0032] Figure 8 A schematic diagram of a training device for a machine translation model according to an embodiment of the present invention is schematically shown.

[0033] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0034] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0035] Figure 1 Schematically shows a block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention. Figure 1As shown, the computing system 100 may include: a central processing unit (CPU) 101, a random access memory (RAM) 102, a read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, the RAM 102, the ROM 103, the hard disk controller 105, the keyboard controller 106, the serial controller 107, the parallel controller 108, and the display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that Figure 1 The structural block diagram is only for the purpose of illustration, rather than limiting the scope of the present invention. In some cases, some equipment may be added or reduced according to specific circumstances.

[0036] It is known to those skilled in the art that the embodiments of the present invention may be implemented as a device, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit", "module" or "device". In addition, in some embodiments, the present invention may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0037] In a sixth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, which stores a program for expanding a corpus and / or a program for training a machine translation model. When the program is run by a processor, the method according to any one of the first aspects of the embodiments of the present invention and / or the training method according to any one of the third aspects of the embodiments of the present invention are executed.

[0038] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0039] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0040] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0041] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0042] The following will describe the implementation of the present invention with reference to the flowchart of the method of the embodiment of the present invention and the block diagram of the device. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby producing a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the function / operation specified in the box in the flowchart and / or block diagram.

[0043] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0044] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0045] According to an embodiment of the present invention, a method and device for expanding corpus, a training method and training device for a machine translation model, and an electronic device and a computer-readable storage medium are proposed.

[0046] In this article, it is to be understood that the terms involved include the following:

[0047] MT: Machine Translation, commonly known as machine translation, belongs to the category of computational linguistics, and uses computer technology to translate text or speech from one natural language into another.

[0048] NMT: Neural Machine Translation, also known as neural network machine translation, is a technology that uses neural network technology for machine translation.

[0049] ANN: Artificial Neural Network, also known as Neural Network (NN) or quasi-neural network, is a mathematical model or computational model that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) in the field of machine learning and cognitive science, and is used to estimate or approximate functions.

[0050] MLM: Masked Language Model, a masked language model, which was proposed by BERT and is used to mask some words in a sequence and replace them with [MASK] so as to predict the masked words using the masked sequence. For example, given the sequence x=(x1, x2, x3, x4, x5), masking x2 and x4 respectively, the masked sequence (x1, [MASK], x3, [MASK], x5) is obtained. Based on the masked sequence, the MLM model can be prompted to learn good language representations to predict x2 and x4.

[0051] BERT: Bidirectional Encoder Representations from Transformers, bidirectional encoder representations based on the translation model.

[0052] LM: language model, which is used to predict the probability of a sentence (or a sequence of words) or the probability of the next word in a sequence. It predicts the probability of the word to be predicted by referring to the words before the word to be predicted in the sentence.

[0053] Parallel corpus is a pair of contrastive texts that mutually interpret each other in multiple languages. For example, a Chinese-English parallel corpus can include Chinese texts and their English interpretation texts.

[0054] In addition, the quantity of any element in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0055] Next, with reference to several representative embodiments of the present invention, the principles and spirit of the present invention will be elaborated in detail. SUMMARY OF THE INVENTION

[0057] The inventor of the present invention found that another method of data augmentation in the field of natural language processing is to use a language model to predict some words in the training corpus sentences and sample from the obtained prediction distribution to replace the corresponding words in the sentences with the sampled words. However, due to the characteristic that using a language model for prediction is unidirectional (that is, the language model predicts based on the words before the word to be predicted in the sentence), using the sampled words from the prediction distribution of the language model to replace the corresponding words in the sentence may result in a phenomenon of being coherent only with the left-side statement but not with the right-side statement. For example, for the original sentence "I have an apple", when using a language model to predict the word "个", if only considering the left part (i.e., "I have a"), augmented corpora such as "I have a strip of apple" and "I have a handful of apple" may appear.

[0058] Furthermore, the present inventor also found that current data augmentation methods can only independently augment the source text or the target text, which may lead to the problem that the meaning of the augmented source text is inconsistent with that of the target text, thereby affecting the training effect. For example, if the source sentence is "我有一个苹果" (I have an apple), and the target sentence is "I have an apple", when predicting the word "苹果" (apple) in the source sentence using a language model, if only the left part (i.e., "我有一个" (I have an)) is considered, augmented corpora such as "我有一个香蕉" (I have a banana) and "我有一个桃子" (I have a peach) may appear, which is different from the meaning of the target sentence. Obviously, training a machine translation model with such augmented corpora may affect the translation accuracy of the trained translation model.

[0059] In order to ensure that the semantic and syntactic structures of the sentence are not damaged, and at the same time ensure the bidirectional coherence of the sentence and the meaning consistency between the source text and the target text, the present inventor found that when predicting a word in the source text, the corresponding target text can be referred to. Similarly, when predicting a word in the target text, the corresponding source text can be referred to. This can effectively avoid damaging the syntactic and semantic structures of the augmented sentence and can obtain augmented corpora that are bidirectionally coherent and have consistent meanings. Taking the previous example where the source sentence is "我有一个苹果" (I have an apple) and the target sentence is "I have an apple", when predicting the word "个" (a), if both directions are considered (i.e., referring to the words before and after "个"), and at the same time referring to the corresponding interpreted word "an" in the target sentence, a more appropriate augmented corpus should be "我有一颗苹果" (I have an apple), achieving bidirectional coherence of the sentence; when predicting the word "苹果" (apple), since the corresponding interpreted word "apple" in the target sentence is considered, augmented corpora that are significantly different from the meaning of the target sentence such as "我有一个香蕉" (I have a banana) and "我有一个桃子" (I have a peach) will not appear.

[0060] After introducing the basic principle of the present invention, the various non-limiting embodiments of the present invention will be specifically introduced below.

[0061] Application Scenario Overview

[0062] The method for augmenting corpora according to the embodiments of the present invention can be implemented by an application program running on a machine. The method for augmenting corpora according to the embodiments of the present invention can be applied to the augmentation of parallel corpora in any language, such as Chinese-English bilingual corpora, Chinese-Japanese bilingual corpora, English-French bilingual corpora, Japanese-Korean bilingual corpora, etc.

[0063] Furthermore, the augmented corpora obtained by using the method for augmenting corpora according to the embodiments of the present invention can be applied not only to the training of machine translation models, but also to, for example, the training of grammar correction models.

[0064] Exemplary Methods

[0065] Reference below Figure 2 The method for expanding corpus according to an exemplary embodiment of the present invention is described. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present invention, and the embodiments of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0066] like Figure 2 As shown, the method 200 for expanding the corpus can include: in step 201, the source text in the parallel corpus can be connected with the corresponding target text. The source text and the corresponding target text can be a contrasting text pair that interprets each other in the parallel corpus. Corresponding here can be understood as being able to interpret each other. The source text and the corresponding target text can be texts of different languages, wherein the source text can be selected from the text of any language in English, French, German, Spanish, Chinese, Japanese, Korean, etc.: the target text can be selected from the text of the language different from the source text in Chinese, English, French, German, Spanish, Japanese, Korean, etc.

[0067] The connection described above may include a direct connection or an indirect connection. For example, in some embodiments, a preset mark may be used to connect the source text with the corresponding target text. The preset mark may serve as a connection so that the source text is connected in series with the corresponding target text, or may serve as a separation so that the mask prediction model can distinguish and identify the source text and the corresponding target text. The preset mark may include characters, symbols, and the like.

[0068] Next, in step 202, the source-end text and / or the target-end text may be subjected to masking processing before or after the connection to generate a connection text containing masked words. Masking processing of the source-end text and / or the target-end text may include masking processing of only the source-end text, or masking processing of only the target-end text, or masking processing of both the source-end text and the target-end text. Masking processing may include masking one or more source-end words in the source-end text and / or one or more target-end words in the target-end text, for example, the one or more source-end words and / or the target-end words may be replaced with specific symbols or specific words to achieve masking. The words obtained after masking are masked words.

[0069] The connection text described above may include one or more masking words. In some embodiments, the source end words and / or target end words to be masked may be randomly selected. In other embodiments, the source end words and / or target end words to be masked may be selected in sequence, for example, the masking process may be performed in sequence according to the order of the source end words in the source end text, and / or according to the order of the target end words in the target end text, each masking process may generate a connection text, and multiple masking processes may generate multiple connection texts.

[0070] In another embodiment of the present invention, when masking the source text and / or the target text, the number of selected masked words can meet at least one of the following conditions: the proportion of the number of masked words in the source text to the number of all words in the source text is less than a first preset threshold; the proportion of the number of masked words in the target text to the number of all words in the target text is less than a second preset threshold; and the proportion of the number of masked words in the connection text to the number of all words in the connection text is less than a third preset threshold. The first preset threshold, the second preset threshold, and the third preset threshold can be set as needed. In some embodiments, the first preset threshold or the second preset threshold can be 15%. In other embodiments, the third preset threshold can be 15% to 30%. The setting of the first preset threshold, the second preset threshold, and the third preset threshold can limit the number of masked words to avoid the situation where too many masked words are included in a connection text, resulting in a decrease in prediction accuracy.

[0071] In another embodiment of the present invention, when masking is performed on both the source text and the target text, the masked words in the source text and the masked words in the target text can be made into non-parallel corpora. The non-parallel corpora here refer to that the masked words in the source text and the masked words in the target text cannot be selected from words that are mutually interpreted positions. According to such a setting, it can be ensured that the masked words in the source text have corresponding interpretations in the target text, and that the masked words in the target text also have corresponding interpretations in the source text, so as to ensure that there is a reliable prediction basis for predicting the masked words, so that the expanded corpus can maintain semantic consistency with the corpus before expansion.

[0072] Then, as further shown in the figure, the process can proceed to step 203, and the connection text can be predicted based on the mask prediction model to obtain the predicted distribution of the masked words, wherein the predicted distribution can be used to expand the parallel corpus. Predicting the connection text based on the mask prediction model can be inputting the connection text into the mask prediction model for prediction. In some embodiments, the mask prediction model may include a conditional masked language model ("Conditional Masked Language Model", which can be abbreviated as CMLM), which can have the same or similar structure as the masked language model. The difference between the conditional masked language model and the masked language model is that the conditional masked language model can process the connection text after training and learning.

[0073] The predicted distribution of masked words described above may include the probability distribution of masked words, that is, the probability of one or more predicted words corresponding to the masked words. The probability distribution can be used to characterize the similarity between the predicted words and the source end words and / or the target end words corresponding to the masked word positions. In some application scenarios, one or more predicted words can be selected to replace the masked words according to the probability of the predicted words to form a new parallel corpus, thereby expanding the original parallel corpus.

[0074] Combination of the above Figure 2 The method for expanding the corpus according to the embodiment of the present invention is described exemplarily. The execution order of the steps shown in the figure is exemplary and not restrictive. For example, step 202 may not be limited to being executed after step 201 in the figure. When masking is performed before connection, step 202 may be executed before step 201. In order to facilitate understanding of the process of obtaining the predicted distribution, the following will be combined with Figure 3 Provide explanation.

[0075] Figure 3 The following schematic diagram shows a process of predicting a connection text based on a mask prediction model according to an embodiment of the present invention. Figure 3 The process shown in can be combined with the previous Figure 2 The step 203 is a visual representation of the above. Figure 2 The description can also be applied to Figure 3 Description.

[0076] like Figure 3 As shown in n Represents the source word in the source text; y1, y2…y nRepresents the target word in the target text. Exemplarily, masking x2 and x4 in the source text and connecting the source text and the target text can generate a connection text 310 (shown in the dotted box) containing a mask word (such as [MASK] in the diagram), wherein the source text and the target text can be connected by a preset mark (such as [SEP] in the diagram) 311. Then, the connection text 310 can be input into the mask prediction model 320 to predict the source word at the position of the mask word [MASK], and the predicted distributions P(x2) and P(x4) of the two mask words [MASK] in the diagram can be obtained.

[0077] Understandably, Figure 3 What is shown in is exemplary and non-restrictive. For example, it may not be limited to masking only the source text as shown in the figure. In other embodiments, masking may be performed only on the target text, and the predicted distribution of the target words at the masked word position may be obtained based on the mask prediction model. In some other embodiments, both the source text and the target text may be masked, and the predicted distribution of the source words and the target words at the masked word position may be obtained based on the mask prediction model. For another example, the mask prediction model may not be limited to outputting only the predicted distribution of masked words as shown in the figure. In some other embodiments, unmasked words may also be output to generate source text, or target text, or connected text containing the predicted distribution.

[0078] Furthermore, in order to obtain a better prediction distribution and improve the performance of the mask prediction model, in some embodiments, before predicting the connection text based on the mask prediction model, the first prediction model may be trained to obtain the mask prediction model. Figure 4 An exemplary description is given.

[0079] Figure 4 The flowchart of the method for obtaining the mask prediction model according to an embodiment of the present invention is schematically shown. Figure 4 As shown in , method 400 may include: in step 401, original training corpus for translation training may be obtained, wherein the original training corpus may include source training samples and target training samples. The original training corpus may be a known corpus for training an existing translation model. The source training samples and the target training samples may be parallel corpora in different languages ​​that paraphrase each other.

[0080] Next, in step 402, the source end training sample and the target end training sample can be connected. The connection method of step 402 is combined with the above Figure 2 The connection method described in step 201 is the same or similar and will not be repeated here.

[0081] Then, the process can proceed to step 403, and masking processing can be performed on part of the original words in the source training sample and / or the target training sample before or after the connection to generate a connection training sample containing masked sample words. The original word is a word that originally existed in the source training sample and / or the target training sample. The part of the original word can include one original word or multiple original words. The word obtained after masking processing the original word is the masked sample word. The masking processing method is combined with the above Figure 2 The steps are the same or similar to those described in step 202 and are not described in detail here.

[0082] like Figure 4 As further shown in, in step 404, the first prediction model can be trained using the connection training samples and the original words corresponding to the masked sample words to obtain the mask prediction model. The original words corresponding to the masked sample words are the original words at the position of the masked sample words. In some embodiments, the connection training samples can be input into the first prediction model, and the original words corresponding to the masked sample words are used as supervisory signals for training the first prediction model. In other embodiments, the source-side training samples and / or the target-side training samples before masking processing can be used as supervisory signals for training the first prediction model.

[0083] In some other embodiments, the first prediction model may be constructed based on a masked language model. The first prediction model may include a conditional masked language model. By training the first prediction model using the connection training sample and the original word corresponding to the masked sample word, the parameters and weights of the first prediction model may be continuously optimized. After training the first prediction model in step 404, a trained first prediction model may be obtained, and the trained first prediction model may be referred to as a masked prediction model.

[0084] Combination of the above Figure 4 A method for training a first prediction model according to an embodiment of the present invention is described. It can be understood that by training the first prediction model using connection training samples, the first prediction model can be trained to learn to bidirectionally consider the source training samples and the target training samples in the connection training samples. In other embodiments, the source training samples and the target training samples can be connected using preset tags to generate connection training samples containing preset tags. By training the first prediction model using such connection training samples, the first prediction model can learn the meaning of the preset tags so that the first prediction model can recognize that the preset tags connect two texts that interpret each other.

[0085] Exemplary Devices

[0086] After introducing the method of the exemplary embodiment of the present invention, next, refer to Figure 5 A device for expanding corpus according to an exemplary embodiment of the present invention is described.

[0087] Figure 5 The following schematically shows a schematic diagram of a device for expanding corpus according to an embodiment of the present invention. Figure 5 As shown in the figure, the device 500 for expanding the corpus may include: a connecting device 501, which can be configured to connect the source text in the parallel corpus with the corresponding target text to generate a connected text containing masked words; a masking device 502, which can be configured to mask the source text and / or the target text before or after the connection; and a prediction device 503, which can be configured to predict the connected text based on a mask prediction model to obtain a predicted distribution of masked words, wherein the predicted distribution is used to expand the parallel corpus.

[0088] In one embodiment of the present invention, the connection device 501 may also be configured to: connect the source-end text with the corresponding target-end text using a preset tag.

[0089] In another embodiment of the present invention, the masking device 502 can also be configured to select a number of masked words that satisfies at least one of the following conditions: the ratio of the number of masked words in the source text to the number of all words in the source text is less than a first preset threshold; the ratio of the number of masked words in the target text to the number of all words in the target text is less than a second preset threshold; and the ratio of the number of masked words in the connected text to the number of all words in the connected text is less than a third preset threshold.

[0090] In another embodiment of the present invention, the device 500 may also include a first training device, which may be configured to: obtain original training corpus for translation training, wherein the original training corpus includes a source-side training sample and a target-side training sample; connect the source-side training sample and the target-side training sample; before or after the connection, mask part of the original words in the source-side training sample and / or the target-side training sample to generate a connected training sample containing the masked sample word; and train the first prediction model using the connected training sample and the original word corresponding to the masked sample word to obtain a mask prediction model.

[0091] Understandably, Figure 5 The functionality of the device 500 shown has been previously described. Figure 2-Figure 4 The method shown has been described and explained in detail and will not be repeated here.

[0092] Exemplary Training Methods

[0093] After introducing the method and device for expanding corpus according to the exemplary embodiment of the present invention, next, reference is made to Figure 6 A training method according to an exemplary embodiment of the present invention is described.

[0094] Figure 6 The flowchart of the training method for a machine translation model according to an embodiment of the present invention is schematically shown. Figure 6 As shown in FIG. 6 , a training method 600 for a machine translation model may include: in step 601, the training method 600 may include: Figure 2-Figure 4 The predicted distribution of masked words obtained by any of the methods described in is used to determine one or more sample words corresponding to the masked words. The predicted distribution of each masked word may include the probabilities of one or more predicted words, and the sample words may be selected from the predicted words in the predicted distribution. In one embodiment, the predicted distribution of the masked word includes only the probability of one predicted word, and the one predicted word may be selected as the sample word corresponding to the masked word. In other embodiments, the predicted distribution of the masked word may include the probabilities of multiple predicted words, and one or more of the multiple predicted words may be selected as the sample words, and the principle of selection may include random selection from multiple predicted words, or selection based on the probability of multiple predicted words. For example, one or more of the multiple predicted words with a larger probability may be selected as the sample words.

[0095] Next, in step 602, one or more sample words may be used to replace the position of the masked word in the parallel corpus to form a parallel training corpus. In the parallel corpus, the position of the unmasked word remains unchanged, and one or more sample words may be used to replace the position of the masked word to form a parallel training corpus including the sample words.

[0096] In one embodiment of the present invention, replacing the position of the masked word in the parallel corpus with one or more sampled words may include: replacing the position of the masked word in the parallel corpus with a word vector corresponding to a sampled word; or replacing the position of the masked word in the parallel corpus with a weighted average of the word vectors corresponding to multiple sampled words. The word vector described herein may be a vector of a word (or phrase) mapped to a real number, which involves mathematical embedding from a one-dimensional space of a word to a continuous vector space with a lower dimension. In some embodiments, the word vector corresponding to a word (such as the sampled word in this article) may be obtained by querying a word table database including words and their corresponding vectors. In other embodiments, the weighted average of the word vectors corresponding to multiple sampled words may be obtained in the following manner: determining the weights of the word vectors corresponding to multiple sampled words according to the probabilities of multiple sampled words; and performing a weighted average operation on the word vectors corresponding to multiple sampled words according to the weights of the word vectors corresponding to multiple sampled words.

[0097] Then, the process can proceed to step 603, and the machine translation model can be trained using the parallel training corpus. In step 603, the word vectors corresponding to the unmasked words in the parallel training corpus can be directly input into the machine translation model, and the position of the masked word can be input using the word vector corresponding to the sampled word described in step 602 or the weighted average of multiple word vectors. In some embodiments, the machine translation model can adopt a neural machine translation model. The network structure of the commonly used neural machine translation model may include an encoder and a decoder, wherein the encoder is used to convert the text of an input language into a series of context vectors that can express the characteristics of the input text, and the decoder is used to receive the output of the encoder and output a corresponding text sequence in another language.

[0098] In order to better understand the training method for the machine translation model according to the embodiment of the present invention, Figure 7 The visualization training process shown is further described as an example.

[0099] Figure 7 The following is a schematic diagram of a process of a training method for a machine translation model according to an embodiment of the present invention. Figure 7 The process shown is a combination of Figure 6 A visual representation of the training method described above, so the previous article combines Figure 6 The method 600 described may also be applied to Figure 7 in the description.

[0100] like Figure 7 As shown in n Represents the source word in the source text; y1, y2…y n Represents the target word in the target text. Exemplarily, masking x2 and x4 in the source text and connecting the source text and the target text can generate a connection text 710 (shown in the dotted box) containing a mask word (such as [MASK] in the diagram), wherein the source text and the target text can be connected by a preset mark (such as [SEP] in the diagram) 711. Then, the connection text 710 can be input into the mask prediction model 720 to predict the source word at the position of the mask word [MASK], and the predicted distributions P(x2) and P(x4) of the two mask words [MASK] in the diagram can be obtained.

[0101] like Figure 7As further shown in, when training the machine translation model, the unmasked words or corresponding word vectors in the source text can be directly input into the embedding layer ("Embedding") 730, and for the masked words in the source text, the determined sampled words or the corresponding word vectors of the sampled words can be input into the embedding layer 730 shown in the figure according to the predicted distribution P(x2) and P(x4) of the masked words. The embedding layer 730 can be used to convert discrete variables into continuous vector representations. Then, the word vector sequence processed by the embedding layer 730 can be input into the encoder 740 of the machine translation model for encoding operations and other subsequent operations of the machine translation model (not shown in the figure). In this embodiment, the parallel training corpus consists of a source text containing sampled words (or word vectors corresponding to sampled words) and a target text that does not contain masked words, wherein the target text that does not contain masked words can be used as a supervisory signal for training the machine translation model.

[0102] It should be noted that Figure 7 The training process is demonstrated by taking the example of obtaining an extended corpus by masking the source text. Those skilled in the art can understand that the training process is similar by masking the target text, or by masking both the source text and the target text to obtain an extended corpus (i.e., the parallel training corpus in this article), which will not be described in detail here.

[0103] Exemplary Training Equipment

[0104] After introducing the training method of the exemplary embodiment of the present invention, next, refer to Figure 8 A training device for a machine translation model according to an exemplary embodiment of the present invention is described.

[0105] Figure 8 The following schematically shows a schematic diagram of a training device for a machine translation model according to an embodiment of the present invention. Figure 8 As shown in the figure, a training device 800 for a machine translation model may include: a sampling device 801, which may be configured to determine one or more sampled words corresponding to a masked word according to the predicted distribution of the masked word obtained by using any method described in the first aspect of the embodiments of the present invention; a corpus forming device 802, which may be configured to replace the position of the masked word in the parallel corpus with one or more sampled words to form a parallel training corpus; and a second training device 803, which may be configured to train the machine translation model using the parallel training corpus.

[0106] In one embodiment of the present invention, the corpus forming device 802 can also be configured to: replace the position of the masked word in the parallel corpus with a word vector corresponding to a sampled word; or replace the position of the masked word in the parallel training corpus with a weighted average of the word vectors corresponding to multiple sampled words.

[0107] Understandably, Figure 8 The functions of the training device 800 shown have been previously described. Figure 6 and Figure 7 The training method shown is described and explained in detail and will not be repeated here.

[0108] Through the above description of the technical solutions and multiple embodiments of the method for expanding the corpus and the method for training a machine translation model according to the present invention, it can be understood that the method for expanding the corpus according to the embodiment of the present invention can be predicted by inputting the connected text containing the masked word into the mask prediction model, so that the obtained predicted distribution of the masked word can be closer to the meaning of the masked word before masking, thereby ensuring that the expanded parallel training corpus can maintain the bidirectional coherence of the corpus and the complete and correct semantic and grammatical structure, etc., so as to effectively solve the problems that may exist in the traditional data expansion method, such as the context independence of the expanded corpus, the destruction of the grammatical structure, and the inconsistency of the meaning between the source and the target.

[0109] Furthermore, according to the training method for a machine translation model according to an embodiment of the present invention, by introducing the method for expanding the corpus according to an embodiment of the present invention, not only the amount of training corpus can be increased, but also the quality of the expanded training corpus can be effectively guaranteed. Therefore, training the machine translation model using the corpus expanded by the connected text is beneficial to improving the performance and translation quality of the machine translation model.

[0110] In some embodiments, when training a machine translation model, the weighted average of the word vectors corresponding to multiple sampled words can be used to replace the position of the masked word in the parallel corpus. Such a setting does not require the determination of a specific sampled word, and the use of a weighted average can take into account the probability distribution of multiple sampled words, which can not only avoid the possibility that the sampled words with a lower probability will affect the quality of the expanded corpus, but also avoid the situation where the sampled word is the original word corresponding to the masked word. Therefore, this embodiment provides a new corpus expansion method, and in some application scenarios, it can obtain a higher degree of similarity to the original word to further improve the quality of the expanded training corpus.

[0111] It should be noted that although several devices or sub-devices of the device are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided into multiple devices to be embodied.

[0112] In addition, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0113] The use of the verbs "comprise", "include" and their conjugations mentioned in the application documents does not exclude the presence of elements or steps other than those recorded in the application documents. The article "a" or "an" before an element does not exclude the presence of a plurality of such elements.

[0114] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and this division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A method for expanding a corpus, comprising: Connecting source text in the parallel corpus with corresponding target text using preset tags, wherein the preset tags include characters and symbols; Before or after the connection, masking the source-end text and / or the target-end text to generate a connection text containing masked words, wherein the ratio of the number of masked words in the connection text to the number of all words in the connection text is less than a third preset threshold, and the third preset threshold is 15% to 30%; and Predicting the connected text based on a mask prediction model to obtain a predicted distribution of the masked words, wherein the predicted distribution is used to expand the parallel corpus; Among them, when the source text and the target text are masked, the masked words in the source text and the masked words in the target text are made into non-parallel corpus; the non-parallel corpus means that the masked words in the source text and the masked words in the target text cannot be selected from words that are mutually paraphrasing positions.

2. According to the method of claim 1, the selected number of masked words also satisfies at least one of the following conditions: The ratio of the number of masked words in the source text to the number of all words in the source text is less than a first preset threshold; The ratio of the number of masked words in the target-side text to the number of all words in the target-side text is less than a second preset threshold.

3. The method according to claim 1, wherein: Before predicting the connection text based on the mask prediction model, the method further includes: Acquire original training corpus for translation training, wherein the original training corpus includes source-end training samples and target-end training samples; Connecting the source end training sample and the target end training sample; Before or after the connection, masking some original words in the source-end training sample and / or the target-end training sample to generate a connection training sample containing masked sample words; and The first prediction model is trained using the connection training samples and the original words corresponding to the masked sample words to obtain the mask prediction model.

4. A device for expanding corpus, comprising: A connection device configured to connect the source-side text in the parallel corpus with the corresponding target-side text using preset tags to generate a connected text containing masked words, wherein the preset tags include characters and symbols; a masking device configured to mask the source-end text and / or the target-end text before or after connection to generate a connection text containing masked words, wherein the ratio of the number of masked words in the connection text to the number of all words in the connection text is less than a third preset threshold, and the third preset threshold is 15% to 30%; and A prediction device configured to predict the connected text based on a mask prediction model to obtain a predicted distribution of the masked words, wherein the predicted distribution is used to expand the parallel corpus; Among them, when the source text and the target text are masked, the masked words in the source text and the masked words in the target text are made into non-parallel corpus; the non-parallel corpus means that the masked words in the source text and the masked words in the target text cannot be selected from words that are mutually paraphrasing positions.

5. The device according to claim 4, wherein the masking device is further configured to select the number of masked words to satisfy at least one of the following conditions: The ratio of the number of masked words in the source text to the number of all words in the source text is less than a first preset threshold; The ratio of the number of masked words in the target-side text to the number of all words in the target-side text is less than a second preset threshold.

6. The apparatus according to claim 4, further comprising a first training device configured to: Acquire original training corpus for translation training, wherein the original training corpus includes source-end training samples and target-end training samples; Connecting the source end training sample and the target end training sample; Before or after the connection, masking some original words in the source-end training sample and / or the target-end training sample to generate a connection training sample containing masked sample words; as well as The first prediction model is trained using the connection training samples and the original words corresponding to the masked sample words to obtain the mask prediction model.

7. A training method for a machine translation model, comprising: Determine one or more sample words corresponding to the masked word according to the predicted distribution of the masked word obtained by using the method described in any one of claims 1 to 3; Replacing the position of the masked word in the parallel corpus with the one or more sampled words to form a parallel training corpus; as well as The machine translation model is trained using the parallel training corpus.

8. The training method according to claim 7, wherein replacing the position of the masked word in the parallel corpus with the one or more sample words comprises: Replace the position of the masked word in the parallel corpus with a word vector corresponding to a sampled word; or The position of the masked word in the parallel corpus is replaced by a weighted average of word vectors corresponding to a plurality of sampled words.

9. A training device for a machine translation model, comprising: A sampling device configured to determine one or more sample words corresponding to the masked word according to the predicted distribution of the masked word obtained by using the method according to any one of claims 1 to 3; A corpus forming device configured to replace the position of the masked word in the parallel corpus with the one or more sampled words to form a parallel training corpus; as well as The second training device is configured to train the machine translation model using the parallel training corpus.

10. The training device according to claim 9, wherein the corpus forming device is further configured to: Replace the position of the masked word in the parallel corpus with a word vector corresponding to a sampled word; or The position of the masked word in the parallel corpus is replaced by a weighted average of word vectors corresponding to a plurality of sampled words.

11. An electronic device comprising: at least one processor; A memory storing program instructions, which, when executed by the at least one processor, enables the electronic device to perform the method according to any one of claims 1-3 and / or any one of claims 7-8.

12. A computer-readable storage medium storing a program for expanding a corpus and / or a program for training a machine translation model, wherein when the program is executed by a processor, the method according to any one of claims 1 to 3 and / or any one of claims 7 to 8 is executed.

Images