Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement

Abstract

A Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement, Mongolian case component analysis and annotation are performed in combination with Mongolian grammatical rules; different masking methods are used to perform mask data enhancement on Mongolian case components, and masked words are predicted based on the source and target ends; semantic information of Chinese and Mongolian after data enhancement is mapped in the same vector space through a semantic encoder, and an adjacent semantic region containing vector representations of multiple Mongolian and Chinese with similar semantics is generated; samples are sampled in this region, and decoded using a decoder integrated into a translation model using a broadcast network; the translation model uses a Transformer‑XL model, and when decoding, the output of each decoder module is dynamically combined; the original sample is trained together with the sample after data enhancement, and the trained translation model is used for Mongolian-Chinese machine translation. The present invention can expand the scale of training corpus and improve translation quality.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence technology, relates to language machine translation, and particularly to a Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement. Background Art

[0002] Given a source language, the process of converting a source language into a target language using a model or algorithm is called machine translation. Machine translation is a comprehensive subject involving multiple disciplines and is also a very important research task in the field of natural language processing. Deep neural machine translation is essentially training a large neural network consisting of tens of thousands of neurons. In order to fully learn the weight values ​​of the network, neural machine translation requires high-quality corpus as training data. The higher the quality of the corpus, the better the training effect. However, for resource-scarce languages ​​such as Mongolian, it is very difficult to obtain sufficient high-quality training corpus. The existing Mongolian-Chinese neural machine translation has problems such as small scale of training corpus and uneven quality, which makes the generalization ability of Mongolian-Chinese machine translation model very weak. Summary of the invention

[0003] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement, which is mainly used to solve the problems of poor translation quality caused by the lack of parallel corpus in the Mongolian-Chinese translation process and long-distance dependency in the training process. This method can expand the scale of training corpus and improve the quality of translation.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement includes the following steps:

[0006] Step 1, combining Mongolian grammatical rules to perform case component analysis and case component labeling in Mongolian;

[0007] Step 2, perform mask data enhancement on the grid components;

[0008] Step 3, mapping the semantic information of Chinese and Mongolian after data enhancement in step 2 into the same vector space through a semantic encoder, and generating an adjacent semantic region; the adjacent semantic region contains vector representations of multiple Mongolian and Chinese with similar semantics;

[0009] Step 4, sampling samples in the adjacent semantic region, and performing a decoding operation using a decoder integrated into the translation model using the broadcast network;

[0010] Step 5: The translation model uses the Transformer-XL model, which dynamically combines the outputs of each decoder module during the decoding operation.

[0011] Step 6: Train the original samples together with the data-augmented samples, and use the trained translation model to perform Mongolian-Chinese machine translation.

[0012] In one embodiment, the step 2 uses the pre-trained model BERT as the initial case component masking model, splices source Mongolian and target Chinese sentences during training, masks the case components of the source Mongolian using three methods: random masking, partial fixed masking, and full masking, and trains to obtain a Mongolian case component masking language model that predicts masked case components; wherein the random masking is to randomly mask the case components of Mongolian, and determine the optimal masking range in the experiment; the partial fixed masking is to mask using a fixed collocation probability table of fixed phrases or phrases in the word embedding stage; and the full masking is to mask all case components in an input sentence.

[0013] In one embodiment, a soft word replacement method is used for the candidate words generated when the lattice component mask model is predicted, and the masked words are replaced with probability distribution vectors, so that the lattice component mask model predicts multiple possible words, and the soft word w is taken as a candidate word. The soft word replacement method is expressed as follows:

[0014]

[0015] Among them, e w is the embedding of soft word w, E W Representing the embedding vector of soft word w, the distribution of p(w) is defined by the probability formula The decision,p(w) is based on the context and covers the entire vocabulary,E j is the embedding matrix of the jth word, P j (w) is the probability distribution of the jth word, and V is the number of words.

[0016] In one embodiment, in step 3, the pre-trained model BERT is used as a semantic encoder, in which a semantic encoder forward function parameterized by θ′ is defined. To map discrete sentences into continuous vector representations; the semantic encoder converts the source sentence x and the target sentence y into real-valued vectors Thereby, the semantic information of Mongolian and Chinese is mapped in the same vector space.

[0017] In one embodiment, a contrastive learning method is used to optimize the vector space so that each vector generates a semantically similar region in a continuous space, namely, the adjacent semantic region, which contains a variant of the same semantic representation as the source sentence.

[0018] In one embodiment, the adjacent semantic regions are defined as and The union of two closed spheres centered at is, and the radius of the two spheres is Right now and The Euclidean distance between them, d is also a slack variable for determining semantic equivalence, that is, or A vector whose distance does not exceed d is semantically equivalent to or Where (x (i) ,y (i) ) and (x (j) ,y (j) ) are two instances randomly selected from the training corpus.

[0019] In one embodiment, the step 4 defines r z and r y is the vector representation in the adjacent semantic region, r′ is r x and r y The mean value of With r′ as the center, Generate a sphere for radius and sample inside the sphere.

[0020] In one embodiment, a set of parameters following a Gaussian distribution is set Transform the direction and magnitude of r″ so that when hour, The value of is smaller than the radius of the circle, thus ensuring that the values ​​obtained are all within the sphere.

[0021] In one embodiment, in step 5, the translation model adopts the Transformer-XL model, that is, a loop mechanism between segments is introduced into the original Transformer model. When processing the subsequent segments in the training phase, each hidden layer will accept two types of inputs, one is the output of the previous node of the current segment, and the other is the output of the previous segment node, so that the current segment can use the information of the previous segment to achieve long-term dependency when modeling; wherein the outputs of each decoder are dynamically combined by the following formula:

[0022]

[0023] Among them B irepresents different decoder modules, W i Represents the weight of each module, W i It is obtained through the back-propagation algorithm, and N represents the number of modules.

[0024] In one embodiment, the Transformer-XL model uses relative position encoding. When calculating the attention score, only the relative position relationship between the query vector and the key vector is considered, and the relative position relationship is added to the attention calculation of each layer.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. The present invention conducts research from two aspects: the morphological composition and semantic information of Mongolian. Aiming at the problems of lack of parallel corpus and poor translation quality in the process of Mongolian-Chinese neural machine translation, a method based on Mongolian case component mask data enhancement and improved Transformer-XL is used for translation to improve the translation quality.

[0027] 2. In order to solve the problems of lack of parallel corpus and inconsistent semantics of generated enhanced sentences in the translation process, the present invention performs mask data enhancement on the case components of Mongolian and performs semantic re-enhancement on the enhanced sentences in the semantic space.

[0028] 3. The present invention removes case components in different states in Mongolian sentences while retaining the main components to provide a very rich semantic state structure for training data. In the process of training the case component mask model, the words at the source and target ends are spliced ​​simultaneously to predict the masked Mongolian case components to enhance semantic consistency. In the word replacement process, a soft masking method is used to replace the masked words with probability distribution vectors on the vocabulary, so that a better word replacement distribution can be generated.

[0029] 4. To address the data enhancement problem in the semantic space, the present invention uses the BERT pre-trained model as a semantic encoder to map the semantic information of Mongolian and Chinese into a shared vector space, thereby achieving semantic representation and understanding of the two languages. Then, a contrastive learning method is used to optimize the space, in which Mongolian is sampled and formed into a parallel corpus with Chinese for data enhancement.

[0030] 5. To address the problem of long-distance dependency in the Mongolian-Chinese translation process, the present invention adopts the Transformer-XL model as the translation model. Compared with the Transformer model, Transformer-XL has two innovations. One is that a loop mechanism is introduced between segments, so that the current segment can use the information of the previous segment to achieve long-term dependency when modeling. The other is that the position encoding is performed according to the relative distance between words.

[0031] 6. The present invention improves the decoder side of the Transformer-XL model by dynamically combining the outputs of each module on the decoder side. Each module can capture different semantic or grammatical features. The combination of these features can improve the quality of machine translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The overall flow chart of the present invention is

[0033] Figure 2 This is a diagram showing the impact of changes in Mongolian affixes and case components on vocabulary.

[0034] Figure 3 It is the grid component mask model diagram.

[0035] Figure 4 These are three different Mongolian case component mask images.

[0036] Figure 5 This is an example diagram of Mongolian case component mask.

[0037] Figure 6 This is a graph of adjacent semantic regions for Mongolian and Chinese.

[0038] Figure 7 Sampling structure diagram of adjacent semantic regions.

[0039] Figure 8 This is the Transformer-XL training structure diagram.

[0040] Fig. 9 This is an example picture of Mongolian-Chinese neural machine translation. DETAILED DESCRIPTION

[0041] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0042] As mentioned above, Mongolian is a low-resource language with a relatively scarce parallel corpus. Its word formation and grammatical structure are usually very different from those of Chinese, and its semantic structure is relatively simple. The main problem faced in Mongolian-Chinese translation is the lack of parallel corpus, and data enhancement must be performed on the corpus to solve this problem. Conventional Mongolian-Chinese data enhancement methods are mostly direct transfers of enhancement methods from other languages, without taking into account the characteristics of Mongolian itself.

[0043] Therefore, the present invention provides a Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement, which adopts the method of masking data enhancement of Mongolian case components and once again enhancing the corpus in the semantic space to alleviate the problems of corpus shortage and low translation quality in Mongolian-Chinese neural machine translation; and adopts an improved Transformer-XL model for Mongolian-Chinese neural machine translation to solve the problems of long-distance dependency and semantic inconsistency in the Mongolian-Chinese translation process.

[0044] like Figure 1 As shown in the figure, this figure is an overall flow chart. The present invention first marks the case components of Mongolian, and then masks the marked Mongolian case components. When masking, only the Mongolian side is masked, but the data of the Mongolian side and the Chinese side are used at the same time during prediction. The sentence processed by the case component masking model is further data enhanced in the semantic space. The semantic information of Mongolian and Chinese is mapped to a shared vector space through the BERT pre-trained semantic encoder. The semantic space is optimized using the contrastive learning method to generate an adjacent semantic area with similar semantics to the original sentence. Then, the semantically similar sentence vectors are sampled from the corresponding adjacent semantic areas and integrated into the decoder through the broadcast integration network to participate in the training. When training, the translation model adopts the improved Transformer-XL model.

[0045] The main steps of the present invention are described in detail below:

[0046] Step 1: Propose a simple Mongolian case marking method to assist in decoding case components in machine translation, and combine Mongolian grammatical rules to perform Mongolian case component analysis and case component marking.

[0047] This step first requires marking the case components of Mongolian. The case components in Mongolian are additional components of conjunctions or expressions of tense and state. They have no actual semantics and are mainly used to assist in expressing the state of words in different situations. Mongolian case components have fixed Unicode codes and can be embedded separately during encoding. In specific applications, a simple script can be created based on the grammatical rules of Mongolian to automatically add case markers to each component based on features such as part of speech. The case change rules of Mongolian are relatively complex, so no matter which annotation method is used, the grammatical role of each component in the sentence still needs to be carefully considered when annotating.

[0048] Figure 2It shows the morphological changes of several typical affixes and case components in Mongolian vocabulary. From left to right, the four boxes are defined as the first box, the second box, the third box, and the fourth box. The connecting arrow between the first box and the second box is defined as the first arrow, the connecting arrow between the second box and the third box is defined as the second arrow, and the connecting arrow between the third box and the fourth box is defined as the third arrow. The translations of the Mongolian words involved are explained as follows:

[0049] The Mongolian in the first box means "knowledge" in Chinese. The Mongolian above the first arrow represents an affix without a specific meaning, which can be defined as the "first affix".

[0050] In the second box, the Mongolian on the left side of the arrow in the box represents "knowledge + first affix", and the Mongolian on the right side of the arrow in the box represents the new Mongolian after morphological change, which means "wisdom" in Chinese. The Mongolian above the second arrow represents an affix without a specific meaning, which can be defined as the "second affix".

[0051] In the third box, the Mongolian on the left side of the arrow indicates "knowledge + second affix", and the Mongolian on the right side of the arrow indicates the new Mongolian after morphological change, which means "wise man" in Chinese. The Mongolian above the third arrow indicates an additional case component, which has no specific meaning.

[0052] The Mongolian word in the fourth box is a new word obtained by adding additional case components to "wise man", and its Chinese meaning is "to make one wise".

[0053] There are different case components in Mongolian, such as attribute case, position case, entity case, subordinate case, and additional case. The state of Mongolian words or sentences can be changed by adding different case components. When performing data enhancement on Mongolian case components, it is necessary to clarify the role of case components in model encoding and decoding. In previous research work, it has been determined that Mongolian case components act as representations and connections of tense and state in sentence composition, and removing Mongolian case components will not have a significant impact on the semantics of the sentence.

[0054] Step 2: After marking the case components of Mongolian, mask data enhancement is performed on the case components.

[0055] In order to solve the problem of lack of corpus and semantic inconsistency in Mongolian-Chinese translation, this step first uses the pre-trained multilingual BERT to initialize the case component masking model, and then splices the source Mongolian and target Chinese sentences in training. The case components of the source Mongolian are masked by three methods: random masking, partial fixed masking, and full masking. The Mongolian case component masking language model that predicts the masked case components is trained. The masked Mongolian case components can be predicted by the case component masking model. The sentences generated by the case component masking model are then subjected to data enhancement in the semantic space to solve the problem that traditional data enhancement methods cannot maintain semantic consistency.

[0056] refer to Figure 3 , This figure is a diagram of a case component mask model. The pre-trained model BERT is used as the initial model, and then fine-tuned on this basis. BERT itself has the function of masking, so using it as a mask model can achieve better results. During fine-tuning, the information of the source and target ends can be input together to predict the information of the source end, which can significantly improve the accuracy of the prediction, and finally train a Mongolian case component mask language model. The model is composed of multiple TransformerEncoders. A Transformer Encoder unit is generated by the superposition of a multi-head attention mechanism, a normalization layer, a feedforward neural network, and a normalization layer. Each layer consists of such an Encoder unit. From the shallow to the high level, the model can learn the surface, phrase level, syntactic level, and semantic level information respectively. Therefore, using it as a case component mask language model can better capture the semantic information of the sentence, thereby improving the accuracy of the prediction.

[0057] Figure 3 The Mongolian involved in the top row refers to an input masked Mongolian sentence (the middle of the sentence is omitted), and its Chinese meaning is "It seems that you are right".

[0058] Figure 3 In the figure, the Mongolian involved in the bottom row refers to the Mongolian predicted based on the Mongolian on the source side and the words on the target side, and its Chinese meaning is the same as that in the top row.

[0059] refer to Figure 4, the figure shows three different Mongolian case component masking methods. The first is random masking, that is, randomly masking the case components of Mongolian. The optimal masking range is determined in the experiment. This method randomly masks the labeled case components based on the idea of ​​simple data enhancement method. The second is partial fixed masking, that is, using the fixed collocation probability table of fixed phrases or phrases in the word embedding stage for masking. This method can mask the case components in the fixed phrases in Mongolian sentences according to the probability of word vector distribution in the output of the training stage. The third is full masking, that is, masking all case components in an input sentence. This method can mask all case components in the corpus according to the labeled case components.

[0060] Figure 4 In the above example, the Mongolian words involved in each row all have the same Chinese meaning of “I am worried whether he can pass the exam”, and the case components are masked using three different case component masking methods.

[0061] refer to Figure 5 , there is a Mongolian and Chinese (X, Y) sentence pair, representing the source sentence and the target sentence. X=(x1,x2,x3,...,x m ) and Y=(y1,y2,y3,....,y n ) are two label sequences in the source language and the target language, with lengths of m and n respectively. The Mongolian case component masking method proposed in the present invention acts on both sides, but is only used to predict one side. Specifically, for each sentence pair (X, Y), the pre-trained multilingual BERT is first used to initialize the case component masking model, and then Mongolian and Chinese are spliced ​​in training. The case components of Mongolian are masked using three methods: random masking, partial fixed masking, and full masking. A case component masking model is trained to predict the masked Mongolian case components. The probability formula used is: In order to solve the problem of sample diversity in the data enhancement process, many candidate words will be generated when the lattice component mask model is used to predict the masked content. In order to ensure the diversity of samples, a large amount of data must be processed. The direct sampling and replacement method is relatively inefficient. Therefore, the present invention adopts the idea of ​​soft word replacement in the process of word replacement. Instead of sampling specific words, the expected word vectors in the vocabulary are calculated based on the predicted distribution, and the words in the masked position are replaced with the probability distribution in the vocabulary. That is, for the candidate words generated when the lattice component mask model is predicted, the soft word replacement method is adopted to replace the masked words with the probability distribution vectors in the vocabulary, so that the lattice component mask model predicts multiple possible words, reducing the training instability caused by a single prediction error. Specifically, taking the soft word w as a candidate word, the distribution of p(w) is defined by the probability formula The decision, p(w) is based on the context and covers the entire vocabulary. Suppose E is the embedding matrix of all |V| words, EW represents the embedding vector of soft word w, E j is the embedding matrix of the jth word, P j (w) is the probability distribution of the jth word, V is the number of words, and the soft word replacement method is expressed as follows:

[0062]

[0063] Among them, e w represents the embedding of soft word w.

[0064] Figure 5 In the text, the Mongolian words involved in each row mean “It is a pity not to accept his suggestion” in Chinese (part of the content is omitted in the middle of the input sentence).

[0065] In the present invention, candidate words refer to words that will replace the original components. Soft words refer to replacing the original words with the probability distribution of words in the vocabulary, rather than directly selecting a word for replacement.

[0066] This data augmentation method can ensure semantic consistency when generating enhanced sentences. The so-called semantic consistency requires that the source language and the target language are fluent and grammatically correct in their respective languages, and the target language is a high-quality translation of the source language.

[0067] The first two steps of the present invention increase high-quality training samples by masking Mongolian case components and combining data enhancement technology in adjacent semantic space.

[0068] Step 3: Map the semantic information of Chinese and Mongolian after data enhancement in the second step into the same vector space through a semantic encoder, and generate an adjacent semantic region; the adjacent semantic region contains multiple vector representations of Mongolian and Chinese with similar semantics.

[0069] After masking the Mongolian case components, the present invention further enhances the samples in a continuous semantic space. Specifically, there is a common semantic space between the source language and the target language of neural machine translation, which is generated by the semantic encoder. Therefore, for data enhancement in the semantic space, the present invention uses the pre-trained model BERT as the semantic encoder. In the semantic encoder, a semantic encoder forward function parameterized by θ′ is defined. It can map discrete sentences into continuous vector representations. This semantic encoder converts the source sentence x and the target sentence y into real-valued vectors Thereby, the semantic information of Mongolian and Chinese is mapped in the same vector space, thus achieving semantic representation and understanding of the two languages.

[0070] The present invention can specifically generate an adjacent semantic region by the following method: Assume (x (i) ,y (j) ) and (x (j) ,y (j) ) are two instances randomly selected from the training corpus, adjacent to the semantic region Defined as and The union of two closed spheres centered at is, and the radius of the two spheres is Right now and The Euclidean distance between . d is also considered to be a slack variable for determining semantic equivalence, that is, or A vector whose distance does not exceed d is semantically equivalent to or The contrastive learning method is used to optimize the semantic space range (i.e., the vector space) so that each vector generates a semantically similar region in the continuous space, i.e., an adjacent semantic region, which contains a variant of the same semantic representation as the source sentence, i.e., an adjacent semantic region v(r) in the semantic space. x ,r y ) contains enough variants of the described sentence pair (x, y), and contains many vector representations of Mongolian and Chinese with similar semantics in this region. Dynamic negative sampling is used to select negative samples in contrastive learning. Dynamic negative sampling is an adaptive negative sample selection method. Its idea is to select negative samples based on the current representation of the model. For example, during the training process, as the model is updated, the negative sample selection strategy may be dynamically adjusted based on the erroneous samples generated by the model.

[0071] like Figure 6 As shown, semantic encoders are very necessary for constructing a common semantic space between different languages, and using a pre-trained model as a semantic encoder has a better effect. The BERT pre-trained model adopts a self-attention mechanism, which enables it to perform bidirectional context modeling on the input text. This means that it can not only utilize the information of the previous text, but also effectively capture the information of the following text, so as to better understand the meaning of words in sentences and thus better extract semantic information. The present invention uses the BERT pre-trained model as a semantic encoder, maps the semantic information of Mongolian and Chinese to a shared vector space, thereby realizing the semantic representation and understanding of the two languages, and then uses a contrastive learning method to optimize the space. In this common semantic space, similar sentences or phrases have similar vector representations between the source language and the target language, and Mongolian is sampled in this space for data enhancement.

[0072] Step 4: Samples are sampled in adjacent semantic regions and decoded using the broadcast network integrated into the decoder of the translation model.

[0073] After generating a contiguous semantic space of Mongolian and Chinese, we need to sample samples in this area. Figure 7 , define r x and r y is the vector representation in the adjacent semantic region, let (black arrow), is r x and r y The mean of the circle, with r′ as the center, (yellow arrow) is the radius to generate a sphere and sample the samples within the sphere. Sampling vectors in the region where the semantics of Mongolian and Chinese overlap can ensure that the sampled samples have a closer semantic representation to the original training samples, so that the enhanced samples will produce better results during the training process. Set a parameter Transform the direction and magnitude of r″ so that (green arrow), when hour, The value of will be smaller than the radius of the circle, so this ensures that the obtained value is within the sphere. The selection, assuming Following the Gaussian distribution, we only need to find a suitable set of This will ensure smooth sampling. (Purple arrow) is the sampled sample. The sampled sample is integrated into the decoder using a broadcast integration network for decoding. This method can alleviate the problem of corpus scarcity and semantic inconsistency in the Mongolian-Chinese translation process.

[0074] Step 5: The translation model uses the Transformer-XL model. During the decoding operation, the output of each decoder module is dynamically combined to express richer semantic information. After being processed by the encoder and decoder, it finally passes through the Softmax layer to output the probability of the word.

[0075] In the field of natural language processing, the encoding ability of Transformer surpasses that of neural network models such as RNN, but its modeling ability for long-distance dependencies is still insufficient. During training, it usually divides the input into segments and learns from each segment. During the test phase, if it needs to process longer inputs, the model will move the input to the right by one character at each step to predict the next character. Therefore, using ordinary Transformer for translation has the following disadvantages: one is that the maximum dependency distance between characters is limited by the input length, and the model cannot see the words that appear several sentences ago. In addition, there is no context dependency between segments, which makes training inefficient and affects the performance of the model. During the test phase, each prediction of the next word requires rebuilding the context and starting the calculation from scratch, which is very slow. In addition, the original Transformer model is not very sensitive to long sequence problems when processing sentences, and data does not flow between segments. Therefore, the upper limit of the long-term dependency that the model can capture is the length of the segment. Transformer-XL introduces a loop mechanism between segments when modeling, so that the current segment can use the information of the previous segment to achieve long-term dependency during modeling, thereby better capturing the semantic information of the sentence.

[0076] like Figure 8 The figure shows the training structure diagram of Transformer-XL. This model can better solve the problem of long encoders capturing long-distance dependencies. The Transformer-XL model introduces a loop mechanism between segments in the original Transformer model. When processing the subsequent segments in the training phase, each hidden layer will accept two types of inputs, one is the output of the node in front of the current segment, and the other is the output of the node in the previous segment. The output enables the model to create long-term dependencies and utilize the information of more previous segments, so that the current segment can utilize the information of previous segments to achieve long-term dependencies when modeling. The Transformer-XL model uses relative position encoding, based on the relative distance between words instead of absolute position encoding like in Transformer. It proposes a new position encoding method, which is part of each attention module. It will not only encode the position before the first layer, but will encode based on the relative distance between representations instead of absolute position. The specific operation is that when calculating the attention score, only the relative position relationship between the query vector and the key vector is considered, and the relative position relationship is added to the attention calculation of each layer. The formula for calculating attention in Transformer is:

[0077]

[0078] After decomposition:

[0079]

[0080] Where E x represents the input embedding of the token, U is the absolute position encoding embedding, and the two W are the query matrix and the key matrix respectively. The way to calculate attention in Transformer-XL is to replace the absolute position encoding U with the relative position encoding R i-j , W k The matrix is ​​further divided into two groups of matrices W k,E and W k,R Generate content-based key vectors and position-based key vectors respectively.

[0081]

[0082] Each module of the decoder part of the Transformer-XL model can capture different semantic and grammatical information. The present invention dynamically combines the outputs of each decoder module. Each module can capture different semantic features. After the combination, richer semantic information can be obtained to improve the quality of machine translation. The outputs of each decoder are dynamically combined through the following formula:

[0083]

[0084] Among them B i represents different decoder modules, W i Represents the weight of each module, W i Obtained through the back-propagation algorithm, N represents the number of modules. The processed corpus first enters the encoder part of the improved Transformer-XL model, after input embedding and position encoding, it is input into the multi-head attention mechanism and the feedforward neural network to output the word vector, and then input into the decoder for decoding. After decoding, it is input into the Softmax layer for prediction and outputs the predicted probability. The Softmax function is used to normalize the classification results to form a probability distribution. The definition of the Softmax function is given below (taking the output of the i-th node as an example):

[0085]

[0086] Where Z i is the output value of the ith node, and C is the number of output nodes. The Softmax function can be used to convert the output value into a probability distribution in the range [0,1] and 1.

[0087] Step 6: Train the original samples together with the data-augmented samples, and use the trained translation model to perform Mongolian-Chinese machine translation.

[0088] refer to Fig. 9 The present invention performs Mongolian case component mask data enhancement and data enhancement in adjacent semantic regions on the input Mongolian, and then trains the improved Transformer-XL translation model, and finally uses the trained model to perform Mongolian-Chinese machine translation.

[0089] Fig. 9 In the text, the Mongolian words mentioned in each row mean “it will take us a long time to compile this dictionary” in Chinese.

[0090] The present invention uses Mongolian case component mask data enhancement and improved Transformer-XL method to perform Mongolian-Chinese neural machine translation. First, the case components of Mongolian are annotated, and then the annotated case components are masked. When the Mongolian case components are data enhanced, the source and target languages ​​are used as conditions to predict the masked words, and then the probability distribution vectors on the vocabulary are used to replace the masked words. At the same time, a semantic encoder is pre-trained with BERT so that the training samples produce a continuous adjacent semantic region, and the contrastive learning method is used for optimization. The region is a variant with the same semantics as the original training sentence, and can be sampled from the region for training. Finally, the improved Transformer-XL model is used for machine translation. The above technology can significantly improve the quality and efficiency of Mongolian-Chinese neural machine translation.

Claims

1. A Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement, characterized in that: The steps include: Step 1, combining Mongolian grammatical rules to perform case component analysis and case component labeling in Mongolian; Step 2, perform mask data enhancement on the grid components; Step 3, mapping the semantic information of Chinese and Mongolian after data enhancement in step 2 into the same vector space through a semantic encoder, and generating an adjacent semantic region; the adjacent semantic region contains vector representations of multiple Mongolian and Chinese with similar semantics; Step 4, sampling samples in the adjacent semantic region, and performing a decoding operation using a decoder integrated into the translation model using the broadcast network; Step 5: The translation model uses the Transformer-XL model, which dynamically combines the outputs of each decoder module during the decoding operation. Step 6: Train the original samples together with the data-augmented samples, and use the trained translation model to perform Mongolian-Chinese machine translation; in: In step 2, the pre-trained model BERT is used as the initial Mongolian case component masking model, and sentences of source Mongolian and target Chinese are spliced ​​during training, and the case components of source Mongolian are masked by three methods: random masking, partial fixed masking, and full masking, so as to obtain a Mongolian case component masking model through training; wherein the random masking is to randomly mask the case components of Mongolian, and determine the optimal masking range in the experiment; the partial fixed masking is to mask using the fixed collocation probability table of fixed phrases or phrases in the word embedding stage; the full masking is to mask all case components in an input sentence; For the candidate words generated during the prediction of the lattice component masking model, a soft word replacement method is used to replace the masked words with probability distribution vectors, so that the lattice component masking model predicts multiple possible words. The soft word w is taken as a candidate word, and the soft word replacement method is expressed as follows: Among them, e w is the embedding of soft word w, E W represents the embedding vector of soft word w. The distribution of p(w) is determined by the defined probability formula. p(w) is based on the context and covers the entire vocabulary. E j is the embedding matrix of the jth word, P j (w) is the probability distribution of the jth word, V is the number of words; The Transformer-XL model introduces a loop mechanism between segments in the original Transformer model. When processing the subsequent segments in the training phase, each hidden layer receives two types of inputs: one is the output of the previous node of the current segment, and the other is the output of the previous segment node. This allows the current segment to use the information of the previous segment to achieve long-term dependency when modeling. The outputs of each decoder are dynamically combined using the following formula: Among them B i represents different decoder modules, W i Represents the weight of each module, W i It is obtained through the back-propagation algorithm, and N represents the number of modules.

2. According to claim 1, the Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement is characterized in that: In step 3, the pre-trained model BERT is used as a semantic encoder, in which a semantic encoder forward function parameterized by θ′ is defined. To map discrete sentences into continuous vector representations; the semantic encoder converts the source sentence x and the target sentence y into real-valued vectors Thereby, the semantic information of Mongolian and Chinese is mapped in the same vector space.

3. The Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement according to claim 1 or 2 is characterized in that: The vector space is optimized by using a contrastive learning method so that each vector generates a semantically similar region in a continuous space, namely, the adjacent semantic region, which contains a variant of the same semantic representation as the source sentence.

4. The Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement according to claim 3 is characterized in that: The adjacent semantic regions are defined as and The union of two closed spheres centered at is, and the radius of the two spheres is Right now and The Euclidean distance between them, d is also a slack variable for determining semantic equivalence, that is, or A vector whose distance does not exceed d is semantically equivalent to or Where (x (i) ,y (i) ) and (x (j) ,y (j) ) are two instances randomly selected from the training corpus.

5. The Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement according to claim 4 is characterized in that: In step 4, define r x and r y is the vector representation in the adjacent semantic region, r′ is r x and r y The mean value of With r′ as the center, Generate a sphere for radius and sample inside the sphere.

6. The Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement according to claim 5 is characterized in that: Set a set of parameters that follow a Gaussian distribution Transform the direction and magnitude of r" so that when hour, The value of is smaller than the radius of the circle, thus ensuring that the values ​​obtained are all within the sphere.

7. The Mongolian-Chinese neural machine translation method based on Mongolian case component mask data enhancement according to claim 1 is characterized in that: The Transformer-XL model uses relative position encoding. When calculating the attention score, it only considers the relative position relationship between the query vector and the key vector, and adds this relative position relationship to the attention calculation of each layer.

Images