Mongolian-Chinese neural machine translation method based on constraint enhanced alignment

Abstract

The invention discloses a Mongolian-Chinese neural machine translation method based on constraint enhanced alignment, which comprises the following steps: segmenting Mongolian-Chinese parallel corpora, and preprocessing the segmented corpora; performing random dynamic sampling from the reference translation to obtain Chinese constraints, and splitting Mongolian corresponding to the Chinese constraints into stem and affix forms as Mongolian constraints; adding Chinese and Mongolian constraints to the end of the source sentence to generate a pseudo source sentence, and mixing the pseudo source sentence with the original Mongolian-Chinese parallel corpus to construct a Mongolian-Chinese synthetic parallel corpus; source sentences and constraints in the pseudo source sentences are distinguished through special marks; respectively encoding the source sentence, the Chinese constraint and the Mongolian constraint by using double encoders; adding a word alignment module into the decoder, carrying out decoding operation, training to obtain a final Mongolian-Chinese neural machine translation model, and carrying out Mongolian-Chinese translation; according to the method, the Mongolian-Chinese neural machine translation decoding speed and the proportion of the constraint words appearing in the target sentence can be increased, and the overall translation quality of the Mongolian-Chinese neural machine translation model is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation in natural language processing, and in particular relates to a Mongolian-Chinese neural machine translation method based on constraint enhanced alignment. Background Art

[0002] Machine translation uses computers to automatically translate one language into another. With its deep understanding of semantics, neural machine translation has become the mainstream machine translation method. It relies on large-scale, high-quality parallel corpora. The higher the quality of the corpus, the better the training effect. However, the scarcity of Mongolian corpora has limited the performance of Mongolian-Chinese neural machine translation. How to use existing corpora to alleviate the problem of low translation quality caused by insufficient resources has become an important topic in neural machine translation research.

[0003] The constraint-enhanced neural machine translation method is to input the constraint words that must appear in the source language and the target language sentences into the machine translation model, and generate translation results that meet the constraint requirements through the decoder. However, due to the scarcity of Mongolian corpus and the complex morphological changes, the current Mongolian constraint-enhanced neural machine translation, whether by designing a new decoding algorithm or changing the training process of Mongolian-Chinese neural machine translation, has problems such as slow decoding speed, low proportion of constraint words appearing in the target sentence, and poor quality of the generated translation. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a Mongolian-Chinese neural machine translation method based on constraint-enhanced alignment to solve the problems that arise in the Mongolian translation process of the existing constraint-enhanced machine translation method. The invention aims to improve the decoding speed of Mongolian-Chinese neural machine translation, the proportion of constraint words appearing in the target sentence, and improve the overall translation quality of the Mongolian-Chinese neural machine translation model.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A Mongolian-Chinese neural machine translation method based on constraint enhanced alignment includes the following steps:

[0007] Step 1, segmenting the Mongolian-Chinese parallel corpus and preprocessing the segmented corpus;

[0008] Step 2, randomly and dynamically sampling the Chinese constraints from the reference translation, splitting the Mongolian corresponding to the Chinese constraints into stems and affixes as Mongolian constraints;

[0009] Step 3, adding the Chinese constraint and the Mongolian constraint to the end of the source sentence to generate a pseudo source sentence, and mixing the pseudo source sentence with the original Mongolian-Chinese parallel corpus to construct a Mongolian-Chinese synthetic parallel corpus;

[0010] Step 4, distinguish the source sentence and the constraint in the pseudo source sentence with special marks;

[0011] Step 5, use a dual encoder to encode the source sentence and the Chinese constraints and Mongolian constraints respectively;

[0012] Step 6, add a word alignment module to the decoder and perform decoding operation;

[0013] Step 7: The output of the dual encoder is used as the input of the decoder to train the final Mongolian-Chinese neural machine translation model;

[0014] Step 8: Use the final Mongolian-Chinese neural machine translation model to perform Mongolian-Chinese translation.

[0015] In one embodiment, the step 1 uses byte pair encoding (BPE) technology to segment the Mongolian-Chinese parallel corpus, and segments the words in each sentence into subword units, so that rare words and unregistered words can be broken down into smaller, more frequent subword units. The segmented corpus is preprocessed using an open source model to divide it into a training set, a validation set, and a test set. For example, a random sampling method can be used to determine the data set division, randomly extracting 10% of the corpus as a validation set, randomly extracting 10% of the corpus as a test set, and the remaining 80% of the corpus as a training set.

[0016] In one embodiment, in step 2, firstly, the number of constraints K is randomly set, K words are randomly selected from the reference translation as Chinese constraints, then the Mongolian translation corresponding to the Chinese constraints is obtained, and then the Mongolian is decomposed into stems and affixes, and finally the Mongolian constraints are obtained. Since the high-frequency words have been learned relatively well by the model, the high-frequency words in the sampled constraint words are removed.

[0017] In one embodiment, in step 3, the selected Chinese constraints and Mongolian constraints are first added to the source sentence (i.e., Mongolian), and the order of placing the constraints is Chinese constraints first and then Mongolian constraints, to construct a pseudo source sentence, and then the pseudo source sentence is mixed with the original Mongolian-Chinese parallel corpus to construct a Mongolian-Chinese synthetic parallel corpus.

[0018] In one embodiment, in step 4, in the encoding process of the pseudo source sentence, in order to enable the model to distinguish between the Mongolian source sentence and the constraint vocabulary, three different embedding vectors are used to distinguish between the source sentence and the constraint vocabulary. First, the word embedding vector is introduced to encode the source sentence, Chinese constraint and Mongolian constraint into the form of a vector; secondly, the position embedding vector is introduced to capture the order and position relationship of the words in the source sentence; finally, different fragment embedding vectors are introduced for the source sentence, Chinese constraint and Mongolian constraint to distinguish between the source sentence part and the constraint vocabulary part; the three embedding vectors are summed to obtain the final input vector of the encoder, which is expressed as:

[0019] E=Etok +E pos +E seg

[0020] Where E tok is the word embedding vector, which represents the semantic information of the word; E pos is the position embedding vector, indicating the position of the word in the sentence; E seg is a fragment embedding vector, indicating whether the word belongs to the source sentence or the constraint part.

[0021] The final pseudo-source sentence is as follows:

[0022]

[0023] Where x is the source sentence, c 1 ,c 2 ,...,c N Indicates the 1st to Nth constraints, special mark <sep>Used to distinguish between source sentences and constraints, and between constraints. <eos>It is the end marker of the pseudo source sentence.

[0024] In one implementation example, the dual encoder in step 5 is a source sentence encoder and a constraint word encoder. The source sentence encoder encodes the semantic and syntactic structure of the source sentence, i.e., Mongolian, to generate a context representation H of the source sentence. tok =ENCODER tok (E tok ), the constraint word encoder encodes the Chinese constraint and the Mongolian constraint to generate a context representation H of the constraint word cons =ENCODER cons (E cons ), the dual encoder outputs the final joint hidden state H aug =ENCODER joint (E′ aug ).

[0025] in is the vector representation of the source sentence, is the word vector of the i-th word in the source sentence, is the vector representation of M constraints, is the word vector representation of the jth constraint, E a ' ug =[H tok ,E cons ] is the Mongolian and constraint concatenation vector in the pseudo source sentence, ENCODER joint It is the joint encoding of the source sentence encoder and the constraint word encoder.

[0026] In one implementation example, the source sentence encoder and the constraint word encoder both use Transformer encoders. For the source sentence encoder, the Mongolian source sentence is first encoded into a vector form through an embedding layer, and then a multi-head self-attention layer focuses on the correlation between different positions in the input sequence. The output of the multi-head self-attention layer is passed to a feedforward neural network layer, and the feedforward neural network layer captures richer Mongolian word features; for the constraint word encoder, Mongolian constraints and Chinese constraints are first encoded into a vector form before encoding, and then the multi-head self-attention layer receives the Mongolian constraint words, Chinese constraint words and hidden states output by the source sentence encoder, and focuses on other words in the input sentence when encoding each word, and obtains the hidden state of the dual encoder through the output of the feedforward neural network layer.

[0027] In one implementation example, the step 6, adding a word alignment module to the decoder, is implemented as follows:

[0028] The first step is to determine the position of the word alignment module in the decoder.

[0029] In order to select the decoder layer that is most suitable for the word alignment module, the present invention first trains a Mongolian-to-Chinese model θ given a Mongolian-Chinese parallel sentence pair x, y> x→y and a Chinese to Mongolian model θ y→x Then, we extract the word alignment information from Mongolian to Chinese and from Chinese to Mongolian from the two models respectively, and use the word alignment information from Chinese to Mongolian as the alignment reference to evaluate the alignment error rate (Alignment Error Rate, AER) from Mongolian to Chinese on the validation set.

[0030] For each model, l different word alignment information can be obtained from l decoder layers, so there are l×l AER scores in total. Finally, the decoder layer with the lowest alignment error rate is selected as the position of the word alignment module in the decoder. The specific representation is as follows:

[0031]

[0032] The second step is to insert the word alignment module.

[0033] The input of the word alignment module includes: the joint hidden state H output by the dual encoder aug =ENCODER joint (E′ aug ), the output of the decoder self-attention sublayer And the last word y t The embedding vector e(y t ).

[0034] In one implementation example, the calculation method after inserting the word alignment module in step 6 is:

[0035]

[0036]

[0037] Where n is the number of attention heads, t is the position of the target marker, refers to the output of the self-attention sublayer of the lth layer of the decoder at the tth target token, l∈{1,...,L}, K n 、V n are query, key, and value matrices respectively, P(a t |x,y ≤t ) is the product of the query vector and the key vector, d is the dimension size of the attention head, is the output of the attention layer, is a learnable weight matrix;

[0038] The final objective function is calculated as follows:

[0039]

[0040] Where T is the length of the target sentence, S is the length of the source sentence, and S i,j The i-th target word corresponds to the j-th source word.

[0041] In one implementation example, in step 7, before the pseudo source sentence is input into the dual encoder, it passes through three embedding layers, namely word embedding, position embedding and fragment embedding. Then the source sentence and the constraint are respectively put into the two encoders for encoding, and the hidden states extracted therefrom are used as the context vectors of the source sentence and the constraint word. Then the hidden states output by the dual encoder are used as the input of the decoder. The decoder calculates the alignment scores of Mongolian and Chinese through the word alignment module, and finally outputs the target text according to the context and alignment information.

[0042] The conditional probability calculation method of the Mongolian-Chinese neural machine translation model is as follows:

[0043]

[0044] Where X = {x 1 ,x 2 ,...,x s } is the source sentence, Y={y 1 ,y 2 ,....,y T } is the target sentence, C=(C 1 ,C 2 ,...,C N ) is a binding word.

[0045] The final loss function calculation method of the Mongolian-Chinese neural machine translation model is as follows:

[0046]

[0047] in It is a pseudo-source sentence.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. In view of the scarcity of Mongolian-Chinese parallel corpora, the present invention proposes to adopt a constraint enhancement method to expand the scale of training data by constructing synthetic parallel corpora. The constructed synthetic parallel corpora contain both pseudo source sentences with constrained words and source sentences without constrained words. The former helps to train the Mongolian-Chinese neural machine translation model on how to include specified vocabulary in the output of the target language; the latter helps with conventional Mongolian-Chinese translation tasks. The model is trained in both constrained and unconstrained corpora, which is consistent with the diversified translation needs in practical applications.

[0050] 2. Since the neural network cannot balance two different types of information well, if the two different types of information, Mongolian and Chinese, are integrated into the same encoder, one type of information dominates the other, resulting in poor quality of the generated translation. To address this problem, the present invention proposes to use two encoders to encode the source sentence and the constraint words separately, so that the encoding processes do not interfere with each other and the model can flexibly handle different constraints, thereby generating grammatically correct and contextually consistent translations to improve translation quality.

[0051] 3. Due to the significant differences in grammatical structure between Mongolian and Chinese and the relatively complex morphological changes in Mongolian, the problem of slow decoding speed during constraint-enhanced Mongolian-Chinese neural machine translation is caused. In view of this problem, the present invention proposes to split the Mongolian corresponding to the Chinese constraint word into the form of stems and affixes, and input them into the Mongolian-Chinese neural machine translation model as Mongolian constraints. The stem contains the main meaning of Mongolian, which improves the efficiency of the model in searching the Mongolian vocabulary corresponding to the Chinese constraint and the source sentence, thereby improving the decoding speed.

[0052] 4. Constraint enhancement is achieved by changing the translation process of the Mongolian-Chinese neural machine. When generating the target text, the decoder gives priority to contextual fluency and grammatical correctness, rather than strictly following that all constrained words appear in the target text, resulting in some constrained words being ignored and the proportion of constrained words appearing in the output text being low. To address this problem, the present invention proposes inserting a word alignment module into the decoder. By calculating the alignment score between the source word and the target word, when the decoder generates the translation, it gives priority to words with high alignment scores, thereby achieving the purpose of increasing the proportion of constrained words appearing in the output text. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is the overall flow chart of the present invention.

[0054] Figure 2 This is the pseudo-source sentence structure diagram.

[0055] Figure 3 Diagram of the architecture for encoding source sentences and constraints separately.

[0056] Figure 4 Transformer architecture diagram with word alignment module inserted.

[0057] Figure 5 This is the overall architecture diagram of the Mongolian-Chinese neural machine translation model.

[0058] Figure 6 This is an example picture of Mongolian-Chinese neural machine translation. DETAILED DESCRIPTION

[0059] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0060] Mongolian is a morphologically rich, low-resource language. Due to the incomplete construction of Mongolian language resources, the research on Mongolian-Chinese machine translation has made very slow progress; the scarcity of Mongolian-Chinese bilingual parallel corpus also affects the development of Mongolian-Chinese machine translation.

[0061] Based on this, the present invention provides a Mongolian-Chinese neural machine translation method based on constraint-enhanced alignment. First, a reference translation is constructed through existing Mongolian-Chinese parallel corpora, then Chinese constraint words are randomly selected, and Mongolian corresponding to the Chinese constraint is split into stems and affixes to obtain Mongolian constraint words, thereby improving the speed of the model searching Mongolian corresponding to Chinese; then, a pseudo source sentence is constructed by adding Chinese and Mongolian constraint words to the end of the source sentence, and the pseudo source sentence is mixed with the original parallel corpus to form a synthetic parallel corpus, thereby expanding the scale of training data and alleviating the problem of scarcity of Mongolian-Chinese parallel corpora; then, the source sentence and the constraint words are encoded respectively to avoid mutual interference and dominance between the source sentence and the constraint words, and the problem of poor Mongolian-Chinese translation quality is alleviated by balancing the weights of two different types of information; finally, a word alignment module is inserted into the decoder, and the model finds the corresponding parts of the source word and the constraint words according to the semantic information and structural information of Mongolian, and prioritizes the constraint words to increase the proportion of the constraint words appearing in the target text.

[0062] like Figure 1 As shown, the figure is an overall flow chart of the present invention. The present invention first pre-processes the Mongolian-Chinese parallel corpus, then randomly and dynamically samples Chinese constraint words from the reference translation (i.e., the Chinese reference translation), splits the Mongolian corresponding to the Chinese constraint words into stems and affixes, and obtains Mongolian constraint words; then, both Mongolian and Chinese constraint words are attached to the source sentence to construct a pseudo source sentence; finally, the source sentence and the constraint are encoded separately using a dual encoder-decoder framework, and a decoder containing an alignment module is used to perform a decoding operation to output the final target language.

[0063] The main steps of the present invention are described in detail below:

[0064] Step 1: Use byte pair encoding (BPE) technology to segment Mongolian and Chinese data, and preprocess the corpus using an open source model.

[0065] Use BPE to segment the Mongolian-Chinese parallel corpus. The words in each sentence are segmented into sub-word units, so that rare words and out-of-vocabulary words can be broken down into smaller and more frequent sub-word units. This alleviates the problem of low-frequency words and out-of-vocabulary words in Mongolian and enhances the translation ability of the model. The corpus preprocessing part divides the existing dataset into three parts: training set, validation set, and test set. The dataset division is determined by random sampling. Randomly sample 10% of the corpus as the validation set, which is used to adjust the parameters of the model during training; randomly sample 10% of the corpus as the test set, which is used to evaluate the performance of the model after training is completed. The remaining 80% of the corpus is used as the training set to train the Mongolian-Chinese neural machine translation model.

[0066] Step 2: Randomly and dynamically sample from the reference translation to obtain Chinese constraints, and split the corresponding Mongolian of the Chinese constraints into the form of stems and affixes as Mongolian constraints.

[0067] Since Mongolian has complex morphological changes and there is a small amount of Mongolian-Chinese parallel corpus, many words appear at low frequencies in the training data, and there are even many out-of-vocabulary words, which leads to low-quality Mongolian-Chinese translation results. To solve this problem, the present invention adopts a random dynamic sampling method. This method introduces different low-frequency words and out-of-vocabulary words from the reference translation during each training process. Different from a fixed bilingual dictionary, dynamic sampling is not limited to existing high-frequency words, but can make full use of potential low-frequency words and out-of-vocabulary words in the Mongolian-Chinese parallel corpus, reducing the translation error rate of low-frequency words and out-of-vocabulary words.

[0068] The specific approach is as follows: During the actual training process, first randomly set the number of constraints K as the number of constraint words required for training. Obtain a set of reference translations from the existing Mongolian-Chinese parallel corpus, randomly select K Chinese words from each reference translation as the Chinese constraint words for training. Then use the Mongolian-Chinese parallel corpus to translate the selected Chinese constraints into corresponding Mongolian constraints, and decompose the Mongolian into stems and affixes to obtain Mongolian constraints.

[0069] There are many low-frequency words and out-of-vocabulary words in Mongolian. High-frequency words in Mongolian have been learned well by the model, while low-frequency words and out-of-vocabulary words with certain information content affect the overall translation quality of the model. Therefore, the present invention analyzes the training corpus, counts the list of words with higher frequencies in Chinese and Mongolian, and removes the high-frequency constraint words obtained by random dynamic sampling, so that the attention of the model can be more concentrated on low-frequency words and out-of-vocabulary words, improving the diversity and semantic accuracy of translation.

[0070] The reason for splitting the corresponding Mongolian of Chinese into stems and affixes is that Mongolian is an alphabetic language with rich morphological changes and complex rules. After splitting, the speed of searching for the corresponding source word in the source sentence by the target constraint word will be improved. For example, the Chinese word "拿" corresponds to the Mongolian word and They are all composed of stems Different affixes are connected to form Mongolian words, but their basic meaning has not changed, only the form and time have changed. In the Mongolian-Chinese neural machine translation word alignment, if these words with different forms but the same meaning are regarded as different words, the Mongolian-Chinese word alignment space will increase, which will lead to slow decoding speed. The split stem contains the main meaning of the Mongolian word, and the split affix contains information such as tense and voice. Therefore, searching for the source word corresponding to the target word through the stem can help alleviate this problem.

[0071] The selection of constraint words is to generate a set of candidate constraint words by referring to the translation, dynamically screen the set, and use the semantic similarity calculation method to determine the semantic similarity between the candidate constraint words. After each word selection, the selected word is marked as used to avoid selecting semantically repeated words. The similarity between candidate words is calculated as follows:

[0072]

[0073] where v 1 、v 2 are the word vectors of two words.

[0074] Step 3: Append the selected Chinese constraints and Mongolian constraints to the end of the source sentence to generate a pseudo source sentence, and mix it with the original Mongolian-Chinese parallel corpus to construct a Mongolian-Chinese synthetic parallel corpus.

[0075] Firstly, the selected Chinese constraints and Mongolian constraints are appended to the source sentence (Mongolian), and a unified concatenation rule is designed for the constraint words to determine the order of placing the constraints as Chinese constraint words first and Mongolian constraint words later, thus constructing a pseudo-source sentence; then, the pseudo-source sentence is mixed with the original corpus to construct a Mongolian-Chinese synthetic parallel corpus and expand the scale of training data.

[0076] The constraint words are attached to the source sentence, randomly disrupting the order of Chinese constraint words to simulate disordered constraints, allowing the model to freely adjust the order of words when generating a translation, and generating a translation that conforms to Chinese grammatical rules and includes all constraint words.

[0077] Step 4: Use special markers to distinguish the source sentence and the constraint in the pseudo source sentence.

[0078] Three embedding vectors are introduced before the pseudo source sentence is input into the encoder to distinguish the source sentence from the constraints.

[0079] First, we introduce word embedding vectors to map each word in the Mongolian source sentence and each word in the Chinese and Mongolian constraints to a high-dimensional vector space to capture their semantic features. The word embedding vectors provide a detailed semantic representation of the source language and constraints, helping the model understand the content of Mongolian and its corresponding lexical constraints, so that the constraints can be accurately applied when generating Chinese. The word embedding vector is represented as:

[0080]

[0081] Then, we introduce the position embedding vector to provide the position information of the word in the sentence, help the model understand the order and structure of the vocabulary, and ensure that the generated Chinese can retain the correct grammar and order. For constraints, position embedding helps the model understand the specific position and context of the constraints so that the constraints can be better integrated into the translation process. In practical applications, the position embedding vector distinguishes the source sentence and the constraint by marking the constraint starting from the same number greater than the maximum source sentence length. The position embedding vector is expressed as:

[0082]

[0083] Finally, the fragment embedding vector is introduced. Each word in the source Mongolian sentence has the same fragment embedding vector, and different constraint words will be assigned a unique fragment embedding vector. The fragment embedding vector of the source sentence is different from the fragment embedding vector of the constraint, so as to distinguish the source sentence from the constraint vocabulary. The fragment embedding vector is expressed as:

[0084]

[0085] The three embedding vectors are summed to get the final input vector of the encoder:

[0086] E=E tok +E pos +E seg (5)

[0087] Where E tok is the word embedding vector, which represents the semantic information of the word; E pos is the position embedding vector, indicating the position of the word in the sentence; E seg is the fragment embedding vector, indicating whether the word belongs to the source sentence or the constraint part.

[0088] The final pseudo-source sentence is as follows:

[0089]

[0090] Where x is the source sentence x={x 1 ,x 2 ,...,x s }, c is a constraint word c=(c 1 ,c 2 ,...,c N ); Special mark <sep>Distinguish between source sentences and constraint words, and multiple constraint words, and set an end marker at the end of the pseudo source sentence <eos>.

[0091] like Figure 2 As shown, Mongolian sentences ("The book contains a wealth of knowledge."), assuming that the randomly selected Chinese constraint words are "rich" and "knowledge", the Mongolian words corresponding to the two constraint words are split into stems and affixes. Specifically (enriched) split into stems and affixes (knowledge) split into stems and affixes Then, "enrichment", and "knowledge", and As a constraint attached to the source sentence, separated by a separator, the resulting pseudo source sentence will be " <sep>Rich <sep> <sep> <sep>Knowledge <sep> <sep> <eos>"First, the pseudo source sentence" is embedded into the Token embeddings layer. <sep>Rich <sep> <sep> <sep>Knowledge <sep> <sep> <eos>"Word vectorization; then in the Positional embeddings layer, the source sentence is first marked with consecutive numbers. The maximum length of the source sentence is 3, and the constraints are marked starting from the same number greater than this number, for example Figure 2 The constraint words "rich" and "knowledge" in the source sentence and their corresponding Mongolian constraints will be marked as a number much larger than the length of the source sentence, 1025. Finally, the source sentence is given a segment embedding layer. Each token in is marked with the same fragment, and each token in the constraint word is marked with a different fragment. Figure 2 The segments in the source sentence are marked with 0, and each constraint word is marked with 1, 2, ..., 7 in order.

[0092] Step 5: Use dual encoders to encode the source Mongolian sentence and the constraint words respectively.

[0093] like Figure 3 As shown, the Mongolian source sentence and the constraint words are sent to two encoders for encoding after passing through the three embedding layers described in step 4. Figure 3 The source sentence encoder is on the left, and the constraint word encoder is on the right. The hidden state output by the source sentence encoder is concatenated with the hidden state output by the constraint word encoder to obtain the final hidden state of the pseudo source sentence.

[0094] The hidden state output by the source sentence encoder, that is, the context of the source sentence, is expressed as:

[0095] H tok =ENCODER tok (E tok )(6) Among them is the vector representation of the source sentence, is the word vector of the i-th word in the source sentence, and the hidden state output by the constraint word encoder, that is, the context of the constraint word is expressed as:

[0096] H cons =ENCODER cons (E cons )(7)

[0097] in is the vector representation of M constraints, is the word vector of the jth word in the constraint words.

[0098] The joint hidden state of the final output of the dual encoder is expressed as:

[0099] H aug =ENCODER joint (E a ' ug )(8)

[0100] Where E′ aug =[H tok ,E cons ] is the concatenation vector of Mongolian and constraint words in the pseudo source sentence.

[0101] Through the above two encoders, the source sentence and the constraint words are encoded separately.

[0102] The reason for using two encoders is that neural networks cannot balance two different types of information well. Therefore, integrating two different types of information into the same encoder will cause one type of information to dominate the other, and the model's ability to utilize constraints will decrease. Separately encoding the source sentence and the constraint will firstly allow the two encoders to focus on encoding the source information and the constraint information respectively, without interfering with each other during the encoding process; secondly, in actual applications, constraints can take many different forms. By encoding them separately, the model can handle different constraints more flexibly and make appropriate adjustments during the translation process; finally, separate encoding can adjust the parameters of the two encoders and optimize them to different degrees.

[0103] Step 6: Add a word alignment module to the decoder and perform decoding operations.

[0104] Since the Transformer model has a poor ability to capture word alignment information using the attention mechanism, it is not reliable to use the Transformer's multi-head attention to guide word alignment. For the constraint-enhanced Mongolian-Chinese neural machine translation task, the goal is to accurately translate the source sentence into Mongolian and include multiple Chinese constraint words in the source sentence. However, when the constraint words are attached to the source sentence, the traditional attention model does not provide explicit word alignment information for Mongolian and its corresponding Chinese constraints during decoding. Therefore, it is impossible to know the position of the Chinese constraint words in the source Mongolian sentence, resulting in slow decoding speed of the Mongolian-Chinese neural machine translation model.

[0105] The word alignment module allows the decoder to align the constrained vocabulary with the source sentence vocabulary when generating Chinese. The model can select the vocabulary in the target sentence according to the specific vocabulary in the source sentence to ensure that the translation result meets the specific vocabulary constraints. At the same time, the word alignment module provides additional contextual information for the decoder, helping the model to consider the context of each Mongolian word in the source sentence when generating Chinese, reducing the uncertainty in the translation process. Therefore, the present invention inserts a word alignment module in the Transformer decoder.

[0106] The word alignment module calculates the word alignment score output by the module and selects the word with the lowest word alignment error rate (Alignment Error Rate, AER) as the final target text, thereby improving the accuracy of word alignment and the translation quality of the model.

[0107] The specific process is as follows:

[0108] The first step is to determine the position of the word alignment module in the decoder. In order to select the decoder layer that is most suitable for the word alignment module, the present invention uses a given Mongolian-Chinese parallel sentence pair to<x,y> In the case of , first train a Mongolian to Chinese model θ x→y and a Chinese to Mongolian model θ y→x , then extract word alignment information from Mongolian to Chinese and Chinese to Mongolian from these two models respectively, and use the word alignment information from Chinese to Mongolian as the alignment reference to evaluate the AER from Mongolian to Chinese on the validation set. Calculate the word alignment score:

[0109]

[0110] in is the output of the decoder at layer l-1, H i is the hidden state of the dual encoder output.

[0111] The word alignment information is integrated and passed to the last layer of the decoder to generate the probability distribution of the target word.

[0112]

[0113] Among them C t It is a contextual representation that incorporates alignment information. It is passed to the last layer of the decoder to generate the probability distribution of the target word.

[0114] For each model, l different word alignment information can be obtained from l decoder layers, so there are l×l AER scores in total. Finally, the layer with the lowest AER is selected as the position of the word alignment module in the decoder. The specific representation is as follows:

[0115]

[0116] Among them l b It is the layer that extracts the best word alignment information, that is, the layer with the lowest AER score.

[0117] The present invention places the word alignment module in the penultimate layer of the decoder. Because in the Mongolian-Chinese neural machine translation model, the bottom layer of the decoder processes local features, the middle layer gradually expands the context information, and the high layer comprehensively processes the global semantic information. The last layer is the output layer, which outputs the final translation result. If the word alignment module is placed in this layer, alignment noise will be introduced, interfering with the word prediction process of the output layer. The penultimate layer can ensure that the alignment information is fully utilized for dynamic adjustment before output, and the existing context information and global information are used to the maximum extent to correct the alignment deviation and reduce alignment errors.

[0118] The second step is to insert the word alignment module, such as Figure 4 As shown. The input of the inserted word alignment module includes: the hidden state H output by the encoder aug =ENCODER joint (E′ aug ), the output of the decoder self-attention sublayer And the last word y t The embedding vector e(y t ).

[0119] Before the word alignment module is inserted, the traditional Transformer attention calculation method is as follows:

[0120]

[0121] in is the learnable weight matrix, d is the dimension of the attention head, are query, key, and value matrices respectively, is the product of the query vector and the key vector, is the output of the attention layer.

[0122] Calculation method after inserting the word alignment module:

[0123]

[0124] Where n is the number of attention heads, t is the position of the target marker, refers to the output of the self-attention sublayer of the lth layer of the decoder at the tth target token, l∈{1,...,L}, K n 、V n are query, key, and value matrices respectively, P(a t |x,y ≤t ) is the product of the query vector and the key vector, d is the dimension size of the attention head, is the output of the attention layer, is a learnable weight matrix.

[0125] The final objective function is calculated as follows:

[0126]

[0127] Where T is the length of the target sentence, S is the length of the source sentence, and S i,j The i-th target word corresponds to the j-th source word.

[0128] The e(y t ) is used as the input of the alignment module because in the process of Mongolian-Chinese neural machine translation, the model gradually aligns the source word and the target word through the attention mechanism. This tag carries key contextual information or the main idea of ​​the whole sentence. Using it as the input of the alignment module first helps the model better understand the semantic structure of the whole sentence. The model can dynamically adjust its attention to different parts of the source language sentence. Secondly, it allows the model to more accurately determine the best correspondence and reduce ambiguity in word alignment. Finally, by introducing more contextual information, the performance and generalization ability of the model can be improved, making it more stable and reliable in complex contexts.

[0129] Step 7: The output of the dual encoder is used as the input of the decoder to train the final Mongolian-Chinese neural machine translation model.

[0130] Reference Figure 5 ,The overall architecture diagram of the model. Before the pseudo source sentence is input into the encoder, it passes through three embedding layers, namely word embedding, position embedding, and fragment embedding. Then the source sentence and constraint word are put into two encoders for multi-layer encoding, from which the hidden state is extracted as the context vector of the source sentence and constraint word, which is merged to generate a comprehensive encoding representation. The hidden state output by the encoder is used as the input of the decoder, and then the source sentence context and constraint word are decoded. The decoder calculates the alignment score of Mongolian and Chinese through the word alignment module, and finally outputs the target text according to the context and alignment information.

[0131] The conditional probability calculation method of the Mongolian-Chinese neural machine translation model is as follows:

[0132]

[0133] Where X = {x 1 ,x 2 ,...,x s } is the source sentence, Y={y 1 ,y 2, ....,y T } is the target sentence, C=(C 1 ,C 2 ,...,C N ) is a binding word.

[0134] The final loss function calculation method of the Mongolian-Chinese neural machine translation model is as follows:

[0135]

[0136] in It is a pseudo-source sentence.

[0137] Step 8: Use the final Mongolian-Chinese neural machine translation model to perform Mongolian-Chinese translation.

[0138] refer to Figure 6 ,enter" (Chinese translation: Everyone discussed the method of acquiring knowledge from books)". Assuming that the randomly selected constraint words are "discussion" and "knowledge", we get the pseudo source sentence " <sep>discuss <sep> <sep> <sep>Knowledge <sep> <sep> <eos>", and use the trained model to output the translation result.

[0139] The present invention adopts a constraint enhancement alignment method to perform Mongolian-Chinese neural machine translation. First, the scale of Mongolian-Chinese parallel corpus is expanded by the constraint enhancement method. Then, the Mongolian corresponding to the selected Chinese constraint words is split into stems and affixes as Mongolian constraints, and both the Chinese and Mongolian constraints are attached to the Mongolian source sentence. Then, two encoders are used to encode the source sentence and the constraint words respectively, and the result is input into a decoder of an insertion word alignment module for decoding. Finally, the target text is output.< / eos> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / eos> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / eos> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / eos> < / sep> < / eos> < / sep>

Claims

1. A Mongolian-Chinese neural machine translation method based on constraint-enhanced alignment, characterized in that: The steps include: Step 1, segmenting the Mongolian-Chinese parallel corpus and preprocessing the segmented corpus; Step 2, randomly and dynamically sampling the Chinese constraints from the reference translation, splitting the Mongolian corresponding to the Chinese constraints into stems and affixes as Mongolian constraints; Step 3, adding the Chinese constraint and the Mongolian constraint to the end of the source sentence to generate a pseudo source sentence, and mixing them with the original Mongolian-Chinese parallel corpus to construct a Mongolian-Chinese synthetic parallel corpus; Step 4, distinguish the source sentence and the constraint in the pseudo source sentence with special marks; Step 5, use a dual encoder to encode the source sentence and the Chinese constraints and Mongolian constraints respectively; Step 6, add a word alignment module to the decoder and perform decoding operation; Step 7: The output of the dual encoder is used as the input of the decoder to train the final Mongolian-Chinese neural machine translation model; Step 8: Use the final Mongolian-Chinese neural machine translation model to perform Mongolian-Chinese translation.

2. According to the Mongolian-Chinese neural machine translation method based on constraint enhanced alignment according to claim 1, it is characterized in that: The step 1 uses byte pair encoding technology to segment the Mongolian-Chinese parallel corpus, and segments the words in each sentence into sub-word units; uses an open source model to pre-process the segmented corpus and divide it into a training set, a validation set, and a test set.

3. According to the Mongolian-Chinese neural machine translation method based on constraint enhanced alignment as described in claim 1, it is characterized in that: In step 2, firstly, the number of constraints K is randomly set, K words are randomly selected from the reference translation as Chinese constraints, then the Mongolian translation corresponding to the Chinese constraints is obtained, and then the Mongolian is decomposed into stems and affixes, and finally the Mongolian constraints are obtained.

4. The Mongolian-Chinese neural machine translation method based on constraint-enhanced alignment according to claim 1 is characterized in that: In step 3, the selected Chinese constraints and Mongolian constraints are first added to the source sentence, and the order of placing the constraints is Chinese constraints first and Mongolian constraints second, to construct a pseudo source sentence, and then the pseudo source sentence is mixed with the original Mongolian-Chinese parallel corpus to construct a Mongolian-Chinese synthetic parallel corpus.

5. According to the constraint-enhanced alignment-based Mongolian-Chinese neural machine translation method of claim 1, it is characterized in that: In step 4, firstly, a word embedding vector is introduced to encode the source sentence, Chinese constraints and Mongolian constraints into a vector form; secondly, a position embedding vector is introduced to capture the order and position relationship of words in the source sentence; finally, different fragment embedding vectors are introduced for the source sentence, Chinese constraints and Mongolian constraints to distinguish the source sentence part and the constraint vocabulary part; the three embedding vectors are summed to obtain the final input vector of the encoder: E=E tok +E pos +E seg Where E tok is the word embedding vector, which represents the semantic information of the word; E pos is the position embedding vector, indicating the position of the word in the sentence; E seg is the segment embedding vector, indicating whether the word belongs to the source sentence or the constraint part; The final pseudo-source sentence is as follows: Where x is the source sentence, c1,c2,...,c N Indicates the 1st to Nth constraints, special mark < sep > Used to distinguish between source sentences and constraints, and between constraints. < eos > It is the end marker of the pseudo source sentence.

6. The Mongolian-Chinese neural machine translation method based on constraint enhanced alignment according to claim 1, characterized in that: In step 5, the dual encoder is a source sentence encoder and a constraint word encoder. The source sentence encoder encodes the semantic and syntactic structure of the source sentence, i.e., Mongolian, to generate a context representation H of the source sentence. tok =ENCODER tok (E tok ), the constraint word encoder encodes the Chinese constraint and the Mongolian constraint to generate a context representation H of the constraint word cons =ENCODER cons (E cons ), the dual encoder outputs the final joint hidden state H aug =ENCODER joint (E a ' ug ); in is the vector representation of the source sentence, is the word vector of the i-th word in the source sentence, is the vector representation of M constraints, is the word vector representation of the jth constraint, E a ' ug =[H tok ,E cons ] is the Mongolian and constraint concatenation vector in the pseudo source sentence, ENCODER joint It is the joint encoding of the source sentence encoder and the constraint word encoder.

7. The Mongolian-Chinese neural machine translation method based on constraint enhanced alignment according to claim 6 is characterized in that: The source sentence encoder and the constraint word encoder both use Transformer encoders. The source sentence encoder first passes through an embedding layer to encode Mongolian words into vectors, and then a multi-head self-attention layer focuses on the correlation between different positions in the input sequence. The output of the multi-head self-attention layer is passed to a feedforward neural network layer, which captures richer Mongolian word features. The constraint word encoder first encodes the Mongolian constraints and Chinese constraints into a vector form before encoding them, and then the multi-head self-attention layer receives the Mongolian constraint words, Chinese constraint words and the hidden state output by the source sentence encoder, pays attention to other words in the input sentence when encoding each word, and obtains the hidden state of the dual encoder through the output of the feedforward neural network layer.

8. The Mongolian-Chinese neural machine translation method based on constraint enhanced alignment according to claim 1, characterized in that: The step 6, adding a word alignment module to the decoder, is implemented as follows: The first step is to determine the position of the word alignment module in the decoder; Given a Mongolian-Chinese parallel sentence pair < x,y > In the case of , first train a Mongolian to Chinese model θ x→y and a Chinese to Mongolian model θ y→x , then extract word alignment information from Mongolian to Chinese and from Chinese to Mongolian from the two models respectively, and use the word alignment information from Chinese to Mongolian as an alignment reference, and evaluate the alignment error rate from Mongolian to Chinese on the validation set; each model obtains l different word alignment information from l decoder layers, and selects the decoder layer with the lowest alignment error rate as the position of the word alignment module in the decoder; The second step is to insert the word alignment module; The input of the word alignment module includes: the joint hidden state H output by the dual encoder aug =ENCODER joint (E′ aug ), the output of the decoder self-attention sublayer And the last word y t The embedding vector e(y t ).

9. The Mongolian-Chinese neural machine translation method based on constraint enhanced alignment according to claim 1 or 8, characterized in that: The calculation method after inserting the word alignment module in step 6 is: Where n is the number of attention heads, t is the position of the target marker, refers to the output of the self-attention sublayer of the lth layer of the decoder at the tth target token, l∈{1,...,L}, Kn、V n are query, key, and value matrices respectively, P(a t| x,y ≤t ) is the product of the query vector and the key vector, d is the dimension size of the attention head, is the output of the attention layer, is a learnable weight matrix; The final objective function is calculated as follows: Where T is the length of the target sentence, S is the length of the source sentence, and S i,j The i-th target word corresponds to the j-th source word.

10. The Mongolian-Chinese neural machine translation method based on constraint enhanced alignment according to claim 1, characterized in that: In step 7, before the pseudo source sentence is input into the dual encoder, it passes through three embedding layers, namely word embedding, position embedding and fragment embedding. Then the source sentence and the constraint are respectively put into the two encoders for encoding, and the hidden states extracted therefrom are used as the context vectors of the source sentence and the constraint word. Then the hidden states output by the dual encoder are used as the input of the decoder. The decoder calculates the alignment scores of Mongolian and Chinese through the word alignment module, and finally outputs the target text according to the context and alignment information.

Images