A neural machine translation method based on source language reordering

Abstract

The invention discloses a neural machine translation method based on source language reordering, and the steps are: pre-processing training data to obtain reordering information of subwords and source language corresponding to the training data; sending source language subword information vectors and source language position vectors to feedforward neural networks respectively to obtain intermediate vectors containing information of the two; mapping the reordering information of the source language into reordering position vectors, using the reordering position vectors to supervise the generation of intermediate vectors containing source language subword information and source language position information, and introducing the reordering information into the intermediate vectors by updating the intermediate vectors through relative entropy constraints; mapping the intermediate vectors, source language word vectors and source language position vectors through different feature maps, and combining through nonlinear functions to obtain the input of a model encoder, thereby realizing translation from the source language to the target language. The invention helps the decoder model capture the input information on which the generation of the target word depends, can effectively utilize the reordering features of the source language, and improve the translation quality of the model.

Description

Technical Field

[0001] The present invention relates to a neural machine translation technology, in particular to a neural machine translation method based on source language reordering. Background Art

[0002] Machine translation, also known as automatic translation, is the process of using computers to convert one natural language (source language) into another natural language (target language). It is a branch of computational linguistics and one of the ultimate goals of artificial intelligence, and has important scientific research value. Machine translation technology can be roughly divided into two methods, namely rule-based machine translation methods and data-driven machine translation methods. Data-driven machine translation methods can be further divided into statistical machine translation methods and neural machine translation methods.

[0003] With the development of machine learning technology, neural machine translation based on deep learning has gradually emerged. In neural machine translation, word strings are represented as real number vectors, that is, distributed vector representation. The translation process is not performed on discrete words and phrases, but calculated on real number vector space. Usually, machine translation is regarded as the transformation of one sequence to another. In neural machine translation, the sequence-to-sequence transformation process can be implemented by the encoder-decoder framework. Among them, the encoder encodes the source language sequence and extracts the information in the source language for distributed representation, and then the decoder converts this information into an expression in another language.

[0004] As the most mainstream neural machine translation model, the Transformer model still uses the encoder-decoder structure. The encoder or decoder is composed of multiple layers of the same structure, each of which contains a self-attention sublayer (Self-Attention) and a feedforward network sublayer (Feed Forward Network). The decoder is similar to the encoder, but there is an encoder-decoder attention sublayer (Encoder-Decoder Attention) between the self-attention sublayer (Self-Attention) and the feedforward network sublayer (Feed Forward Network), which is responsible for processing the output of the encoder. In order to enhance the expressiveness of the model and the stability of the training process, each sublayer will have residual connection (Add) and layer regularization (LayerNorm) operations.

[0005] Human language is structured, and the structures of different languages ​​often vary greatly. In the era of statistical machine translation, adjusting the position of the source language according to the order of the target language will improve the performance of the statistical machine translation system. In the era of neural machine translation, the acquisition of reordering information and the effective introduction of external reordering information have attracted people's attention. Position information is an important component of the self-attention network. The encoder and decoder networks model the position of the source language and the target language respectively. However, the word order in different languages ​​is often quite different, and modeling cross-language positional relationships can effectively improve the performance of the translation system. By changing the order of the source language according to the order of the target language, the cross-language position information corresponding to the source language and the target language can be obtained, which is also called reordering information.

[0006] In practical applications, the source language reordering information cannot be obtained directly. In the era of statistical machine translation, people obtained the source language reordering information by pre-training the pre-ordering model, which was a cumbersome process and not convenient for application and promotion. In the era of neural machine translation, people use neural networks to train the pre-ordering model, which also has these problems.

[0007] Since the original transformer model only has encoder and decoder modules and no model structure specifically responsible for adjusting cross-language information, the neural machine translation model based on the transformer structure cannot model cross-language information. In the current neural translation model, the lack of cross-language modeling capabilities leads to unsatisfactory translation quality. By introducing the source language's reordering information about the target language, the translation quality of the neural translation model can be improved. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a neural machine translation method based on source language reordering. Before the input is sent to the Transformer model, a reordering supervision signal is set to enable the model to learn the reordering information of the source language to guide the generation of the translation.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0010] A neural machine translation method based on source language reordering comprises the following steps:

[0011] 1) Preprocess the training data to obtain subwords corresponding to the training data; obtain the subword correspondence between the source language and the target language through a word alignment tool, adjust the source language position according to the target language position information, and obtain the reordering information of the source language;

[0012] 2) The source subword information vector and the source position vector are sent to the feedforward neural network respectively, and the intermediate vector containing the information of the two is obtained through a nonlinear function;

[0013] 3) Mapping the source language reordering information into a reordering position vector, using the reordering position vector to supervise the generation of an intermediate vector containing source language subword information and source language position information, and introducing the reordering information into the intermediate vector by constraining the update of the intermediate vector with relative entropy;

[0014] 4) The intermediate vector, source word vector and source position vector are mapped through different features and combined through nonlinear functions to obtain the input of the model encoder, thereby realizing the translation from source language to target language.

[0015] In step 2), the source subword information vector and the source position vector are respectively sent to the feedforward neural network to obtain the intermediate latent variables as follows:

[0016] x=xW1+b1

[0017] p=pW2+b2

[0018] Where x is the source subword information vector, p is the source subword position vector, W1∈R d×d ,W2∈R d×d , b1∈R d , b2∈R d is the parameter of the feedforward neural network. The dimensions of the final input and output are consistent. d×d Represents a matrix dimension of d×d, R d Represents a matrix dimension of d;

[0019] The source subword information intermediate variable and the source subword position information intermediate variable are combined through nonlinear functions:

[0020] pse_reo=sigmoid(x+p)

[0021] Where x is the source subword information vector, p is the source subword position vector, pse_reo is the pseudo reordering vector; sigmoid is a nonlinear activation function that constrains the output space to be between 0 and 1.

[0022] Step 3) includes:

[0023] 301) The reordering information is introduced in the form of position coding, and the reordering information is implicitly introduced into the model. The reordering position coding formula is:

[0024]

[0025]

[0026] Among them, RE is a one-dimensional matrix, which indicates the reordering information; reo indicates the position of the source language after reordering; d model represents the dimension of the reordered vector; i represents the position dimension of the reordered vector;

[0027] The supervisory signal reordering vector is passed through a nonlinear activation function to obtain a reordering matrix with the same output space:

[0028] RE=sigmoid(RE)

[0029] Among them, RE represents the reordering matrix, sigmoid is a nonlinear activation function, and the output space is constrained to be between 0 and 1;

[0030] By constraining the pseudo reordered vector and the reordered vector by relative entropy, the distribution of the two is made closer. The constraint term calculation formula is:

[0031] reo_loss=(KL(pse_reo||RE)+KL(RE||pse_reo)) / 2

[0032] Among them, reo_loss is the bidirectional relative entropy loss, KL is the relative entropy calculation formula, pse_reo is the pseudo reordering vector, and RE is the reordering vector.

[0033] Step 4) includes:

[0034] 401) The generated pseudo-adjustment sequence vector, position vector and subword information vector are nonlinearly combined, and the combination formula is:

[0035] x=x+tanh(pW3+pse_reoW4)

[0036] Among them, x is the source subword information vector, p is the source subword position vector, pse_reo is the source pseudo reordering vector, tanh is a nonlinear activation function, and the output space is between -1 and 1;

[0037] 402) The target language context vector is obtained through the model encoder and the model decoder in turn, the target language translation is decoded, and the gradient is updated through the cross entropy loss function to optimize the model weights.

[0038] The present invention has the following beneficial effects and advantages:

[0039] 1. The present invention introduces the reordering information of the source language into neural machine translation, which improves the learning ability of the neural machine translation model for the linguistic correspondence contained in the two languages, and improves the shortcomings of the existing neural machine translation paradigm in modeling cross-language pairs.

[0040] 2. The source language reordering module introduced in the present invention is independent of the Transformer model. The module is located before the encoder input and has a light structure and is simple and easy to use.

[0041] 3. The source language reordering module introduced in the present invention effectively improves the translation quality of the model with almost no increase in model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a diagram of the original Transformer model in the present invention;

[0043] Figure 2 It is a diagram of the improved Transformer model based on source language reordering in the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings.

[0045] The present invention aims at the traditional Transformer model (such as Figure 1 As shown in the figure, the source word reordering information is not effectively introduced in the source word reordering information. A source word reordering information incorporation method is proposed, and a module based on the source word reordering is designed (as shown in the figure). Figure 2 The method successfully introduces the source language reordering information into the machine translation task, generates the source language reordering features, and fuses them with the source language position information and source language subword information to form the input of the encoder model. Through forward calculation, the translation is generated.

[0046] The present invention provides a neural machine translation model based on source language reordering, comprising the following steps:

[0047] 1) Preprocess the training data to obtain subwords corresponding to the training data; obtain the subword correspondence between the source language and the target language through a word alignment tool, adjust the source language position according to the target language position information, and obtain the reordering information of the source language;

[0048] 2) The source subword information vector and the source position vector are sent to the feedforward neural network respectively, and the intermediate vector containing the information of the two is obtained through a nonlinear function;

[0049] 3) Mapping the source language reordering information into a reordering position vector, using the reordering position vector to supervise the generation of an intermediate vector containing source language subword information and source language position information, and introducing the reordering information into the intermediate vector by constraining the update of the intermediate vector with relative entropy;

[0050] 4) The intermediate vector, source word vector and source position vector are mapped through different features and combined through nonlinear functions to obtain the input of the model encoder, thereby realizing the translation from source language to target language.

[0051] In step 1), the bilingual training data is firstly cut into subword data by a subword segmentation tool, wherein the subword segmentation word list size is 32000, so that the correspondence between the source language and the target language is more statistically significant, and then the subword data is aligned by a word alignment tool fast align, wherein the statistical word alignment tool uses default parameters when aligning bilingual data to obtain the correspondence between the subwords of the source language and the target language. Then, the order of the subwords of the source language is rearranged according to the order of the subwords of the target language and the correspondence between the subwords of the source language and the target language, thereby obtaining the reordering information of the source language.

[0052] In step 2), the source subword information vector and the source position vector are respectively sent to the feedforward neural network to obtain the intermediate latent variables about the source subword information and the intermediate latent variables about the source position information, as follows:

[0053] x=xW1+b1

[0054] p=pW2+b2

[0055] Where x is the source subword information vector, p is the source subword position vector, W1∈R d×d ,W2∈R d×d , b1∈R d , b2∈R d is the parameter of the feedforward neural network. The dimensions of the final input and output are consistent. d×d Represents a matrix dimension of d×d, R d Represents a matrix dimension of d;

[0056] The source subword information intermediate variable and the source subword position information intermediate variable are combined through a nonlinear activation function to obtain a pseudo reordering vector containing reordering information:

[0057] pse_reo=sigmoid(x+p)

[0058] Where x is the source subword information vector, p is the source subword position vector, pse_reo is the pseudo reordering vector; sigmoid is a nonlinear activation function that constrains the output space to be between 0 and 1.

[0059] Step 3) includes:

[0060] 301) Encode the reordering information in the traditional trigonometric position coding method, and implicitly integrate the reordering information into the neural translation model through the trigonometric position coding method. The reordering position coding formula is:

[0061]

[0062]

[0063] Among them, RE is a one-dimensional matrix, which indicates the reordering information; reo indicates the position of the source language after reordering, and d model represents the dimension of the reordered vector; i represents the position dimension of the reordered vector;

[0064] The supervisory signal reordered vector is subjected to a nonlinear activation function to obtain a reordered matrix with the same output space as the pseudo reordered vector:

[0065] RE=sigmoid(RE)

[0066] Among them, RE represents the reordering matrix, sigmoid is a nonlinear activation function, and the output space is constrained to be between 0 and 1;

[0067] By constraining the pseudo reordered vector and the reordered vector with relative entropy, the distribution of the pseudo reordered vector is made closer to the reordered vector, thereby achieving the purpose of the generated pseudo reordered vector containing reordering information. The constraint term calculation formula is:

[0068] reo_loss=(KL(pse_reo||RE)+KL(RE||pse_reo)) / 2

[0069] Among them, reo_loss is the bidirectional relative entropy loss, KL is the relative entropy calculation formula, pse_reo is the pseudo reordered vector, and RE is the reordered vector. Since the relative entropy calculation is not symmetrical, the bidirectional relative entropy calculation is used to measure the difference in distribution between the two, so that the distribution of the pseudo reordered vector is close to the distribution of the reordered vector in a gentle way.

[0070] Step 4) includes:

[0071] 401) The generated pseudo reordering vector is feature fused with the position vector, and the fitting ability of the model output is enhanced through the activation function. The obtained vector contains the position information and reordering information. The vector is linearly combined with the subword information vector to obtain the input of the encoder model. The fusion formula is:

[0072] x=x+tanh(pW3+pse_reoW4)

[0073] Among them, x is the source subword information vector, p is the source subword position vector, pse_reo is the source pseudo reordering vector, tanh is a nonlinear activation function, and the output space is between -1 and 1;

[0074] 402) The input of the fused encoder model is sent to the model encoder to obtain the encoder output, and then the encoder input and output are sent to the model decoder to obtain the decoder output, that is, the target language context vector, and the target language translation is decoded according to the target language context vector. The gradient of the parameters is updated through the cross entropy loss function to optimize the model weights and complete the model training.

[0075] To verify the effectiveness of the method, the present invention applies the neural machine translation model based on source language reordering to the English-German task of the machine translation standard dataset WMT 14. The WMT 14 English-German dataset has approximately 4.5 million training data, 2999 verification data, and 3003 test data. Table 1 shows the BLEU values ​​of the translations of the original Transformer model and the Transformer model based on source language reordering. The BLEU value of the model translation has been significantly improved. The BLEU value is used to evaluate the quality of the translation. The higher the BLEU value, the better the translation quality. This proves that the method proposed in the present invention can effectively improve the translation performance of the model.

[0076] Model WMT 14 England and Germany Original Transformer 27.18 Source Reordering Transformer 27.48

[0077] Table 1 Comparison of experimental results

[0078] The present invention introduces the reordering information obtained by the word alignment tool into the model calculation, which helps the decoder model capture the input information on which the target word is generated. The final experimental results prove that the neural machine translation model using reordering information proposed by the present invention can effectively utilize the reordering features of the source language, thereby improving the translation quality of the model.

Claims

1. A neural machine translation method based on source language reordering, characterized in that The following steps are involved: 1) Preprocess the training data to obtain subwords corresponding to the training data; obtain the subword correspondence between the source language and the target language through word alignment tools, adjust the source language position according to the target language position information, and obtain the reordering information of the source language; 2) The source subword information vector and the source position vector are sent to the feedforward neural network respectively, and the intermediate vector containing the information of the two is obtained through a nonlinear function; 3) Map the source language reordering information into a reordering position vector, use the reordering position vector to supervise the generation of an intermediate vector containing the source language subword information and the source language position information, and introduce the reordering information into the intermediate vector by constraining the update of the intermediate vector with relative entropy; 4) The intermediate vector, source word vector, and source position vector are mapped to different features and combined through nonlinear functions to obtain the input of the model encoder, thereby achieving translation from the source language to the target language; In step 2), the source subword information vector and the source position vector are respectively sent to the feedforward neural network to obtain the intermediate latent variables as follows: in is the source subword information vector, is the source subword position vector, , , , are the parameters of the feedforward neural network, and the dimensions of the final input and output are consistent. Represents the matrix dimension as , Represents the matrix dimension as ; The source subword information intermediate variable and the source subword position information intermediate variable are combined through nonlinear functions: among them is the source subword information vector, is the source subword position vector, is a pseudo reordering vector; It is a nonlinear activation function that constrains the output space between 0 and 1; Step 3) includes: 301) The reordering information is introduced in the form of position coding, and the reordering information is implicitly introduced into the model. The reordering position coding formula is: in, RE is a one-dimensional matrix, indicating the reordering information; reo indicates the position of the source language after reordering; represents the dimension of the reordered vector; i represents the position dimension of the reordered vector; The supervisory signal reordering vector is passed through a nonlinear activation function to obtain a reordering matrix with the same output space: in, RE represents the reordering matrix, It is a nonlinear activation function that constrains the output space between 0 and 1; By constraining the pseudo reordered vector and the reordered vector by relative entropy, the distribution of the two is made closer. The constraint term calculation formula is: in, is the bidirectional relative entropy loss, is the relative entropy calculation formula, is a pseudo reordered vector, is the reordered vector.

2. The neural machine translation method based on source language reordering according to claim 1, characterized in that Step 4) includes: 401) The generated pseudo-adjustment sequence vector, position vector and subword information vector are nonlinearly combined, and the combination formula is: in, is the source subword information vector, is the source subword position vector, is the source pseudo reorder vector, It is a nonlinear activation function, and the output space is between -1 and 1; 402) The target language context vector is obtained through the model encoder and the model decoder in sequence, the target language translation is decoded, and the gradient is updated through the cross entropy loss function to optimize the model weights.

Images