A neural machine translation method based on Transformer model optimization

Abstract

The present invention proposes a neural machine translation method based on Transformer model optimization, establishes a translation model, obtains Chinese and English data sets in News Commentary v12 in WMT17, and normalizes the data; introduces sentence structure information into the translation model, obtains word vectors, and obtains new input sequences; trains the translation model using a sequence that integrates position information and syntactic information, and uses an adam optimizer to optimize and iterate to obtain a neural machine translation model; inputs the test set into the neural machine translation model optimized in step S3 to obtain a translation result; evaluates the translation result and ends the translation; and uses the BLEU value as the evaluation criterion for the translation model. The method can improve the translation accuracy of Chinese-English neural machine translation and improve translation performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation, and in particular, relates to a neural machine translation method based on Transformer model optimization. Background Art

[0002] In the field of natural language processing, machine translation is the earliest research topic. Machine translation (MT) refers to the process of computers translating source language into target language sentences with the same semantics. It is an important branch of natural language processing. Since the first experiment at Georgetown University in the United States in 1954, machine translation has been developed for more than 60 years, and has experienced several ups and downs. Until now, people still have mixed views on machine translation. With the continuous development of my country's economy, the language service industry has also shown a rapid development trend, the language service market has expanded rapidly, and the traditional manual translation model can no longer meet the needs. The development of machine translation has become inevitable. However, compared with manual translation, although machine translation is far more efficient than manual translation and can translate tens of thousands of sentences in a short time. However, the accuracy is not satisfactory. Therefore, the improvement and optimization of machine translation models is crucial to the efficiency and accuracy of language translation.

[0003] There are three main types of machine translation: rule-based machine translation, statistical machine translation (SMT) and neural machine translation (NMT). With the continuous development of deep learning, neural machine translation has outperformed statistical machine translation in most language pairs and has become the mainstream in the field of machine translation. The neural machine translation system uses an encoder-decoder framework and is a Seq2Seq model. In order to improve the accuracy of the machine translation system, many researchers in academia have begun to improve the machine translation model. Some related work has been tried from different angles and different methods have been used to build translation models. In 2017, Vaswani et al. proposed a Transformer model based on the attention mechanism, which greatly improved the training speed and translation quality. However, there are still some solvable problems.

[0004] Neural network-based translation models such as recurrent neural networks are sequential structures that inherently contain the position information of words in a sequence. However, Transfromer does not contain recurrent neural networks (RNN) and convolutional neural networks (CNN). Transformer processes each word in a sentence in parallel, does not have the ability to identify the order of each word, and does not contain any position information. Adding positional embedding (PE) can help the model identify position information. The Sinusoidal positional encoding proposed by Vaswani et al. calculates positional encoding information through a predefined function, does not contain learnable parameters, and is not flexible enough. The relative position encoding proposed by Shaw et al. integrates the relative position representation into the self-attention mechanism of the transformer, which greatly improves the translation performance of the neural machine translation system. However, it sacrifices long-distance position dependence to a certain extent. The position information of each word in the source language sentence can be obtained by integrating the long short-term memory network into the recurrent neural network, and the memory characteristics of the LSTM (Long short-term memory, LSTM) are used to capture long-distance information, making the position encoding information more accurate.

[0005] The neural machine translation model uses an encoder-decoder framework and is an end-to-end model. In neural machine translation, the neural machine translation model treats the source language sentence as a sequence of characters or words, while ignoring the inherent structural information in the language. In previous rule-based machine translation and statistical machine translation, linguistic information plays an irreplaceable role. However, the neural machine translation model framework does not contain any linguistic knowledge. Both the source language and the target language are treated as string sequences and processed in parallel, resulting in the loss of linguistic information. Studies have shown that introducing linguistic knowledge into neural machine translation will greatly improve translation performance. Summary of the invention

[0006] In view of the above problems, the present invention proposes a neural machine translation method based on Transformer model optimization to obtain more accurate translation results.

[0007] The present invention is achieved through the following scheme:

[0008] A neural machine translation method based on Transformer model optimization:

[0009] The method specifically comprises the following steps:

[0010] Step S1, building a translation model, obtaining the Chinese and English datasets in News Commentary v12 in WMT17, and normalizing the data;

[0011] Step S2, introducing sentence structure information into the translation model, obtaining word vectors, and obtaining a new input sequence;

[0012] Step S3, training the translation model using the sequence that integrates the position information and the syntactic information, and using the adam optimizer to perform optimization iterations to obtain a neural machine translation model;

[0013] Step S4, inputting the test set into the neural machine translation model optimized in step S3 to obtain a translation result;

[0014] Step S5, using the BLEU value as the evaluation criterion of the translation model, and ending the translation.

[0015] Furthermore, the specific steps of step S1 are:

[0016] Step S101, the downloaded Chinese-English dataset is from the news field; 50,000 Chinese-English language pairs are extracted from the Chinese-English dataset as a training set; newsdev2017 is used as a validation set, and newstest2017 is used as a test set;

[0017] Step S102, preprocessing the data set, deleting sentences with a length greater than 100 and repeated sentences;

[0018] Step S103, obtain a dictionary based on the training set, and use the unregistered words <unk>express.

[0019] Furthermore, the step S2 specifically includes:

[0020] Step S201, using the Stanford CoreNLP toolkit to obtain the dependency syntax tree of each sentence, introducing the dependency matrix, extracting the source language sentence structure information and integrating it into the translation model;

[0021] Step S202, using the CBOW model to predict the target word according to the dependency words and context words to obtain a word vector;

[0022] Step S203, using the Stanford CoreNLP toolkit to convert the source language sentence into a dependency syntax tree form, and then into a corresponding dependency matrix; the calculation formula is:

[0023]

[0024] D(i,j) is multiplied by the word vector obtained in step S202 to obtain a new input sequence;

[0025] Where i is the current node index, j is the other node index, and w j For other nodes, w i is the current node;

[0026] Step S204, using LSTM to replace the original Transformer model to obtain the position encoding method, and obtaining the output of each time step of LSTM as the position information of each word.

[0027] Furthermore, the step S204 specifically includes:

[0028] LSTM includes forget gate, input gate and output gate;

[0029] Forget gate: decides whether to keep or discard information.

[0030] f t =σ(W f h t-1 +V f X t +b f )

[0031] Input gate: determines whether to update the input information to the cell unit.

[0032] i t =σ(W i h t-1 +V i X t +b i )

[0033] Candidate cell unit,

[0034]

[0035] The latest status of the memory cell node,

[0036]

[0037] Output gate: determines the output of the memory cell state value.

[0038] o t =σ(W o h t-1 +V o X t +b o )

[0039] The output of the final hidden state is

[0040] h t =o t *tanh(c t )

[0041] Where t represents the current time, t-1 represents the previous time, and f t represents the state of the forget gate, i t Indicates the input gate state, o t represents the output gate state, σ represents the sigmoid activation function, tanh represents the tanh activation function, W is the weight matrix of the neural network layer, b is the bias matrix, and V represents h t-1 The weight matrix, x t Indicates the input value at the current moment, h t-1 Indicates the output value received at the previous moment at the current moment, c t-1 Indicates the state value at the previous moment, c t Indicates the current state value; h t is the output value at the current moment.

[0042] Furthermore, in step S5, the BLEU value is used as a criterion for judging the translation model.

[0043] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0044] A computer-readable storage medium is used to store computer instructions, and the computer instructions implement the above steps when executed by a processor.

[0045] Beneficial effects of the present invention

[0046] The present invention adopts the Stanford CoreNLP toolkit to construct the dependency syntax tree of the source language sentence and obtain the syntactic structure information of the source language sentence, thereby solving the shortcoming that the neural machine translation does not introduce linguistic knowledge; introduces the long short-term memory network to obtain word position information, thereby avoiding the shortcomings of fixed and inflexible encoding methods, and further enhancing the translation performance of the transformer; and finally can achieve a more ideal Chinese-English translation effect;

[0047] (1) Introducing syntactic knowledge into neural machine translation and using dependency syntax trees to construct the syntactic structure of source language sentences to improve translation performance;

[0048] (2) Improvements are made to the position encoding of the Transformer model, using LSTM’s ability to capture long-distance information to obtain more accurate position encoding information, further improving the translation accuracy of the neural machine translation model;

[0049] (3) Applying the neural machine translation method based on Transformer model optimization to the field of Chinese-English translation can significantly improve the translation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the present invention;

[0051] Figure 2 This is a schematic diagram of a specific implementation of model optimization in one embodiment of the present invention;

[0052] Figure 3 A schematic diagram of a model architecture in one embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of ablation experiment comparison in one embodiment of the present invention. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0055] Combination Figures 1 to 4 ,

[0056] A neural machine translation method based on Transformer model optimization:

[0057] The method specifically comprises the following steps:

[0058] Step S1, building a translation model, obtaining the Chinese and English datasets in News Commentary v12 in WMT17, and normalizing the data;

[0059] Step S101, the downloaded Chinese-English dataset is from the news field; 50,000 Chinese-English language pairs are extracted from the Chinese-English dataset as a training set; newsdev2017 is used as a validation set, and newstest2017 is used as a test set;

[0060] Step S102, preprocessing the data set, screening the data in the data set, and deleting sentences with a sentence length greater than 100 and repeated sentences;

[0061] Step S103, obtain a dictionary based on the training set, and use the unregistered words <unk>express.

[0062] Since the computational complexity of the decoder in generating target words is extremely high, in order to reduce the complexity, a fixed-size dictionary is used on both the source language side and the target language side. However, words that appear very rarely in a sentence are generally not added to the dictionary. These words are called unregistered words and are usually <unk>express

[0063] Step S2, introducing sentence structure information into the translation model, obtaining word vectors, and obtaining a new input sequence;

[0064] Step S201, using the Stanford CoreNLP toolkit to obtain the dependency syntax tree of each sentence, introducing the dependency matrix, extracting the source language sentence structure information and integrating it into the translation model;

[0065] Step S202, using the CBOW model to predict the target word according to the dependency words and context words to obtain a word vector;

[0066] Dependency syntax analyzes a sentence into a dependency syntax tree, describing the dependency relationship between each word, which is called dependency grammar. Dependency syntax shows that there is a master-slave relationship between words. In a sentence, if a word modifies another word, the modifier is called a subordinate word, and the modified word is called a dominant word. The grammatical relationship between the two is called a dependency relationship. The dependency relationship matrix can clearly indicate whether there is a relationship between two words and how close the relationship is. Then, the closeness of the relationship between the two words is obtained, so that the neural machine translation system can understand the general meaning of the sentence and obtain more accurate translation results.

[0067] Step S203, using the Stanford CoreNLP toolkit to convert the source language sentence into a dependency syntax tree form, and then into a corresponding dependency matrix; the calculation formula is:

[0068]

[0069] D(i,j) is multiplied by the word vector obtained in step S202 to obtain a new input sequence;

[0070] Where i is the current node index, j is the other node index, and w j For other nodes, w i is the current node;

[0071] Step S204, using LSTM to replace the original Transformer model to obtain the position encoding method, and obtaining the output of each time step of LSTM as the position information of each word.

[0072] Long short-term memory network is a recurrent neural network. Compared with recurrent neural network, LSTM is special. It can be applied to longer sequences to learn long-term dependent information. It is mainly to solve the gradient vanishing and gradient exploding problems of RNN during training. LSTM has the recurrent structure of RNN, but the difference is that LSTM introduces three gate structures to realize the transmission of information in the network structure. They are forget gate, input gate, and output gate. The gate mechanism is used to control the discarding and adding of information to realize the forgetting and memory functions. The gate control mechanism is a way to selectively pass information, which consists of a Sigmoid neural network layer and a point-by-point multiplier.

[0073] Forget gate: decides whether to keep or discard information.

[0074] f t =σ(W f h t-1 +V f X t +b f )

[0075] Input gate: determines whether to update the input information to the cell unit.

[0076] i t =σ(W i h t-1 +V i X t +b i )

[0077] Candidate cell unit,

[0078]

[0079] The latest status of the memory cell node,

[0080]

[0081] Output gate: determines the output of the memory cell state value.

[0082] o t =σ(W o h t-1 +V o X t +b o )

[0083] The output of the final hidden state is

[0084] h t =o t *tanh(c t )

[0085] Where t represents the current moment, and t-1 represents the previous moment;

[0086] f t represents the state of the forget gate, i t Indicates the input gate state, o t represents the output gate state, σ represents the sigmoid activation function, tanh represents the tanh activation function, W is the weight matrix of the neural network layer, b is the bias matrix, and V represents h t-1 The weight matrix, x t Indicates the input value at the current moment, h t-1 Indicates the output value received at the previous moment at the current moment, c t-1 Indicates the state value at the previous moment, c t Indicates the current state value; h t is the output value at the current moment.

[0087] Step S3, training the translation model using the sequence that integrates the position information and the syntactic information, and using the adam optimizer to perform optimization iterations to obtain a neural machine translation model;

[0088] Transformer only uses the Attention mechanism to perform machine translation tasks. It has achieved good translation results. Transformer uses an encoder-decoder framework, including an encoder layer and a decoder layer. The encoder layer and the decoder layer are respectively composed of n identical encoder-decoders stacked together. Each encoder includes two sub-layers, namely a multi-head self-attention mechanism and a fully connected feedforward neural network. A residual connection is used between each two sub-layers, followed by a layer normalization operation, in order to facilitate gradient transfer and speed up model convergence. It can be expressed as follows.

[0089] layernorm(h+sublayer(h))

[0090] Where layernorm(·) represents the output of the layer regularization function, sublayer(·) represents the sublayer output, and h represents the sublayer input.

[0091] Compared with the encoder, the decoder has one more sublayer, and performs a multi-head attention mechanism on the output of the encoder. It also needs to use residual connections between every two sublayers, followed by layer regularization. The encoder converts the input source language into a vector with semantic information, and the decoder is responsible for generating the target language based on the vector with semantic information. The Transformer model does not contain loops and convolutional neural networks, and processes each word in parallel. It does not have the ability to recognize the order of each word, so position encoding is needed to help the model recognize the position information of each word. The basic unit of the multi-head attention mechanism in the encoder and decoder sublayers of the Transformer is the scaled dot-product attention mechanism, whose input is the query matrix Q, the key-value pair matrix K and V. The dot-product attention mechanism training process is as follows.

[0092] After word embedding and positional encoding, the source language sentence gets the input sequence Define three matrices W Q , W K , W V , use these three matrices to perform three linear transformations on all input sequence vectors, and all vectors derive three new vectors q t , k t , v t . All q t The vectors are assembled into a large matrix, denoted as the query matrix Q, and all k t The vectors are combined into a large matrix, denoted as the key matrix K, and all v t The vectors are combined into a large matrix, denoted as the value matrix V. To obtain the attention weight of the first word, the query vector q1 of the first word needs to be multiplied by the key matrix K. Then the obtained values ​​need to be passed through softmax so that their sum is 1. After obtaining the weights, the weights are multiplied by the value vector v of the corresponding word. t Finally, these weighted value vectors are summed up to get the output of the first word. The same operation is performed on other input vectors to get all the outputs after self-attention.

[0093]

[0094] Finally, the multi-head attention mechanism concatenates the results of multiple dot-product attention mechanisms.

[0095] MultiHead(Q,K,V)=Concat(head i )W O

[0096] head i =Attention(QW Q ,KW K ,VW V )

[0097] Where W j Q ,W j K ,W j V ,W O is the parameter matrix. h is the number of heads in the multi-head attention mechanism.

[0098] The feedforward neural network is a two-layer fully connected network layer that uses RELU as the activation function and is defined as follows:

[0099] FN(x)=max(0,xW1+b1)W2+b2

[0100] Where W1, W2, b1, b2 are model parameters.

[0101] Since Transfromer does not contain loops and convolutional neural networks, self-attention does not contain position information. Transformer processes each word in parallel and does not have the ability to recognize the order of each token. Adding position encoding can help the model recognize position information. Therefore, in order to enable the model to utilize the order of the sequence, "position encoding" is added to the input sequence at the bottom of the encoder and decoder stacks. The position encoding has the same dimension as the word vector, and the two can be directly added as the encoder input. Use sine and cosine functions of different frequencies.

[0102]

[0103]

[0104] Where pos is the position of the element in the sequence, 2i is the current dimension of the positional encoding vector, and d model represents the dimension of the input sequence. Each dimension of the positional encoding corresponds to a sinusoid. The wavelengths form a geometric series from 2π to 10000·2π.

[0105] Step S4, inputting the test set into the neural machine translation model optimized by the optimization model in step S3 to obtain a translation result;

[0106] Step S5, evaluating the translation result, using the BLEU value as the evaluation criterion of the translation model, and ending the translation.

[0107] Machine translation evaluation methods are mainly divided into manual evaluation and automatic evaluation. The manual evaluation method refers to manually scoring the translation results from the two aspects of "fidelity" and "fluency" of the translation. Among them, "fidelity" is used to evaluate the completeness of the translation to the original meaning, and "fluency" is used to evaluate the fluency of the translation. Although the manual evaluation method can achieve better results, it is difficult to unify the evaluation standards of the evaluators, the cost is very high and the efficiency is very low. Compared with manual translation, machine translation is far more efficient than manual translation and can translate tens of thousands of sentences in a short time. Therefore, machine automatic evaluation methods are often used in actual reality. The automatic evaluation method calculates the score by examining the similarity between the translation result and the translation. The present invention uses BLEU as an automatic evaluation method for the translation result.

[0108] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0109] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps of the above method are implemented.

[0110] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0111] The above is a detailed introduction to the neural machine translation method based on Transformer model optimization proposed in the present invention, and the principles and implementation methods of the present invention are explained. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.< / unk> < / unk> < / unk>

Claims

1. A neural machine translation method based on Transformer model optimization, characterized by: The method specifically comprises the following steps: Step S1, building a translation model, obtaining the Chinese and English datasets in News Commentary v12 in WMT17, and normalizing the data; Step S2, introducing sentence structure information into the translation model, obtaining word vectors, and obtaining a new input sequence; Step S201, using the Stanford CoreNLP toolkit to obtain the dependency syntax tree of each sentence, introducing the dependency matrix, extracting the source language sentence structure information and integrating it into the translation model; Step S202, using the CBOW model to predict the target word according to the dependency words and context words to obtain a word vector; Step S203, using the Stanford CoreNLP toolkit to convert the source language sentence into a dependency syntax tree form, and then into a corresponding dependency matrix; the calculation formula is: D(i,j) is multiplied by the word vector obtained in step S202 to obtain a new input sequence; Where i is the current node index, j is the other node index, and w j For other nodes, w i is the current node; Step S204, using LSTM to replace the original Transformer model to obtain the position encoding method, and obtaining the output of each time step of LSTM as the position information of each word; Step S3, training the translation model using the sequence that integrates the position information and the syntactic information, and using the adam optimizer to perform optimization iterations to obtain a neural machine translation model; Step S4, inputting the test set into the neural machine translation model optimized in step S3 to obtain a translation result; Step S5, evaluating the translation result and ending the translation.

2. The method according to claim 1, characterized in that: The specific steps of step S1 are: Step S101, the downloaded Chinese-English dataset is from the news field; 50,000 Chinese-English language pairs are extracted from the Chinese-English dataset as a training set; newsdev2017 is used as a validation set, and newstest2017 is used as a test set; Step S102, preprocessing the data set, deleting sentences with a length greater than 100 and repeated sentences; Step S103, obtain a dictionary based on the training set, and use the unregistered words <unk> express.< / unk> 3. The method according to claim 2, characterized in that: The step S204 specifically includes: LSTM includes forget gate, input gate and output gate; Forget gate: decides whether to keep or discard information. f t =σ(W f h t-1 +V f X t +b f ) Input gate: determines whether to update the input information to the cell unit. i t =σ(W i h t-1 +V i X t +b i ) Candidate cell unit, The latest status of the memory cell node, Output gate: determines the output of the memory cell state value. the t =σ(W o h t-1 +V o X t +b o ) The output of the final hidden state is h t =o t *tanh(c t ) Where t represents the current time, t-1 represents the previous time, and f t represents the state of the forget gate, i t Indicates the input gate state, o t represents the output gate state, σ represents the sigmoid activation function, tanh represents the tanh activation function, W is the weight matrix of the neural network layer, b is the bias matrix, and V represents h t-1 The weight matrix, x t Indicates the input value at the current moment, h t-1 Indicates the output value of the previous moment received at the current moment, c t-1 Indicates the state value at the previous moment, c t Indicates the current state value; h t is the output value at the current moment.

4. The method according to claim 3, characterized in that: In step S5, the BLEU value is used as the evaluation criterion of the translation model.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images