Machine translation method based on symmetric attention

Abstract

The application discloses a machine translation method based on symmetric attention, and mainly solves the problem that the related information in multiple subspaces cannot be utilized in the prior art, resulting in the lack of reference information in decoding calculation, and the translation result cannot cover all scenarios. In the machine translation model based on symmetric attention constructed by the application, two parallel symmetric subspaces are used to form the symmetric attention layer of the decoder in the model, and the symmetric attention algorithm is used to calculate two symmetric attentions and perform fusion. In the decoder of the application, the information interaction is realized between the symmetric attention layer and the attention matrix output by the encoder, the ability to comprehensively and accurately obtain attention information is improved, and the machine translation model constructed by the application covers more comprehensive semantic scenarios in the translation work, and the accuracy in English translation is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of electricity, and further relates to a machine translation method based on symmetric attention in the technical field of computer-aided design. The application can be used for extracting text feature information of an English sentence to be translated through an encoder, and then decoding to generate a corresponding Chinese translation sentence. BACKGROUND

[0002] With the improvement of computer computing power and the application of big data, deep learning has achieved further application. Machine translation technology based on deep learning is increasingly applied in different fields. Machine translation technology can help people complete translation from a source language to a target language, thereby gradually replacing human beings to complete complex and time-consuming and laborious translation work. At present, in the field of machine translation, the Transformer model, which represents the most advanced level, is a machine translation model completely based on an attention mechanism. The main idea is that the input vector is first linearly transformed to obtain a query matrix Query (Q), a key-value matrix Key (K) and a real value matrix Value (V). The attention value is obtained by using the three matrices for attention calculation, as an embodiment of the correlation degree between elements. In machine translation work, the decoder obtains the correlation information between data points through the attention value as the basis for translation prediction. Therefore, it is helpful to improve the accuracy of machine translation to maximize the expression ability of the global correlation information between data points as much as possible.

[0003] A method for optimizing a machine translation model based on a Transformer model is proposed in the patent technology "A method for optimizing a machine translation model based on a Transformer model" (Patent Application No. 202110361317.8, Authorized Announcement No. CN 113051938 B) owned by Sichuan University. The specific implementation method of the method is as follows: Step one, initialize a population of Transformer models with multiple different structures and parameters as the parent model population; Step two, calculate the word vector learning ability evaluation index of each Transformer model in the parent model population using the BLEU algorithm; Step three, select parent individuals from the parent model population according to the word vector learning ability evaluation index size of step two, and generate a population of offspring individuals using the crossover mutation operator, while calculating the word vector learning ability evaluation index of each model in the population of offspring individuals using the BLEU algorithm; Step four, perform environmental selection on the parent individuals and offspring individuals to generate a new model population, and perform multiple rounds of iterative evolution according to steps two and three until the iteration termination condition is met; Step five, select the optimal individual in the last generation to perform a machine translation task. Although this method can limit the number of evolutions by pre-setting a threshold to achieve the purpose of saving resources, it still has the following shortcomings: because the cross-attention layer of the decoder uses the same multiplicative calculation scheme as other layers in the module, its output only reflects the correlation between data points in a single dimension, which causes the encoder to lose the target language options corresponding to the source language when generating the prediction dictionary, affecting the accuracy of the decoder when outputting the prediction probability based on the source language.

[0004] A method for multi-domain neural machine translation based on self-attention mechanism is proposed in the patent document "Multi-domain neural machine translation method based on self-attention mechanism" (Patent application number: 201910344013.3, Publication number: CN 110059323 A) of Suzhou University. The method makes two important changes to the Transformer: first, the self-attention mechanism based on domain perception, which is added to the key and value vectors of the original self-attention mechanism when representing multiple domains. The weight of the attention mechanism is the relevance of the query and the domain-aware key. The self-attention mechanism based on domain perception can be used in the encoder, decoder or self-attention layer of both the encoder and decoder. Second, a domain representation learning module is added to learn the domain vector. Although this method can establish a system architecture for multiple specific domains under the condition of lacking flexibility in the model, each system is fine-tuned in a given domain to optimize performance. However, this method still has the disadvantage that only part of the correlation information of the input vector is used in the cross-attention layer of the decoder, and the correlation information mapped to other subspaces is not fully utilized, resulting in a lack of reference information in the decoding calculation, which leads to the translation result not covering all scenarios. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art mentioned above, and to propose a machine translation method based on symmetric attention, which fully utilizes the symmetric attention in different subspaces that reflects the global correlation information between data, to solve the problem of insufficient expression of correlation between data points and the problem of not being fully utilized in the prior art, and the problem of low accuracy of translation results in decoding calculation.

[0006] The idea for achieving the object of the present application is that the model constructed by the method of the present application adopts an encoder-decoder structure. The encoder includes a multi-head self-attention layer and a forward feedback layer: the multi-head self-attention layer first performs attention calculation to obtain the correlation information between elements in the input vector as the basis for establishing a translation prediction dictionary; the forward feedback layer performs full connection calculation on the calculated attention to summarize the information in the entire sequence, so that the sequence contains global correlation information. The decoder includes a multi-head self-attention layer, a symmetric attention layer and a forward feedback layer, wherein the multi-head self-attention layer and the forward feedback layer have the same functions as the encoder. The symmetric attention layer uses two symmetric attentions to realize the information interaction between the encoder and the decoder, solving the problem of missing reference in the decoding process due to the failure to use the correlation information of other subspaces of the input vector. The three matrices used in the present application: query matrix Query (Q), key-value matrix Key (K) and real-value matrix Value (V) are equal in physical meaning, and all reflect the correlation between corresponding elements in the matrix in different dimensions. This correlation is an important basis for the decoder to make translation predictions. Considering this feature, the above three matrices are used to calculate two symmetric attentions in the symmetric attention layer of the decoder to extract the correlation information of different subspaces, solving the problem that the correlation information between elements is limited, resulting in the loss of target language options corresponding to the source language when the encoder generates a prediction dictionary. Before calculating in the symmetric attention layer of the decoder, a full connection layer is first added to perform linear transformation, combining the local correlation of each element into global correlation, so that all related options can be considered when calculating the prediction value, solving the problem that the translation result fails to cover all scenarios. According to the dictionary generated by the encoder, the decoder calculates the correlation between the current processed data and the prediction candidates, and selects the candidate with the largest correlation as the final prediction result of the processed data.

[0007] The specific steps for implementing the present application are as follows:

[0008] Step 1, generating a training set:

[0009] Step 1.1, selecting at least 270,000 English sentences and their corresponding Chinese translation sentences to form a sample set;

[0010] Step 1.2, removing duplicate English sentences and their corresponding Chinese translation sentences in the sample set, and removing English sentences and their corresponding Chinese translation sentences that are too short or too long relative to the average length of all English sentences to obtain a training set;

[0011] Step 2, constructing a machine translation model based on symmetric attention:

[0012] Step 2.1, constructing an encoder composed of a multi-head self-attention layer, a residual layer and a forward feedback layer in series:

[0013] The multi-head self-attention layer contains at least 8 subspaces, each of which calculates the self-attention of the current predicted word with all the words in the sentence to which it belongs through dot product multiplication;

[0014] The residual layer is realized by a layer normalization function;

[0015] The structure of the forward feedback layer is input layer, fully connected layer, output layer in turn, the dimensions of the input layer and the output layer are both set to 512, the fully connected layer is realized by a maximum function, and the dimension is set to 2048;

[0016] Step 2.2, a decoder composed of a multi-head self-attention layer, a first residual layer, a symmetric attention layer, a second residual layer, a forward feedback layer, and a third residual layer in series is constructed:

[0017] The structure and parameters of the multi-head self-attention layer and the forward feedback layer in the decoder are the same as those of the multi-head self-attention layer and the forward feedback layer in the encoder in step 2.1;

[0018] The first residual layer, the second residual layer, and the third residual layer in the decoder are all realized by a layer normalization function;

[0019] The symmetric attention layer is composed of two parallel symmetric subspaces, which uses a symmetric attention algorithm to calculate and fuse two branches of symmetric attention, realizing the information interaction between the layer and the attention matrix output by the encoder;

[0020] The symmetric attention algorithm refers to extracting the correlation information in the two parallel symmetric subspaces respectively, and performing linear transformation on the two correlation information to obtain two feature matrices; using dot product multiplication to calculate the symmetric attention of each feature matrix; through a summation function, the two symmetric attentions are fused, and the sum of the two branches of symmetric attention is subjected to layer normalization operation to output a symmetric attention matrix;

[0021] Step 2.3, the input layer, the first embedded expression layer, the first encoder, the second encoder, the third encoder, the fourth encoder, the fifth encoder, and the sixth encoder are sequentially connected to form an encoder group;

[0022] The first embedded expression layer is realized by an embedding algorithm;

[0023] The structures and parameters of the first to sixth encoders are the same;

[0024] Step 2.4, a decoder group is constructed, wherein the second embedded expression layer, the first decoder, the second decoder, the third decoder, the fourth decoder, the fifth decoder, the sixth decoder and the activation layer are connected in sequence, the activation layer is further connected with the second embedded expression layer, and the first to sixth decoders are further connected with the sixth encoder in the encoder group respectively;

[0025] The structure and parameters of the second embedded expression layer are the same as those of the first embedded expression layer;

[0026] The structures and parameters of the first to sixth decoders are the same;

[0027] The activation layer is realized by a Softmax activation function;

[0028] Step 2.5, the encoder group and the decoder group are connected in series to form a machine translation model based on symmetric attention;

[0029] Step 3, training the machine translation model:

[0030] The training set is input into the machine translation model in batches, the loss value of the translated text and the corresponding Chinese text is calculated by using a loss function, the learnable parameters in the machine translation model are updated iteratively by using a gradient descent algorithm, the training process is optimized by using an Adam optimizer, and the training process is stopped until the loss value of the machine translation model converges, so that a trained machine translation model based on symmetric attention is obtained;

[0031] Step 4, using the trained machine translation model for translation:

[0032] Step 4.1, removing the repeated English sentences in the to-be-translated English sentences, and removing the English sentences with a length shorter or longer than the average length of all to-be-translated English sentences;

[0033] Step 4.2, inputting the processed to-be-translated sentences into the trained machine translation model based on symmetric attention, and outputting the corresponding translated text.

[0034] Compared with the prior art, the present application has the following advantages:

[0035] Firstly, the symmetric attention layer in the decoder of the present application optimizes the existing Transformer model by calculating two symmetric attentions and fusing them, overcomes the defect that only the correlation between data points in a single dimension is reflected in the prior art, resulting in the loss of target language options corresponding to the source language when the encoder generates the prediction dictionary, so that the decoder of the present application more comprehensively utilizes the attention information obtained by the encoder, and can refer to more accurate attention information during translation calculation, which is beneficial to improve the accuracy of the prediction probability output by the decoder.

[0036] Secondly, the application uses different arrangement orders of the query matrix Query (Q), the key value matrix Key (K) and the real value matrix Value (V) to calculate the symmetric attention, fully utilizes the characteristics that the three matrices are equal in physical meaning, makes the matrices reflecting the correlation information between elements in different symmetric subspaces all participate in the calculation of the symmetric attention, solves the problem that the existing technology fails to use all the correlation information of the input vector when calculating the cross attention, resulting in the lack of reference information in the decoding calculation. The application further enhances the expression ability of the global correlation information extracted by the decoder, establishes the connection between all elements in the input vector according to the global correlation, and makes the translation results of the machine translation model constructed by the application based on the symmetric attention cover a more comprehensive scene, which is beneficial to improve the accuracy of the machine translation model when performing the translation task in different semantic scenes. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart for implementing the application;

[0038] Figure 2 is a structure schematic diagram of the machine translation model constructed by the application based on the symmetric attention;

[0039] Figure 3 is a structure schematic diagram of the decoder in the machine translation model constructed by the application;

[0040] Figure 4 is a comparison diagram of the evaluation index BLEU of the simulation experiment in the application. DETAILED DESCRIPTION

[0041] The application will be further described in detail below in combination with the drawings and embodiments.

[0042] Reference Figure 1 and embodiments, the implementation steps of the application will be further described in detail.

[0043] Step 1, generating a training set and a test set.

[0044] Step 1.1, the embodiment of the application selects 270,000 English sentences and their corresponding Chinese translation sentences from the machine translation public data set WMT2018 news to constitute a sample set.

[0045] Step 1.2, all English sentences and corresponding Chinese translation sentences are divided into a training set and a test set according to a ratio of 9:1: the training set includes 250,000 English sentences and corresponding Chinese translation sentences, and the test set includes 20,000 English sentences and corresponding Chinese translation sentences.

[0046] Step 2, constructing a machine translation model based on symmetric attention.

[0047] Reference Figure 2 Further description is made to the constructed machine translation model structure based on symmetric attention.

[0048] Step 2.1, an encoder composed of a multi-head self-attention layer, a residual layer and a feed-forward layer is constructed.

[0049] The multi-head self-attention layer contains 8 subspaces, each of which corresponds to a self-attention mechanism.

[0050] The residual layer is realized by a layer normalization function.

[0051] The structure of the feed-forward layer is input layer, full connection layer and output layer in sequence, the dimensions of the input layer and the output layer are both set to 512, the full connection layer is realized by a maximum function, and the dimension thereof is set to 2048.

[0052] In each subspace in the multi-head self-attention layer, the self-attention of the hidden representation of the current processing English sentence feature matrix is calculated, one self-attention result is obtained in each subspace, and then 8 self-attention results are spliced respectively according to the formula MultiHead(Q,K,V)=Concat(head1,…,head h )W O , wherein MultiHead(·) is multi-head self-attention, Q, K and V are the hidden representation of the current processing English sentence feature matrix, Concat(·) is a splicing function, W O is a learnable parameter, the subscript h is a preset multi-head self-attention head number 8, and head i is obtained by the following formula:

[0053]

[0054] , wherein head i is the self-attention in the i-th subspace, i=1,…,8, is the self-attention, wherein the superscript · T represents a transposition operation, d k represents the dimension of the word embedding vector, is the learnable parameter corresponding to the self-attention in the i-th subspace.

[0055] The residual layer is realized by the formula R(x1)=LayerNorm(Add(x1,Sublayer(x1))), wherein x1 is the input of the multi-head self-attention layer, R(·) is the output of the residual layer, LayerNorm(·) is a layer normalization function, Add(·) is a summation function, and Sublayer(x) is the output of the multi-head self-attention layer.

[0056] The fully connected layer in the feedforward layer is represented by the formula FFN(x2)=Max(0,x2W1+b1)W2+b2, where x2 is the input of the feedforward layer, FFN(·) is the output of the feedforward layer, Max(·) is the maximum value operation, and W1, W2, b1, and b2 are learnable parameters that are iteratively updated using the gradient descent algorithm. The initial values ​​of W1 and b1 are set to zero matrices of size 512×2048, and the initial values ​​of W2 and b2 are set to zero matrices of size 2048×512.

[0057] Step 2.2: Construct a decoder consisting of a multi-head self-attention layer, a first residual layer, a symmetric attention layer, a second residual layer, a forward feedback layer, and a third residual layer connected in series.

[0058] refer to Figure 3 The structure of the constructed decoder will be further described.

[0059] The structure and parameters of the multi-head self-attention layer and the forward feedback layer in the decoder are the same as those of the multi-head self-attention layer and the forward feedback layer in the encoder in step 2.1.

[0060] The first residual layer, second residual layer, and third residual layer in the decoder are all derived from the general formula R. i (x i =LayerNorm(Add(x i Sublayer(x) i ))) Implemented, where R i (x i Let be the output of the i-th residual layer, i = 1, 2, 3, LayerNorm(·) is the layer normalization function, Add(·) is the summation function, and x is the output of the i-th residual layer. i Sublayer(x) is the input to the layer preceding the i-th residual layer, concatenated with it. i ) is the output of the layer preceding the i-th residual layer in series.

[0061] The symmetric attention layer is composed of two parallel symmetric subspaces, and the symmetric attention algorithm is used to calculate two symmetric attentions and fuse them to realize the information interaction between the layer and the attention matrix output by the encoder. The principle of the symmetric attention algorithm is that the encoder is mainly responsible for extracting attention information in the sentence, and the decoder is an important part of the machine translation model to complete the translation task. In order to better complete the translation task, the symmetric attention layer in the decoder needs to have deep information interaction with the attention matrix output by the encoder. In order to obtain more rich attention information from the encoder and realize the information interaction between the symmetric attention layer in the decoder and the attention matrix output by the encoder to a greater extent, the symmetric attention algorithm is used in the symmetric attention layer of the decoder to achieve the above purpose. The symmetric attention algorithm extracts attention information in the two symmetric subspaces of the symmetric attention layer, and uses two different permutation orders of the query matrix Q, the key value matrix K and the real value matrix V to calculate the symmetric attention. The algorithm makes full use of the characteristics that the three matrices are equal in physical meaning, so that the matrices reflecting the correlation information between elements in different symmetric subspaces are involved in the calculation of symmetric attention. The algorithm first linearly transforms the input h' of the layer by the formula Q' = W Q′ h' and V' = W V′ h' to obtain matrix Q' and matrix V', where h' is the correlation information between the current processing word and the previous words in the sentence to which the word belongs, W Q′ and W V′ are learnable parameters updated iteratively using the gradient descent algorithm, and their initial matrices are set to zero matrices of size 512x512. Matrix Q' is the projection of the correlation between the current processing word and other words in the sentence in the first symmetric subspace, and matrix V' is the projection of the correlation between the current processing word and other words in the sentence in the second symmetric subspace.

[0062] In the first symmetric subspace of matrix Q', the first symmetric attention result output by the symmetric attention layer is calculated by the formula , where Attention1 is the first symmetric attention result output by the symmetric attention layer, Softmax(·) is an activation function, q1 is the global vector corresponding to matrix Q' in the first symmetric subspace, K1 and V1 are projection matrices of the attention of the English sentence processed at the current iteration of the network parameters in the first symmetric subspace, the superscript ·T is a transpose operation, d k is the word embedding vector dimension of each word in the English sentence processed at the current iteration of the network parameters; in the second symmetric subspace of matrix V', the second symmetric attention result output by the symmetric attention layer is calculated by the formula a second branch symmetric attention result of the symmetric attention layer output is calculated, wherein Attention2 is the second branch symmetric attention result of the symmetric attention layer output, q2 is a global vector corresponding to the second symmetric subspace of a matrix Q2, Q2 and K2 are projection matrices of attention of the network parameters updated in the current iteration in the second symmetric subspace for processing an English sentence, and V' is a projection matrix of correlation between a word and other words in the sentence to which the word belongs in the second symmetric subspace for processing the correlation by the network parameters updated in the current iteration; the two branches of symmetric attention are fused through the formula Attention=LayerNorm(Attention1+Attention2), wherein Attention is a symmetric attention matrix, the matrix size is P×P, P is the length of the English sentence processed by the network parameters updated in the current iteration, and the element a ij represents the symmetric attention between the i th word and the j th word in the English sentence processed by the network parameters updated in the current iteration, LayerNorm(·) is a layer normalization operation, and Attention1 and Attention2 are the two branches of symmetric attention.

[0063] Step 2.3, the input layer, the first embedded expression layer, the first encoder, the second encoder, the third encoder, the fourth encoder, the fifth encoder, and the sixth encoder are sequentially connected to form an encoder group.

[0064] The first embedded expression layer is implemented by an embedding algorithm, which is used to generate an English sentence feature matrix with dimensions (H, P, D), wherein H is the number of English sentences, P is the length of the longest English sentence in the selected English sentences, and D is the dimension of the word embedding vector. The word embedding vector is generated according to the BPE algorithm to generate a vocabulary, and then the word embedding vector is obtained through the word embedding operation. The dimension of the word embedding vector is 512.

[0065] The structures and parameters of the first to sixth encoders are the same.

[0066] Step 2.4, a decoder group is constructed, wherein the second embedded expression layer, the first decoder, the second decoder, the third decoder, the fourth decoder, the fifth decoder, the sixth decoder, and the activation layer are sequentially connected, the activation layer is also connected to the second embedded expression layer, and the first to sixth decoders are also connected to the sixth encoder in the encoder group.

[0067] The structure and parameters of the second embedded expression layer are the same as those of the first embedded expression layer.

[0068] The structures and parameters of the first to sixth decoders are the same.

[0069] The activation layer realizes its function by a Softmax activation function.

[0070] Step 2.5, concatenating the encoder group and the decoder group to form the machine translation model based on symmetric attention.

[0071] Step 3, training the machine translation model.

[0072] The training set is input into the machine translation model in batches, the loss value of the translated text and the corresponding Chinese text is calculated by using the loss function, the learnable parameters in the machine translation model are iteratively updated by using the gradient descent algorithm, the training process is optimized by using the Adam optimizer, and the training process is stopped until the loss value of the machine translation model converges, thereby obtaining the trained machine translation model based on symmetric attention.

[0073] The loss function is set as a cross-entropy loss function, and is realized by the formula , wherein L is the loss value between the translated text feature matrix generated according to the Chinese word table and the Chinese text feature matrix generated according to the Chinese word table, is the translated text feature matrix generated according to the Chinese word table, y is the Chinese text feature matrix generated according to the Chinese word table, and log is a logarithm operation with base 10.

[0074] The translated text is generated by the following process: the training set is input into the machine translation model in batches, and the English sentence feature matrix A is output by the first embedded expression layer of the encoder group; the English sentence feature matrix A is encoded by the six encoders in turn, and the attention matrix M is output by the sixth encoder; the attention matrix M is decoded by the first to sixth decoders in the decoder group in turn, and the correlation matrix is output by the sixth decoder; the correlation matrix outputs the correlation probability between the current English word and all candidate Chinese words in the Chinese word table, and the maximum correlation probability is selected as the Chinese translation word corresponding to the English word. Since there is a high similarity between adjacent words in the same sentence in terms of semantics, the similarity can be an important basis for decoding by the decoder, therefore, the Chinese translation word corresponding to the English word is input into the second embedded expression layer, and the translation of the next English word adjacent to the English word in the English sentence feature matrix A is started, and the translation process is the same as the above process, thereby obtaining the Chinese translation word corresponding to the English word; the above translation process is repeatedly executed until the last word in the selected English sentence is translated, thereby obtaining the Chinese translation words corresponding to all English words in the selected English sentence, and the Chinese translation words corresponding to all English words are sequentially arranged to form the translated text.

[0075] The Adam optimizer is realized by the formula lrate=d -0.5 ·min(step -0.5step·warm_steps -0.5 ) is generated. The Adam optimizer is mainly used to dynamically adjust the learning rate in training the machine translation model. Among them, lrate is the learning rate parameter in the gradient descent algorithm, d is the dimension of the word embedding vector, min(·) is the minimum value function, step is the current iteration number, and warm_steps is the termination iteration number.

[0076] Step 4, using the trained machine translation model to translate the test set.

[0077] The English sentences to be translated in the test set are input into the trained machine translation model based on symmetric attention, and the corresponding translated text is output.

[0078] The effect of the present application will be further described below in combination with a simulation experiment:

[0079] 1. Simulation experiment conditions:

[0080] The hardware platform of the simulation experiment of the present application is: the processor is Intel(R)Core TM i7-10700 CPU, the main frequency is 1.80GHz, the memory is 32GB, and the GPU uses NVIDIA GeForce RTX 3090.

[0081] The software platform of the simulation experiment of the present application is: Ubuntu21.1 operating system and Pytorch 1.10.0+cu111 deep learning framework.

[0082] The input sentences used in the simulation experiment of the present application are part of the sentences in the machine translation public data set WMT2018 news, wherein the training set includes 250,000 English sentences and their corresponding Chinese translation texts, and the test set includes 20,000 English sentences and their corresponding Chinese translation texts.

[0083] 2. Simulation content and result analysis:

[0084] The simulation experiment of the present application is to input the training set into the machine translation model constructed by the present application and the prior art Transformer model for training, respectively, to obtain the trained machine translation model based on symmetric attention and the trained Transformer model. In each iteration during the training process, English sentences are randomly selected from the training set as test samples, and the BLEU value evaluation index of the machine translation model and the Transformer model obtained in each iteration is calculated, respectively. After 50 iterations of training, 50 BLEU values are obtained, and a comparison curve is drawn as shown in Figure 4 , wherein the penalty factor of BLEU is set to 0.6.

[0085] The following is combined Figure 4 The effects of the present invention will be further described.

[0086] Figure 4 The horizontal axis represents the iteration round, and the vertical axis represents the evaluation index BLEU value. Figure 4 The curves marked with "×" represent the BLEU values ​​of the machine translation model using symmetric attention in each iteration, while the curves marked with "△" represent the BLEU values ​​of the existing Transformer model in each iteration.

[0087] Depend on Figure 4 As can be seen, with the increase of the number of iterations, the BLEU value of the existing Transformer model during training gradually increases and stabilizes at 22.72, while the BLEU value of the machine translation model based on symmetric attention during training gradually increases and stabilizes at 24.87. Obviously, the machine translation model constructed in this invention is 2.15 units higher than the Transformer model.

[0088] The test set was input into the trained machine translation model based on symmetric attention and the existing Transformer model, respectively, and the test results were compared. The results are shown in Table 1.

[0089] Table 1. Comparison of BLEU scores for the two models tested in the simulation experiment of this invention.

[0090] Model type BLEU value Transformer model 22.65(↑9.71％) Machine translation model of the invention 24.85

[0091] As shown in Table 1, compared with the existing Transformer model, the machine translation model based on symmetric attention in this invention has a BLEU value 2.13 units higher on the test set.

[0092] pass Figure 4 Together with Table 1, it can be seen that the symmetric attention calculated using the query matrix Q, key matrix K, and real value matrix V in different symmetric subspaces has the same feature representation ability. Fusing two symmetric attentions can significantly enhance the ability to express feature information, making the machine translation model based on symmetric attention perform better in machine translation tasks.

[0093] Table 2 shows the translation of the test set using the machine translation model of the present invention, with two sets of control texts randomly selected from the resulting translated texts.

[0094] Table 2. Comparison of translation results of the test set in the simulation experiment of this invention.

[0095]

[0096]

[0097] By comparing the "Chinese translation sentences" and "model translation results" of the two groups of control texts, it can be directly found that the semantics and numbers in the "model translation results" are completely correct, and there are slight differences in the expression of symbols and synonyms. It is proved that the machine translation model based on symmetric attention has good performance in machine translation work.

[0098] In summary, the machine translation model based on symmetric attention has good effect in machine translation work, and has obvious improvement in accuracy compared with the Transformer model of the prior art.

Claims

1. A machine translation method based on symmetric attention, characterized in that, A symmetric attention layer is set in the decoder of the machine translation model. A first symmetric attention branch is computed in the first symmetric subspace of this layer, where the query matrix Q' comes from the decoder, and the key matrix K and real-value matrix V come from the encoder. A second symmetric attention branch is computed in the second symmetric subspace of this layer, where matrices Q and K come from the decoder, and matrix V' comes from the decoder. The two attention branches are then fused using a normalization function. The specific steps of this method include the following: Step 1, Generate training set: Step 1.1: Select at least 270,000 English sentences and their corresponding Chinese translations to form a sample set; Step 1.2: Remove duplicate English sentences and their corresponding Chinese translations from the sample set. After removing English sentences and their corresponding Chinese translations that are too short or too long relative to the average length of all English sentences, the training set is obtained. Step 2, construct a machine translation model based on symmetric attention: Step 2.1: Construct an encoder consisting of a multi-head self-attention layer, a residual layer, and a feedforward layer connected in series. The multi-head self-attention layer contains at least 8 subspaces, and each subspace calculates the self-attention of the current predicted word and all words in its sentence through dot product multiplication. The residual layer is implemented by a layer normalization function; The structure of the forward feedback layer is as follows: input layer, fully connected layer, and output layer. The dimensions of the input layer and the output layer are both set to 512. The fully connected layer is implemented by a maximum value function and its dimension is set to 2048. Step 2.2: Construct a decoder consisting of a multi-head self-attention layer, a first residual layer, a symmetric attention layer, a second residual layer, a forward feedback layer, and a third residual layer connected in series. The structure and parameters of the multi-head self-attention layer and the forward feedback layer in the decoder are the same as those of the multi-head self-attention layer and the forward feedback layer in the encoder in step 2.

1. The first residual layer, the second residual layer, and the third residual layer in the decoder are all implemented by a layer normalization function; The symmetric attention layer consists of two parallel symmetric subspaces. The symmetric attention algorithm is used to calculate the two symmetric attentions and fuse them to realize information interaction between the layer and the attention matrix output by the encoder. The symmetric attention algorithm refers to extracting correlation information from two parallel symmetric subspaces and performing linear transformations on the two correlation information to obtain two feature matrices. The symmetric attention for each feature matrix is ​​computed using dot product multiplication; By using a summation function, two symmetric attention types are merged, and the sum of the two symmetric attention types is normalized at each layer to output a symmetric attention matrix. Step 2.3: Connect the input layer, the first embedded expression layer, the first encoder, the second encoder, the third encoder, the fourth encoder, the fifth encoder, and the sixth encoder in series to form an encoder group; The first embedded expression layer is implemented by an embedding algorithm; The first through sixth encoders have the same structure and parameters; Step 2.4: Construct a decoder group in which the second embedded expression layer, the first decoder, the second decoder, the third decoder, the fourth decoder, the fifth decoder, the sixth decoder, and the activation layer are connected in series. The activation layer is also connected to the second embedded expression layer. The first to sixth decoders are also connected to the sixth encoder in the encoder group. The structure and parameters of the second embedded expression layer are the same as those of the first embedded expression layer; The first through sixth decoders have the same structure and parameters; The activation layer is implemented using the Softmax activation function; Step 2.5: Connect the encoder group and decoder group in series to form a machine translation model based on symmetric attention; Step 3, train the machine translation model: The training set is input into the machine translation model in batches. The loss function is used to calculate the loss value between the translated text and the corresponding Chinese text. The gradient descent algorithm is used to iteratively update the learnable parameters in the machine translation model. The Adam optimizer is used to optimize the training process until the loss value of the machine translation model converges, thus obtaining the trained machine translation model based on symmetric attention. Step 4: Use the trained machine translation model to perform the translation: Step 4.1: Remove duplicate English sentences from the English sentences to be translated, and eliminate English sentences that are either too short or too long relative to the average length of all English sentences to be translated. Step 4.2: Input the processed sentence to be translated into the trained machine translation model based on symmetric attention, and output the corresponding translated text.

2. The machine translation method based on symmetric attention according to claim 1, characterized in that, The layer normalization function mentioned in step 2.1 is as follows: R(x1)=LayerNorm(Add(x1,Sublayer(x1))) Where x1 is the input of the multi-head self-attention layer, R(·) is the output of the residual layer, LayerNorm(·) is the layer normalization function, Add(·) is the summation function, and Sublayer(·) is the output of the multi-head self-attention layer.

3. The machine translation method based on symmetric attention according to claim 1, characterized in that, The maximum value function described in step 2.1 is as follows: FFN(x2)=Max(0,x2W1+b1)W2+b2 Where x2 is the input of the feedforward layer, FFN(·) is the output of the feedforward layer, Max(·) is the maximum value operation, and W1, W2, b1, and b2 are learnable parameters that are iteratively updated using the gradient descent algorithm. The initial size of the matrices W1 and b1 is set to 512×2048 zero matrices, and the initial size of the matrices W2 and b2 is set to 2048×512 zero matrices.

4. The machine translation method based on symmetric attention according to claim 1, characterized in that, The steps of the symmetric attention algorithm described in step 2.2 are as follows: The first step is to use Q′=W Q′ h′ and V′ = W V′ The h′ formula linearly transforms the h′ input to the symmetric attention layer, where Q′ is the projection matrix of the correlation between the currently processed word and other words in its sentence onto the first symmetric subspace, V′ is the projection matrix of the correlation between the currently processed word and other words in its sentence onto the second symmetric subspace, and W... Q′ and W V′ The learnable parameters are iteratively updated using the gradient descent algorithm. Their initial matrices are all set to zero matrices of size 512×512, and h' is the correlation information between the currently processed word and the preceding words in its sentence. The second step, through The formula calculates the first symmetric attention result output by the symmetric attention layer, where Attention1 is the first symmetric attention result output by the symmetric attention layer, Softmax(·) is the activation function, q1 is the global vector corresponding to matrix Q' in the first symmetric subspace, and K1 and V1 are the projection matrices of the attention processing English sentences in the first symmetric subspace when updating network parameters in the current iteration. The superscript · T For the transpose operation, d k When updating network parameters for the current iteration, process the word embedding vector dimension of each word in the English sentence; The third step, through The formula calculates the second symmetric attention result output by the symmetric attention layer, where Attention2 is the second symmetric attention result output by the symmetric attention layer, q2 is the global vector corresponding to matrix Q2 in the second symmetric subspace, Q2 and K2 are both projection matrices of the attention to the English sentence in the second symmetric subspace when updating the network parameters in the current iteration, and V' is the projection matrix of the correlation between the word being processed and other words in the sentence in the second symmetric subspace when updating the network parameters in the current iteration. The fourth step involves fusing two symmetric attention mechanisms using the formula Attention = LayerNorm(Attention1 + Attention2). Here, Attention is a symmetric attention matrix of size P × P, where P represents the length of the English sentence processed during the current iteration's parameter update, and the element a in the i-th row and j-th column of the matrix is... ij This indicates that when updating network parameters in the current iteration, symmetric attention is applied between the i-th and j-th words in an English sentence. LayerNorm(·) is the layer normalization function.

5. The machine translation method based on symmetric attention according to claim 1, characterized in that: The loss function mentioned in step 3 is: Where L is the loss value between the translated text feature matrix generated based on the Chinese vocabulary and the Chinese text feature matrix generated based on the Chinese vocabulary. Let y be the translated text feature matrix generated based on the Chinese vocabulary, and log be the logarithmic operation to base 10.

Images