Translation method, device and storage medium based on neural machine translation model

Abstract

The present invention discloses a translation method based on a neural machine translation model. The translation is based on an autoregressive decoding method. Each step of generating the next word is based on a source end sequence and a target end history sequence generated by decoding. Each step of the autoregressive decoding includes the following: based on the source end sequence, a basic source end representation and a controller source end representation are obtained by an encoder of a neural machine translation model; based on the target end history sequence and the basic source end representation, the module weights and weight vectors of the basic target end representation and the top cross-attention layer of the decoder of the neural machine translation model are obtained; based on the weight vector of the top cross-attention layer of the decoder, a digital word to be translated is obtained and translated into a target end digital word; based on the basic target end representation and the translated target end digital word, a word probability is comprehensively calculated. The present invention strengthens the digital translation capability of the neural machine translation model and solves the problems of lack of guarantee for the accuracy of digital translation and poor generalization capability.

Description

Technical Field

[0001] The present invention relates to the field of neural networks, and in particular to a translation method, device and storage medium based on a neural machine translation model. Background Art

[0002] Internet technology is developing rapidly and expanding at an unprecedented scale. Driven by the dual catalysts of mobile internet and the pandemic, the internet has become an inseparable part of people's lives. Every day, the internet produces a vast flow of information in a variety of languages. This makes translation a pressing need for human society. However, expensive and time-consuming manual translation not only struggles to process this vast amount of text data, but also fails to meet people's needs for the free-flowing online browsing experience. Machine translation is an important means to address this problem.

[0003] Machine translation is the technology of automatically translating text data using computers. With the development of computer computing power and deep learning technology, machine translation technology based on deep neural networks has become quite practical, even reaching or exceeding the human level in some tasks. Mainstream neural machine translation models typically use an encoder-decoder framework: for a given source language sentence of length (where represents the th word), the model's encoder first encodes it into a source-side representation in the form of a numerical vector, which is then input into the decoder for decoding and translation into a target language sentence of length. The decoder typically operates in an autoregressive manner: in the th step, based on the generated target-side historical sequence, the probability distribution of the th target-side word is predicted, that is,The autoregressive neural machine translation model based on the encoder-decoder framework can be implemented by a variety of model structures: CNN model (reference Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia,6-11 August 2017, volume70 of Proceedings of Ma-chine Learning Research, pages 1243–1252. PMLR.), RNN model (reference Zaremba, Wojciech, Ilya Sutskever, and Oriol Vinyals. "Recurrent neural network regularization."arXiv preprint arXiv:1409.2329(2014).), Transformer model (reference Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.). Among them, the Transformer model is currently the most powerful model.

[0004] Transformer is a model composed entirely of a multi-head attention mechanism and a feed-forward neural network. Figure 1 As shown in Figure 1. The attention mechanism takes three vector sequences as input (where corresponds to K ) and uses a specified similarity function to calculate the similarity between each pair of vectors with K . This produces a similarity matrix, with the linear average of the weights used as the output. The multi-head attention mechanism, on the other hand, first maps the vector sequence of dimension into a sub-vector sequence of dimension using a parameter matrix. These sub-vectors are then calculated separately and then merged back to their original dimensions. On the decoder side, the multi-head attention mechanism uses the hidden state of the previous layer as input, a mechanism known as a self-attention mechanism. In addition to the self-attention mechanism, the decoder also uses a cross-attention mechanism, using the output of the previous self-attention layer of the decoder as and the output of the encoder as . The Transformer model based on the attention mechanism has advantages such as high parallelism and the ability to model long-range dependencies. The attention mechanism also provides a degree of interpretability for the model: by observing the distribution of attention weights, one can understand the contribution of the source word or context to the current prediction.

[0005] The introduction of the Transformer has driven significant progress in both fluency and accuracy in machine translation models, but reaching human-level performance remains challenging in some areas. When using modern translation software, errors that human translators clearly wouldn't make are often found. For example, when faced with complex numbers or numbers in context, modern translation software still makes obvious numerical errors. See Appendix. Figure 2As shown, Google Translate and Microsoft Translate still make mistakes when dealing with this type of numbers. Although many precise Roman numerals can be directly replaced by copying instead of translation, the translation of more common numerical expressions has certain complexities. Different languages may use different number systems. For example, Chinese uses shi (ten), bai (hundred), qian (thousand), wan (ten thousand), yi (hundred million), while English uses hundred, thousand, million, billion, etc. This requires converting the number systems during translation and demands that the translation model has a correct understanding of the numerical magnitudes of the numbers. In different contexts, the same number may also have different meanings. For example, "三日" can either mean "three days" or "the third day of a certain month (3th)". When translating numbers, the translation model not only needs to handle these complexities but also ensure the accuracy of the translated numbers. Different from the translation of other natural languages, the translation of numbers has extremely high requirements for accuracy, and modern translation models based on deep neural networks lack guarantees for this. Additionally, the training sets used by data-driven deep neural translation models are always limited, while the combinations of numbers are infinite, and it is difficult for neural translation models to extend to new magnitudes not seen during training. Although the combination rules of numbers are relatively simple, research shows (Literature LI Y, ZHAO L, WANG J, et al. CompositionalGeneralization for Primitive Substitutions[C / OL] / / Proceedings of the 2019Conference on Empirical Methods in Natural Language Processing and the 9thInternational Joint Conference on Natural Language Processing (EMNLP-IJCNLP).Hong Kong, China : Association for Computational Linguistics, 2019 : 4293 –4302.) that neural translation models such as Transformer still have difficulty in learning and applying these rules well.

[0006] Existing research on the number recognition ability of deep neural networks focuses on the numerical common sense possessed by pretrained language models (references SPITHOURAKIS G, RIEDEL S. Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers[C / OL] / / Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia :Association for Computational Linguistics, 2018 : 2104 – 2115.; references WALLACE E, WANG Y, LI S, et al. Do NLP models know numbers? probing numeracy inembeddings[J]. arXiv preprint arXiv:1909.07940, 2019., etc.) and mathematical word problems (MathWord Problem) (references Zhanming Jie, Jierui Li, and Wei Lu. 2022. Learning toReason Deductively: Math Word Problem Solving as Complex Relation Extraction.In Proceedings of the 60th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers), pages 5944–5955, Dublin, Ireland. Association for Computational Linguistics., etc. There is a lack of research on the problem of digital translation in machine translation tasks and its improvement solutions.

[0007] The existing technology has the following problems: digital translation requires high accuracy, but the existing neural machine translation models lack accuracy guarantees; the training sets of existing neural machine translation models are always limited, making it difficult to generalize to numbers with infinite combinations; as a result, the existing neural machine translation models are prone to errors in translating numbers and have poor generalization capabilities. Summary of the Invention

[0008] In order to overcome the high accuracy requirements for digital translation in the above-mentioned background technology, but the existing neural machine translation model lacks accuracy guarantee; the training set of the existing neural machine translation model is always limited, and it is difficult to generalize to numbers with infinite combinations; resulting in the shortcomings of the existing neural machine translation model being prone to errors and poor generalization ability in translating numbers, the purpose of the present invention is to provide a translation method based on a neural machine translation model.

[0009] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0010] The first aspect of the present invention provides a translation method based on a neural machine translation model, wherein the translation is based on an autoregressive decoding method, and each step generates the next word y j , are based on the source sequence and the decoded target historical sequence Y j-1 ={y0, y1, ...y j-1}, the decoding process of each step of the autoregressive decoding includes the following steps:

[0011] S1: Based on the source sequence, a base source representation and a controller source representation are obtained through an encoder of a neural machine translation model;

[0012] S2: Based on the target-side historical sequence and the basic source-side representation, obtain the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the neural machine translation model;

[0013] S3: Based on the weight vector of the cross-attention layer at the top of the decoder of the neural machine translation model, obtain the digital word to be translated and translate it into the target digital word;

[0014] S4: Based on the basic target-side representation and the translated target-side digital word, comprehensively calculate word probability.

[0015] In some possible implementations, the specific steps of obtaining a basic source representation and a controller source representation based on the source sequence through an encoder of a neural machine translation model include the following:

[0016] Let the source sequence X = {x1, x2, ...x m}, i={1, 2, ...m}, where x iRepresents the i-th subword in the source sentence, where the source sequence uses BPE segmentation and the numeric words use character-level segmentation;

[0017] The source sequence X after word segmentation is input into the encoder of the Transformer translation model, and the encoding is performed to obtain the basic source representation vector sequence E{e1, e2, ...e m}, i={1, 2, ...m}, where e i Represents the i-th subword in the source sentence;

[0018] Assume that the source digital label sequence L src ={l1,l2,…l m}, i={1, 2, ...m}, where l i Indicates that the i-th subword is a number or a subword of a number. If the i-th subword is not a number or a subword of a number, then l i =0;

[0019] The source digital label sequence L src By encoding the label word vector, we get the label word vector sequence E l ={e l1 , e l2 ,…e lm}, i={1, 2, ...m}, where e li The numeric label word vector representing the i-th subword;

[0020] Connect the basic source representation vector and the label word vector element by element to obtain the connection vector H0={h 01 , h 02 ,…h 0m}, i={1, 2, ...m}, where h 0i =e i |e li , “|” indicates vector connection, e li Represents the connection vector corresponding to the i-th subword in the source language sentence;

[0021] After encoding by the controller encoder, the controller source end representation H={h1, h2,…h m}, the controller encoder is constructed as follows:

[0022]

[0023] Among them, FFN represents a fully connected neural network and acti represents an arbitrary activation function.

[0024] In some possible implementations, the specific steps of obtaining the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the neural machine translation model based on the target-side history sequence and the basic source-side representation include the following:

[0025] Input basic source representation vector sequence E and target historical sequence Y j-1 , the next step of decoding is performed through the decoder of the Transformer translation model to obtain the basic target end representation vector f of step j j And the weight vector W of the top cross-attention layer of the decoder of the Transformer translation model j ={w j1 , w j2 ,…w jm}, i={1, 2, ...m}, where w jI Indicates the weight corresponding to the i-th subword at the source end;

[0026] Based on the generated historical target-side representation vector sequence F j ={f1, f2, ...f j} and the controller source representation H, which is decoded by the controller decoder to obtain the module weight t of step j j , the controller decoder adopts the same structure as the decoder of the Transformer translation model;

[0027] During the training phase, a binary 0-1 label l is used t j Supervision module weight t j Learning, l t j =1 means that the jth target terminal word is the starting position of a number word. If the jth target terminal word is not the starting position of a number word, then l t j is 0.

[0028] In some possible implementations, the specific steps of obtaining a numeric word to be translated based on the weight vector of the top cross-attention layer of the decoder of the neural machine translation model and translating it into a target numeric word include the following:

[0029] According to the weight vector w of the cross attention layer j , the position with the largest weight is the subword to be translated in this step, and thus the position of the digital word to be translated is obtained as k=argmax(w j );

[0030] The digital translation module obtains the complete digital word containing the k-th subword of the source through the rules. If the k-th subword is not a digital word, let t j= 0, if the kth subword is a number word, then the complete number word is input into the number translation module and translated into the number word N of the target language in the manner of the rule j ={n j 1,n j 2,…n j pj}, where n i represents the i-th subword of the target language's number word;

[0031] If before step j, there are already translated number words N l ={n l 1,n l 2,…n l pl}, and l+pl>j, then skip the step of translating the complete number word, otherwise let l:=j and N l :=N j .

[0032] In some possible implementations, the specific steps of comprehensively calculating word probabilities based on the basic target-side representation and the translated target-side digital word include the following:

[0033] Calculate the probability distribution of all words in step j, the formula is:

[0034]

[0035] Among them, v j is a random variable of the word generated by decoding in step j, and its value is any word of the target source;

[0036] Obtained by the output layer of the Transformer translation model is a 0-1 function, currently only when When is 1;

[0037] Finally, based on the probability distribution of all words in step j, the complete target language translation result Y={y1, y2, ...y n}, n represents the sequence length;

[0038] If the translation result y in step j j From the digital translation module, after the translation results of the digital translation module are completely output, the translation results of the Transformer translation model are reused; according to the Bayesian posterior theorem, P(v j ) in the digital translation module and the Transformer translation model: , then it is considered that for the subword v j, the contribution of the digital translation module is greater, and then let t j =1 until the result of the digital translation module is output.

[0039] The second aspect of the present invention provides a translation device based on a neural machine translation model, comprising

[0040] A first acquisition module: based on the source sequence, acquires a basic source representation and a controller source representation through an encoder of a neural machine translation model;

[0041] A second acquisition module is configured to acquire, based on the target-side historical sequence and the basic source-side representation, the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the neural machine translation model;

[0042] The third acquisition module: based on the weight vector of the cross-attention layer at the top of the decoder of the neural machine translation model, obtains the digital word to be translated and translates it into the target digital word;

[0043] Calculation module: comprehensively calculates word probability based on the basic target-side representation and the translated target-side digital word;

[0044] The translation is based on autoregressive decoding, generating the next word y at each step. j , are based on the source sequence and the decoded target historical sequence Y j-1 ={y0, y1, ...y j-1}, the decoding process of each step of the autoregressive decoding is implemented through the above modules.

[0045] In some possible implementations, the specific steps of obtaining a basic source representation and a controller source representation based on the source sequence through an encoder of a neural machine translation model include the following:

[0046] Let the source sequence X = {x1, x2, ...x m}, i={1, 2, ...m}, where x i Represents the i-th subword in the source sentence, where the source sequence uses BPE segmentation and the numeric words use character-level segmentation;

[0047] The source sequence X after word segmentation is input into the encoder of the Transformer translation model, and the encoding is performed to obtain the basic source representation vector sequence E{e1, e2, ...e m}, i={1, 2, ...m}, where e i Represents the i-th subword in the source sentence;

[0048] Assume that the source digital label sequence L src ={l1,l2,…l m}, i={1, 2, ...m}, where l i Indicates that the i-th subword is a number or a subword of a number. If the i-th subword is not a number or a subword of a number, then l i =0;

[0049] The source digital label sequence L src By encoding the label word vector, we get the label word vector sequence E l ={e l1 , e l2 ,…e lm}, i={1, 2, ...m}, where e li The numeric label word vector representing the i-th subword;

[0050] Connect the basic source representation vector and the label word vector element by element to obtain the connection vector H0={h 01 , h 02 ,…h 0m}, i={1, 2, ...m}, where h 0i =e i |e li , “|” indicates vector connection, e li Represents the connection vector corresponding to the i-th subword in the source language sentence;

[0051] After encoding by the controller encoder, the controller source end representation H={h1, h2,…h m}, the controller encoder is constructed as follows:

[0052]

[0053] Among them, FFN represents a fully connected neural network and acti represents an arbitrary activation function.

[0054] In some possible implementations, the specific steps of obtaining the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the neural machine translation model based on the target-side history sequence and the basic source-side representation include the following:

[0055] Input basic source representation vector sequence E and target historical sequence Y j-1 , the next step of decoding is performed through the decoder of the Transformer translation model to obtain the basic target end representation vector f of step j j And the weight vector W of the top cross-attention layer of the decoder of the Transformer translation model j ={w j1 , w j2 ,…w jm}, i={1, 2, ...m}, where w jIIndicates the weight corresponding to the i-th subword at the source end;

[0056] Based on the generated historical target-side representation vector sequence F j ={f1, f2, ...f j} and the controller source representation H, which is decoded by the controller decoder to obtain the module weight t of step j j , the controller decoder adopts the same structure as the Transformer decoder;

[0057] During the training phase, a binary 0-1 label l is used t j Supervision module weight t j Learning, l t j =1 means that the jth target terminal word is the starting position of a number word. If the jth target terminal word is not the starting position of a number word, then l t j is 0.

[0058] In some possible implementations, the specific steps of obtaining a numeric word to be translated based on the weight vector of the top cross-attention layer of the decoder of the neural machine translation model and translating it into a target numeric word include the following:

[0059] According to the weight vector w of the cross attention layer j , the position with the largest weight is the subword to be translated in this step, and thus the position of the digital word to be translated is obtained as k=argmax(w j );

[0060] The digital translation module obtains the complete digital word containing the k-th subword of the source through the rules. If the k-th subword is not a digital word, let t j = 0, if the kth subword is a number word, then the complete number word is input into the number translation module and translated into the number word N of the target language in the manner of the rule j ={n j 1,n j 2,…n j pj}, where n i represents the i-th subword of the target language's number word;

[0061] If before step j, there are already translated number words N l ={n l 1,n l 2,…n l pl}, and l+pl>j, then skip the step of translating the complete number word, otherwise let l:=j and N l :=N j.

[0062] In some possible implementations, the specific steps of comprehensively calculating word probabilities based on the basic target-side representation and the translated target-side digital word include the following:

[0063] Calculate the probability distribution of all words in step j, the formula is:

[0064]

[0065] Among them, v j is a random variable of the word generated by decoding in step j, and its value is any word of the target source;

[0066] Obtained by the output layer of the Transformer translation model is a 0-1 function, currently only when When is 1;

[0067] Finally, based on the probability distribution of all words in step j, the complete target language translation result Y={y1, y2, ...y n}, n represents the sequence length;

[0068] If the translation result y in step j is j From the digital translation module, after the translation results of the digital translation module are completely output, the translation results of the Transformer translation model are reused; according to the Bayesian posterior theorem, P(v j ) in the digital translation module and the Transformer translation model: , then it is considered that for the subword v j , the contribution of the digital translation module is greater, and then let t j =1 until the result of the digital translation module is output.

[0069] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the translation method based on the neural machine translation model are implemented.

[0070] The beneficial effect of the present invention lies in: embedding a rule-based digital translation module into a neural machine translation model. Because the translation results of rules are always single and deterministic, errors in the case of multiple meanings are difficult to correct. A controller is also trained to determine the confidence level of the translation results of the digital translation module and the neural machine translation model, and finally combines the two to make a prediction, thereby improving the digital translation accuracy and generalization ability of the neural machine translation model at a low computational cost.

[0071] From a technical perspective, the present invention uses a rule-based digital translation module to supplement the digital translation capabilities of the neural machine translation model. By embedding additional controllers and digital translation modules, the accuracy of the neural machine translation model in translating numbers is enhanced, making up for the lack of accuracy guarantee for digital translation and poor generalization ability of the neural machine translation model. In the process, only a lightweight controller needs to be trained, and the additional computing cost introduced is relatively low.

[0072] From an application perspective, experimental results on the NIST-Chinese-English dataset demonstrate that the present invention can further improve the accuracy of digital translation without reducing the general translation performance of the neural machine translation model itself, demonstrating the effectiveness of the present invention. It can be directly embedded in any trained neural machine translation model for use without the need to retrain or fine-tune the translation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a schematic diagram of the existing Transformer translation model;

[0074] Figure 2 This is a diagram showing an error case of translating numbers using existing translation software;

[0075] Figure 3 This is a flowchart of the overall steps of the translation method based on the neural machine translation model according to an embodiment of the present invention;

[0076] Figure 4 This is a translation flow chart of an embodiment of the present invention;

[0077] Figure 5 4 is a structural diagram of a translation device based on a neural machine translation model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0079] Some existing techniques can also be used to enhance the digital translation ability of the translation model. For example, a copy mechanism can be introduced to copy the numbers from the source side to the corresponding positions on the target side, and then a dedicated digital translation module is used to translate these numbers during post-processing and insert them into the same positions. Or all numbers can be replaced with special identifiers during the preprocessing stage. The translation model is trained to generate the same identifiers on the target side to represent the corresponding numbers on the source side, and then the numbers corresponding to the identifiers are translated and replaced during the post-processing stage. However, these solutions introduce new words into the vocabulary, causing a significant change in the data distribution that the model is exposed to. Therefore, after introducing these solutions, the model needs to be retrained from scratch on the entire reprocessed training set.

[0080] This embodiment is illustrated by taking a Chinese-English Transformer translation model system as an example. The input source language (i.e., the source side) is Chinese, and the source language sequence is "In 2003, the total revenue of China's tourism industry is expected to be 487.4 billion yuan". The desired target language to be output is English, and the desired target language sequence to be output is "total revenue from china's tourism industry estimated to be rmb 487.4 billion in 2003".

[0081] See Appendix Figure 3 and Appendix Figure 4 As shown, this embodiment provides a translation method based on a neural machine translation model. The translation is based on an autoregressive decoding method, and each step generates the next word y j , which is based on the source-side sequence and the target-side historical sequence Y j-1 ={y0, y1,... y j-1}, where y0 = [BOS] is fixed, representing the beginning of the sentence. Each step of the autoregressive decoding process includes the following steps:

[0082] S1: Based on the source-side sequence, use the encoder of the Transformer translation model to obtain the basic source-side representation and the controller source-side representation, which specifically includes the following steps:

[0083] Let the source-side sequence X = {x1, x2,... x m}, i = {1, 2,... m}, where x i represents the i-th subword in the source-side sentence. The source-side sequence uses BPE tokenization, and all numeric words use character-level tokenization. For example, "4874 billion" in the embodiment is tokenized as "4@@ 8@@ 7@@ 4@@ billion".

[0084] The source sequence X after word segmentation is input into the encoder of the Transformer translation model, and the encoding is performed to obtain the basic source representation vector sequence E{e1, e2, ...e m}, i={1, 2, ...m}, where e i Represents the i-th subword in the source sentence;

[0085] Assume that the source digital label sequence L src ={l1, l2, ...l m}, i={1, 2, ...m}, where l i Indicates that the i-th subword is a number or a subword of a number. If the i-th subword is not a number or a subword of a number, then l i =0;

[0086] The source digital label sequence L src By encoding the label word vector, we get the label word vector sequence E l ={e l1 , e l2 ,…e lm}, i={1, 2, ...m}, where e li The numeric label word vector representing the i-th subword;

[0087] Connect the basic source representation vector and the label word vector element by element to obtain the connection vector H0={h 01 , h 02 ,…h 0m}, i={1, 2, ...m}, where h 0i =e i |e li , “|” indicates vector connection, e li Represents the connection vector corresponding to the i-th subword in the source language sentence;

[0088] After encoding by the controller encoder, the controller source end representation H={h1, h2,…h m}, the controller encoder is constructed as follows:

[0089]

[0090] Among them, FFN represents a fully connected neural network and acti represents an arbitrary activation function.

[0091] S2: Based on the target-side historical sequence and the basic source-side representation, obtain the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the Transformer translation model, specifically including the following steps:

[0092] Input basic source representation vector sequence E and target historical sequence Y j-1, the next step of decoding is performed through the decoder of the Transformer translation model to obtain the basic target end representation vector f of step j j And the weight vector W of the top cross-attention layer of the decoder of the Transformer translation model j ={w j1 , w j2 ,…w jm}, i={1, 2, ...m}, where w jI Indicates the weight corresponding to the i-th subword at the source end;

[0093] Based on the generated historical target-side representation vector sequence F j ={f1, f2, ...f j} and the controller source representation H, which is decoded by the controller decoder to obtain the module weight t of step j j , the controller decoder adopts the same structure as the Transformer decoder;

[0094] During the training phase, a binary 0-1 label l is used t j Supervision module weight t j Learning, l t j =1 means that the jth target terminal word is the starting position of a number word. If the jth target terminal word is not the starting position of a number word, then l t j is 0.

[0095] S3: Based on the weight vector of the top cross-attention layer of the decoder of the Transformer translation model, obtain the numeric word to be translated and translate it into the target numeric word, which specifically includes the following steps:

[0096] According to the weight vector w of the cross attention layer j , the position with the largest weight is the subword to be translated in this step, and thus the position of the digital word to be translated is obtained as k=argmax(w j ).

[0097] The digital translation module obtains the complete digital word containing the k-th subword of the source through the rules. If the k-th subword is not a digital word, let t j = 0, if the kth subword is a number word, then the complete number word is input into the number translation module and translated into the number word N of the target language in the manner of the rule j ={n j 1,n j 2,…n j pj}, where n iRepresents the i-th sub-word of the numeric word in the target language.

[0098] If before the j-th step, there is already a translated numeric word N l ={n l 1, n l 2,... n l pl}, and l + pl > j, then skip the step of translating the complete numeric word, otherwise let l := j and N l := N j .

[0099] For example, in the embodiment, "4@@ 8@@ 7@@ 4@@ billion" is translated as "4@@ 8@@ 7@@.@@ 4 billion". If "4@@" has been output at the (j - 1)-th step, the j-th step should then output "8@@", so skip the step of obtaining the complete source-side numeric word at the j-th step and use the previous "4@@ 8@@ 7@@.@@ 4 billion". That is:

[0100] , and l = j - 1.

[0101] S4: Based on the aforementioned base target-side representation and the translated target-side numeric word, comprehensively calculate the word probabilities, which specifically include the following steps:

[0102] Calculate the probability distribution of all words at the j-th step, and the formula is:

[0103]

[0104] [[ID=三十五]]Where v j is the random variable of the word generated by decoding at the j-th step, and its value range is any word in the target language.

[0105] Obtained from the output layer of the Transformer translation model is a 0 - 1 function, and is 1 only when .

[0106] Finally, based on the probability distribution of all words at the j-th step, through the beam search algorithm, obtain the complete target-language translation result Y = {y1, y2,... y n}, n represents the sequence length.

[0107] If the translation result y at the j-th step j comes from the numeric translation module, then after completely outputting the translation result of the numeric translation module, reuse the translation result of the Transformer translation model to ensure its accuracy; according to Bayes' posterior theorem, judge P(v j)The contribution sizes from the digital translation module and the Transformer translation model: If , it is considered that for the subword v j , the contribution of the digital translation module is greater. After that, let t j = 1 until the result of the digital translation module is completely output.

[0108] For example, when translating "4@@ 8@@ 7@@ 4@@ billion" in the translation example, the digital translation module translates it as "4@@8@@ 7@@.@@ 4 billion". In the first step, the translation result of the digital translation module is "4@@". Assume that in this step, in the word probability distribution output by the Transformer translation model, the probability of the subword "4@@" is 0.5, and the weight output by the controller is 0.9. If according to the beam search algorithm, it is determined that the translation result of this step is indeed "4@@", then according to the above discriminant judge the source of the translation result "4@@". In this example, substituting the values gives 0.9 * 1 > (1 - 0.1) * 0.5, and this formula holds. Therefore, it is considered that the translation result "4@@" is obtained by the digital translation module. So in the subsequent steps, the weight output by the controller is always set to 1. In this way, the output of the Transformer translation model will be temporarily blocked, and the beam search algorithm can only adopt the translation result of the digital translation module until the translation result "4@@ 8@@ 7@@.@@ 4 billion" of the digital translation module is completely generated. Through this method, the unity of the translation source of single-digit words is ensured.

[0109] In the above steps, the rule-based digital translation module realizes two functions:

[0110] 1. In S3, given a source-side position, obtain the largest digital word containing this position. The adopted scheme is: In the preprocessing stage, all digital words in the source language sentence are matched by regular expressions, and their starting position s and ending position e are recorded. Then, after obtaining the source-side digital word position k in the decoding of S3, retrieve the digital word that satisfies s ≤ k < e. The recorded digital words are also used to generate the digital label sequence in S2. [[ID=1十八]]

[0111] 2. Translation of source language digital words: Let the obtained source-side digital word be W n , first judge its type through regular expressions, such as time, date, percentage, ordinal number, and numerical value, etc., and then convert it into a numerical form character by character. At this time, different digital types and units may affect the numerical size. For example, the percentage "forty percent" means "40%", and "4" cannot be directly obtained from "four". In addition, time and date may contain multiple numerical values, and the meaning of each numerical value needs to be judged according to the unit or format. Finally, convert the numerical value into a target language string.

[0112] In the above method, in addition to the Transformer translation model, it includes two modules: a controller and a digital translation module. The above controller (including a controller encoder and a controller decoder)

[0113] In the above method, the basic source-side representation is obtained by encoding by the encoder of the Transformer translation model, and the basic target-side representation and the basic target-side sequence probability distribution are obtained by decoding by the decoder of the Transformer translation model.

[0114] In the above method, the module weights are obtained by decoding the controller. The controller also adopts an encoder-decoder framework and uses the source word type label as input in addition to the basic source and target representations.

[0115] In the above method, the translation of the digital word is obtained using a digital translation module, which uses a regular translation method.

[0116] In the above method, the source numeric word to be translated is used to obtain a subword through the cross-attention layer of the decoder of the Transformer translation model, and then the complete numeric word is obtained through the rules.

[0117] In the above method, the controller and digital translation module can be embedded in any trained basic translation model, and the training phase of the controller does not require any training or fine-tuning of the Transformer translation model.

[0118] See attached Figure 5 As shown, this embodiment also provides a translation device based on a neural machine translation model. This translation device implements the steps of the translation method based on the neural machine translation model during operation. The translation device includes

[0119] The first acquisition module: based on the source sequence, obtains the basic source representation and the controller source representation through the encoder of the Transformer translation model;

[0120] The second acquisition module is configured to acquire the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the Transformer translation model based on the target-side historical sequence and the basic source-side representation;

[0121] The third acquisition module: based on the weight vector of the top cross-attention layer of the decoder of the Transformer translation model, obtains the digital word to be translated and translates it into the target digital word;

[0122] Calculation module: comprehensively calculates word probability based on the basic target-side representation and the translated target-side digital word;

[0123] The translation is based on autoregressive decoding, generating the next word y at each step. j , are based on the source sequence and the decoded target historical sequence Y j-1 ={y0, y1, ...y j-1}, the decoding process of each step of the autoregressive decoding is implemented through the above modules.

[0124] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned translation method based on the neural machine translation model are implemented.

[0125] The storage medium stores program instructions capable of implementing all of the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code, or a terminal device such as a computer, server, mobile phone, or tablet.

[0126] The processor may also be referred to as a CPU (Central Processing Unit). A processor may be an integrated circuit chip with signal processing capabilities. A processor may also be:

[0127] DSP (Digital Signal Processor, Digital Signal Processor, Digital Signal Processor is a processor composed of large-scale or ultra-large-scale integrated circuit chips used to complete certain signal processing tasks. It has gradually developed to meet the needs of high-speed real-time signal processing tasks. With the development of integrated circuit technology and digital signal processing algorithms, the implementation methods of digital signal processors are constantly changing, and the processing functions are constantly improving and expanding.)

[0128] ASIC (Application Specific Integrated Circuit) is an integrated circuit designed and manufactured to meet specific user requirements and the needs of specific electronic systems.

[0129] The FPGA (Field Programmable Gate Array) is a further development of programmable devices such as PAL (Programmable Array Logic) and GAL (Generic Array Logic). It emerged as a semi-custom circuit within the field of application-specific integrated circuits (ASICs), addressing the shortcomings of custom circuits while overcoming the limited number of gates in existing programmable devices.

[0130] A general-purpose processor may be a microprocessor or any conventional processor.

[0131] Other programmable logic devices, discrete gate or transistor logic devices and discrete hardware components, etc.

[0132] The above embodiments are only for illustrating the technical concept and features of the present invention. Its purpose is to enable people familiar with this technology to understand the content of the present invention and implement it. It cannot be used to limit the scope of protection of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A translation method based on a neural machine translation model, characterized by: The translation is based on autoregressive decoding, generating the next word y at each step. j , are based on the source sequence and the decoded target historical sequence Y j-1 ={y0, y1, ...y j-1 }, the decoding process of each step of the autoregressive decoding includes the following steps: S1: Based on the source sequence, a base source representation and a controller source representation are obtained through an encoder of a neural machine translation model; S2: Based on the target-side historical sequence and the basic source-side representation, obtain the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the neural machine translation model; S3: Based on the weight vector of the cross-attention layer at the top of the decoder of the neural machine translation model, obtain the digital word to be translated and translate it into the target digital word; S4: comprehensively calculating word probabilities based on the basic target-side representation and the translated target-side digital words; The specific steps of obtaining the basic source representation and the controller source representation based on the source sequence through the encoder of the neural machine translation model include the following: Let the source sequence X = {x1, x2, ...x m }, i={1, 2, ...m}, where x i Represents the i-th subword in the source sentence, where the source sequence uses BPE segmentation and the numeric words use character-level segmentation; The source sequence X after word segmentation is input into the encoder of the Transformer translation model, and the encoding is performed to obtain the basic source representation vector sequence E{e1, e2, ...e m }, i={1, 2, ...m}, where e i Represents the i-th subword in the source sentence; Assume that the source digital label sequence L src ={l1,l2,…l m }, i={1, 2, ...m}, where l i Indicates that the i-th subword is a number or a subword of a number. If the i-th subword is not a number or a subword of a number, then l i =0; The source digital label sequence L src By encoding the label word vector, we get the label word vector sequence E l ={e l1 , e l2 ,…e lm }, i={1, 2, ...m}, where e li The numeric label word vector representing the i-th subword; Connect the basic source representation vector and the label word vector element by element to obtain the connection vector H0={h 01 , h 02 ,…h 0m }, i={1, 2, ...m}, where h 0i =e i |e li , "|" indicates vector connection, e li Represents the connection vector corresponding to the i-th subword in the source language sentence; After encoding by the controller encoder, the controller source end representation H={h1, h2,…h m }, the controller encoder is constructed as follows: Among them, FFN represents a fully connected neural network, acti represents an arbitrary activation function; The specific steps of obtaining the numeric word to be translated based on the weight vector of the top cross-attention layer of the decoder of the neural machine translation model and translating it into the target numeric word include the following: According to the weight vector w of the cross attention layer j , the position with the largest weight is the subword to be translated in this step, and thus the position of the digital word to be translated is obtained as k=argmax(w j ); The digital translation module obtains the complete digital word containing the k-th subword of the source through the rules. If the k-th subword is not a digital word, let t j = 0, if the kth subword is a number word, then the complete number word is input into the number translation module and translated into the number word N of the target language in the manner of the rule j ={n j 1,n j 2,…n j pj }, where n i represents the i-th subword of the target language's number word; If before step j, there are already translated number words N l ={n l 1,n l 2,…n l pl }, and l+pl>j, then skip the step of translating the complete number word, otherwise let l:=j and N l :=N j .

2. The translation method based on the neural machine translation model according to claim 1, characterized in that: The specific steps of obtaining the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the neural machine translation model based on the target-side historical sequence and the basic source-side representation include the following: Input basic source representation vector sequence E and target historical sequence Y j-1 , the next step of decoding is performed through the decoder of the Transformer translation model to obtain the basic target end representation vector f of step j j And the weight vector W of the top cross-attention layer of the decoder of the Transformer translation model j ={w j1 , w j2 ,…w jm }, i={1, 2, ...m}, where w jI Indicates the weight corresponding to the i-th subword at the source end; Based on the generated historical target-side representation vector sequence F j ={f1, f2, ...f j } and the controller source representation H, which is decoded by the controller decoder to obtain the module weight t of step j j , the controller decoder adopts the same structure as the decoder of the Transformer translation model; During the training phase, a binary 0-1 label l is used t j Supervision module weight t j Learning, l t j =1 means that the jth target terminal word is the starting position of a number word. If the jth target terminal word is not the starting position of a number word, then l t j is 0.

3. The translation method based on the neural machine translation model according to claim 1, characterized in that: The specific steps of comprehensively calculating word probabilities based on the basic target-side representation and the translated target-side digital word include the following: Calculate the probability distribution of all words in step j, the formula is: Among them, v j is a random variable of the word generated by decoding in step j, and its value is any word of the target source; Obtained by the output layer of the Transformer translation model is a 0-1 function, currently only when When is 1; Finally, based on the probability distribution of all words in step j, the complete target language translation result Y={y1, y2, ...y n }, n represents the sequence length; If the translation result y in step j j From the digital translation module, after the translation results of the digital translation module are completely output, the translation results of the Transformer translation model are reused; according to the Bayesian posterior theorem, P(v j ) in the digital translation module and the Transformer translation model: , then it is considered that for the subword v j , the contribution of the digital translation module is greater, and then let t j =1 until the result of the digital translation module is output.

4. A translation device based on a neural machine translation model, characterized by: include The first acquisition module: Based on the source sequence, the encoder of the neural machine translation model is used to obtain the basic source representation and the controller source representation; The second acquisition module is configured to acquire the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the neural machine translation model based on the target-side historical sequence and the basic source-side representation; The third acquisition module: based on the weight vector of the cross-attention layer at the top of the decoder of the neural machine translation model, obtains the digital word to be translated and translates it into the target digital word; Calculation module: comprehensively calculates word probability based on the basic target-side representation and the translated target-side digital word; The translation is based on autoregressive decoding, generating the next word y at each step. j , are based on the source sequence and the decoded target historical sequence Y j-1 ={y0, y1, ...y j-1 }, the decoding process of each step of the autoregressive decoding is implemented by the above modules; The specific steps of obtaining the basic source representation and the controller source representation based on the source sequence through the encoder of the neural machine translation model include the following: Let the source sequence X = {x1, x2, ...x m }, i={1, 2, ...m}, where x i Represents the i-th subword in the source sentence, where the source sequence uses BPE segmentation and the numeric words use character-level segmentation; The source sequence X after word segmentation is input into the encoder of the Transformer translation model, and the encoding is performed to obtain the basic source representation vector sequence E{e1, e2, ...e m }, i={1, 2, ...m}, where e i Represents the i-th subword in the source sentence; Assume that the source digital label sequence L src ={l1,l2,…l m }, i={1, 2, ...m}, where l i Indicates that the i-th subword is a number or a subword of a number. If the i-th subword is not a number or a subword of a number, then l i =0; The source digital label sequence L src By encoding the label word vector, we get the label word vector sequence E l ={e l1 , e l2 ,…e lm }, i={1, 2, ...m}, where e li The numeric label word vector representing the i-th subword; Connect the basic source representation vector and the label word vector element by element to obtain the connection vector H0={h 01 , h 02 ,…h 0m }, i={1, 2, ...m}, where h 0i =e i |e li , "|" indicates vector connection, e li Represents the connection vector corresponding to the i-th subword in the source language sentence; After encoding by the controller encoder, the controller source end representation H={h1, h2,…h m }, the controller encoder is constructed as follows: Among them, FFN represents a fully connected neural network, acti represents an arbitrary activation function; The specific steps of obtaining the numeric word to be translated based on the weight vector of the top cross-attention layer of the decoder of the neural machine translation model and translating it into the target numeric word include the following: According to the weight vector w of the cross attention layer j , the position with the largest weight is the subword to be translated in this step, and thus the position of the digital word to be translated is obtained as k=argmax(w j ); The digital translation module obtains the complete digital word containing the k-th subword of the source through the rules. If the k-th subword is not a digital word, let t j = 0, if the kth subword is a number word, then the complete number word is input into the number translation module and translated into the number word N of the target language in the manner of the rule j ={n j 1,n j 2,…n j pj }, where n i represents the i-th subword of the target language's number word; If before step j, there are already translated number words N l ={n l 1,n l 2,…n l pl }, and l+pl>j, then skip the step of translating the complete number word, otherwise let l:=j and N l :=N j .

5. The translation device based on the neural machine translation model according to claim 4, characterized in that: The specific steps of obtaining the basic target-side representation, the module weights and weight vectors of the top cross-attention layer of the decoder of the neural machine translation model based on the target-side historical sequence and the basic source-side representation include the following: Input basic source representation vector sequence E and target historical sequence Y j-1 , the next step of decoding is performed through the decoder of the Transformer translation model to obtain the basic target end representation vector f of step j j And the weight vector W of the top cross-attention layer of the decoder of the Transformer translation model j ={w j1 , w j2 ,…w jm }, i={1, 2, ...m}, where w jI Indicates the weight corresponding to the i-th subword at the source end; Based on the generated historical target-side representation vector sequence F j ={f1, f2, ...f j } and the controller source representation H, which is decoded by the controller decoder to obtain the module weight t of step j j , the controller decoder adopts the same structure as the Transformer decoder; During the training phase, a binary 0-1 label l is used t j Supervision module weight t j Learning, l t j =1 means that the jth target terminal word is the starting position of a number word. If the jth target terminal word is not the starting position of a number word, then l t j is 0.

6. The translation device based on the neural machine translation model according to claim 5, characterized in that: The specific steps of comprehensively calculating word probabilities based on the basic target-side representation and the translated target-side digital word include the following: Calculate the probability distribution of all words in step j, the formula is: Among them, v j is a random variable of the word generated by decoding in step j, and its value is any word of the target source; Obtained by the output layer of the Transformer translation model is a 0-1 function, currently only when When is 1; Finally, based on the probability distribution of all words in step j, the complete target language translation result Y={y1, y2, ...y n }, n represents the sequence length; If the translation result y in step j j From the digital translation module, after the translation results of the digital translation module are completely output, the translation results of the Transformer translation model are reused; according to the Bayesian posterior theorem, P(v j ) in the digital translation module and the Transformer translation model: , then it is considered that for the subword v j , the contribution of the digital translation module is greater, and then let t j =1 until the result of the digital translation module is output.

Images