A machine translation method using part-of-speech information

Abstract

The present invention discloses a machine translation method using part-of-speech information, which is characterized by using a training set of bilingual corpora containing corresponding part-of-speech tags, and using part-of-speech information to assist machine translation after the translation model training is completed; the translation model includes a shared encoder, a decoder and a part-of-speech classifier, and the part-of-speech information is imported in different ways to enhance the model's language understanding and text generation capabilities; the shared encoder is composed of a source sentence encoder and an auxiliary sentence encoder; the decoder includes an original multi-head attention module and an auxiliary multi-head attention module. Compared with the prior art, the present invention has the function of improving the performance of the machine translation model by introducing part-of-speech information. The present invention uses the Transformer as the basic translation model, and imports part-of-speech information in different ways in the encoder and decoder components, so that the translation model can use the part-of-speech information to enhance the model's language understanding and text generation capabilities, thereby improving the quality of machine translation.

Description

Technical Field

[0001] The present invention relates to the technical field of machine translation, and in particular to a machine translation method for improving translation quality by using part-of-speech information. Background Art

[0002] At present, machine translation technology generally adopts typical machine translation, multi-source machine translation, Transformer model, part-of-speech tagging or BPE algorithm. The goal of typical machine translation is to translate sentences in the source language into sentences in the target language. The source language is marked with S, the target language is marked with T, and the i-th sentence of a language L (i.e. S or T) is marked with X. L (i) Indicates that the data set on this language consists of these sentences, expressed as N L is the number of these statements. Each statement consists of a series of ordered tokens, namely Each symbol comes from the vocabulary V of the language L L , that is, any symbol x i ∈V L The goal of the translation task is to make the input sentence X in the source language S The translation result and the target language output sentence X T The semantics are the same. Multi-source machine translation is a technology that uses knowledge information from additional sources to assist machine translation. Multi-source machine translation is an enhanced method of machine translation. Its goal is the same as that of typical machine translation, which is to translate sentences in the source language into sentences in the target language. However, other useful information from the source or target end (such as grammar and syntax information) is added during the training process of multi-source machine translation. The difference between multi-source machine translation and typical machine translation is whether the model utilizes multi-source information. Multi-source information can come from the nature of the sentence itself (grammar and syntax information, etc.) and non-text modal information (associated images, sounds, etc.).

[0003] The Transformer model is a model designed based on the Attention mechanism, which is suitable for processing sequence data such as text. The Transformer consists of multiple stacked Transformer blocks. Each block contains a multi-head Attention module and a fully connected layer, which are connected using a residual method. The multi-head Attention module in a Transformer block is based on the traditional Attention mechanism. The Attention mechanism acts on sequence data, and the input is N vectors. The matrix X = [x1 x2...x N]. In the Attention mechanism, they are also called value variables. In addition to value variables, the input of the Attention mechanism also has a series of key variables The matrix K = [k1 k2...k N ] and a query variable q (which can be a series of query variables when multiple calculations are performed). In the two, the key variable corresponds to the input variable one-to-one and is often a function of the value variable; while the query variable is the object of reference for this Attention calculation. In the Self-Attention scenario, the query variable can also be a function of the value variable.

[0004] Part-of-speech tagging is the task of determining the most appropriate part-of-speech tag for each word in a given sentence. The correctness of part-of-speech tagging will directly affect the subsequent syntactic analysis and semantic analysis, and is one of the basic topics of information processing. Commonly used part-of-speech tagging models include N-gram model, hidden Markov model, maximum entropy model, decision tree-based model, etc. Among them, hidden Markov model is one of the most widely used and effective models.

[0005] BPE Algorithm In the field of machine translation, text exists in the form of word strings. Words can be of different types, and a word can represent a phrase, a word, or even a character. The Byte Pair Encoding (BPE) algorithm is the current mainstream text preprocessing algorithm. The BPE algorithm divides the text into multiple subwords before model training, and re-merges these subwords after model prediction. At this time, the word tokens exist in the form of subwords. The advantage of the BPE algorithm is that it can control the dictionary size according to different requirements and make the text sequence length as short as possible, which alleviates the problem of many unregistered words and rare words that are easy to exist in the subword translation model in the form of words, and can improve the performance of the translation model.

[0006] The existing machine translation technology does not make sufficient use of part-of-speech information. Part-of-speech information is only used for the semantic understanding of the translation model, but not for improving the text generation ability of the translation model. Summary of the invention

[0007] The purpose of the present invention is to design a machine translation method using part-of-speech information in response to the deficiencies of the prior art. The method of adding a multi-head attention module to the Transformer model is adopted to integrate multi-source data into the calculation process of the Transformer, so that the translation model can use the part-of-speech information to enhance the model's language understanding and text generation capabilities, improve the performance of the machine translation model, use a training set of bilingual corpora containing corresponding part-of-speech tags, and use the part-of-speech information to enhance the model's language understanding and text generation capabilities. With the help of bilingual part-of-speech information, good translation performance is achieved, the training time is reduced, and the performance of the machine translation model is improved, and the translation quality is greatly improved.

[0008] The specific technical scheme for achieving the purpose of the present invention is: a machine translation method using part-of-speech information, characterized in that a translation model composed of a shared encoder, a decoder and a part-of-speech classifier is used to complete the machine translation of multi-source information, the shared encoder includes a source sentence encoder and an auxiliary sentence encoder, whose inputs are the source sentence and the auxiliary coding sequence, and the outputs are the corresponding coding matrices; the decoder includes an original multi-head attention module and an auxiliary multi-head attention module, whose inputs are the output matrices of the source sentence encoder and the auxiliary sentence encoder, respectively, and the outputs of the two modules are fused to generate a target sentence. At the same time, in the training stage of the translation model, the target sentence is transmitted to the part-of-speech classifier to generate a part-of-speech sequence; the machine translation of multi-source information includes the following specific steps:

[0009] Step 1: Download the English-German and English-French datasets from the official website of the wmt14 dataset (http: / / www.statmt.org / wmt14 / ), which contain the training and test sets of the corresponding language translations. After executing word segmentation and BPE algorithm on each sentence, the preprocessed text is obtained.

[0010] Step 2: Use the part-of-speech tagging tool to extract the part-of-speech tags of the source language and target language texts in the training set, as well as the part-of-speech tags of the source language in the test set; divide the words in each source sentence into content words and function words according to the part-of-speech of the words, and extract the content words to form an auxiliary coding sequence.

[0011] Step 3: The source sentence and the auxiliary encoding sequence are simultaneously passed into the encoder of the translation model. After encoding, they are passed into the original multi-head attention module and the auxiliary multi-head attention module of the translation model respectively.

[0012] Step 4: The part-of-speech tags of the target sentence are combined into an auxiliary decoding sequence and passed to the decoding end of the translation model; a multi-task training method is adopted, and the auxiliary decoding sequence is used as an auxiliary task in the training of the translation model.

[0013] Step 5: Based on the trained translation model, use part-of-speech information to assist machine translation.

[0014] The auxiliary multi-head attention module is based on the original multi-head attention module, and the original multi-head attention module conforms to the following formulas 1 to 5:

[0015] α i =softmax(s(k i ,q)) (1);

[0016]

[0017]

[0018] Multi-Head(Q,K,V)=concat(head1,...,head h )W o (4);

[0019] Original-Attention(H e , S d )=Multi-Head(H e , S d , S d ) (5);

[0020] The auxiliary multi-head attention module conforms to the following 6 formulas:

[0021] Auxiliary-Attention(H a , S d )=Multi-Head(H a , S d , S d ) (6)

[0022] Where: k i is a key variable in the Attention mechanism; q is the query variable in the Attention mechanism; s is the scoring function; α i is the Attention weight; K is the matrix of key variables; X is the matrix of value variables; a is the weighted average of value variables, which is also the output of the Attention mechanism; each head in the multi-head Attention module i In the example, the query variable can form a matrix Q; and the key variable, query variable, and value variable will undergo a linear transformation, and the corresponding linear transformation matrices are Each head headi of the multi-head Attention module is concatenated and then transformed by the linear transformation matrix W oAfter a linear transformation, the calculation result of the multi-head attention module is Multi-Head (Q, K, V); finally, the translation model includes the traditional original multi-head attention module Original-Attention (H e , S d ), which represents the encoding of the source language H e As the query variable, the decoded representation S of the target language d As key variables and value variables. In the present invention, a multi-head Attention module is designed, namely, the Auxiliary-Attention (H a , S d ), which encodes the auxiliary coding sequence to obtain the auxiliary coding representation H a As the query variable, we also take the decoded representation S of the target language d As key variables and value variables.

[0023] The multi-task training method is to output the translation as the main task, and to generate the part-of-speech tag of each word in the corresponding translation as an auxiliary task; the multi-task method is used, and the part-of-speech recognition task is used as an auxiliary task, combined with the machine translation task, and the corresponding part-of-speech tag is predicted by merging the generated words of the decoder into subwords, so as to enhance the ability of the translation model to generate correct multi-subword words, which is reflected in reducing the mistranslation rate at the word level. The main reason is that the "half right and half wrong" phenomenon in the translation is alleviated. The intuitive reason for its occurrence is that the BPE algorithm does not consider the relationship between different subwords in the same word, and the wrong subword combination will lead to wrong words. In order to illustrate the "half right and half wrong" phenomenon more specifically, the present invention uses a translation example to embody it. When translating the English sentence "We are afraid it will encourageintervention." into the standard German translation "Wir fürchten, das begünstigt Interventionen.", some words have deviations, and the translation model generates the translation "Wir befürchten, dass dadurch eineIntervention wird." For some mistranslated words, such as "fürchten" is mistranslated into "befürchten", "das" is mistranslated into "dass" and "Intervention" is mistranslated into "Intervention", these words have different lengths, but the phenomenon of such mistranslation has a common reason, that is, the translation model generates wrong subwords, causing the BPE algorithm to generate wrong words in combination. In the BPE algorithm, "fürchten" will be cut to generate several subwords "für@@", "ch@@" and "ten". Once an extra word "be@@" is generated in the decoder, it will lead to the generation of the wrong word "befürchten". In order to alleviate this problem, the detailed steps of the present invention to implement the auxiliary task include:

[0024] Step a1: Extract the part-of-speech of the decoding end sentences in the training set to form a decoding end part-of-speech sequence. The target sentence is composed of sub-word level tokens. The part-of-speech tag is the grammatical feature information at the word level, which can be used to associate all sub-words in a word. The decoding end part-of-speech sequence is composed of word-level tokens, so it needs to be associated with the corresponding sub-words of the target sentence;

[0025] Step a2: Input the decoded part-of-speech sequence into the decoder of the translation model and use it as the training label for the auxiliary task;

[0026] Step a3: After the translation model finishes generating all subwords in the same word, it is passed through the part-of-speech classifier designed by the present invention, the translation model output is compared with the training label, and the loss term is calculated using the cross entropy loss function;

[0027] Step a4: Perform gradient descent calculation according to the loss function.

[0028] Compared with the prior art, the present invention has the following beneficial effects and significant technical progress:

[0029] 1) Based on the Transformer model, the present invention adds a fusion multi-head attention module to enhance the translation model's understanding of the main information in the source sentence.

[0030] 2) Use a shared encoder to reduce the number of parameters of the translation model and improve translation performance.

[0031] 3) Use a multi-task approach to train the translation model, use part-of-speech tag generation as an auxiliary task, and train the translation model to enhance its text generation capabilities.

[0032] 4) It helps to alleviate the "half right, half wrong" problem existing in sub-word level machine translation models. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the present invention;

[0034] Figure 2 This is a schematic diagram of the translation model.

[0035] Figure 3 Schematic diagram of a multi-source Transformer model trained for multiple tasks. DETAILED DESCRIPTION

[0036] The present invention learns translation ability through a Transformer model that fuses multi-source information with the assistance of part-of-speech information. Transformer is a model designed for bilingual sequence data, which constitutes the basis of the translation model in the present invention. Multi-source data is not integrated into the calculation process of the original Transformer. Transformer is composed of multiple layers of Transformer blocks stacked together, each of which mainly includes a multi-head Attention module and a fully connected layer. In order to apply Transformer to multi-source scenarios, some changes need to be made to the Transformer model. In order to introduce multi-source data, that is, an auxiliary coding sequence composed of content words in the source sentence and a part-of-speech sequence of the target sentence, the present invention introduces a module for fusing multi-head attention at the decoding end, and processes the target end sentence in parallel with the original multi-head attention module. The difference is that the newly added multi-head attention module is used to match the attention value of the target sentence and the auxiliary coding sequence, rather than the source sentence. The design of the fusion multi-head attention module enables the Transformer to refer to the main information in the sentence during the calculation process, which is used to improve the model's semantic understanding of the source sentence. In addition, a multi-task method is introduced to convert the vector generated by the decoder into a part-of-speech tag as an auxiliary task to reduce the error rate of the translation.

[0037] The following is a specific embodiment of training a translation model. Among them, step one and step two are preprocessing of text data. The process of training a translation model is divided into two parts: in step three, the encoding end uses an auxiliary coding sequence to complete the ability to enhance semantic understanding; in step four, the decoding end uses the part-of-speech sequence of the target sentence to enhance the ability of text generation. In this embodiment, translation from English to German is implemented. The training process of the present invention is divided into two parts: the encoding end uses an auxiliary coding sequence to complete the ability to enhance semantic understanding, and the decoding end uses the part-of-speech sequence of the target sentence to enhance the ability of text generation.

[0038] The present invention is further described in detail below by taking the translation from English to German as an example. Among them, step one and step two are the preprocessing of text data; the translation model training process is divided into two parts: in step three, the encoding end uses the auxiliary encoding sequence to complete the ability to enhance semantic understanding; in step four, the decoding end uses the part-of-speech sequence of the target sentence to enhance the ability of text generation. The decoding end uses the part-of-speech sequence of the target sentence to enhance the ability of text generation.

[0039] Example 1

[0040] See attached Figure 1 , follow the steps below to perform machine translation of multi-source information:

[0041] Step 1: Prepare a bilingual corpus

[0042] The English-German and English-French datasets were downloaded from the official website of the wmt14 dataset (http: / / www.statmt.org / wmt14 / ), and preprocessed texts were obtained after executing word segmentation and BPE algorithm on each sentence. The datasets include training sets and test sets for corresponding language translations.

[0043] The text in a language is first segmented and divided into multiple tokens, and then preprocessed by BPE encoding. BPE encoding is a word segmentation method that divides words into subwords. After training on a single language corpus, it can find high-frequency subwords and use them as the smallest unit of text.

[0044] Step 2: Extract part-of-speech tags

[0045] Use the part-of-speech tagging tool spacy to annotate the part-of-speech tags of the bilingual corpus, extract the part-of-speech tags of the source language and target language texts in the training set, and form part-of-speech tag sequences in the two languages. Extract nouns, verbs, and adjectives from the source sentences as content words, and merge all content words into auxiliary coding sequences. There is no need to extract content words from the target sentences, and the model training needs to use its part-of-speech tag sequence.

[0046] Step 3: Build a translation model

[0047] See attached Figure 2, after processing the input data, a translation model is established. The present invention adopts a translation model composed of a shared encoder, a decoder and a part-of-speech classifier to complete the machine translation of multi-source information. The shared encoder includes a source sentence encoder and an auxiliary sentence encoder, whose inputs are the source sentence and the auxiliary coding sequence, and the outputs are the corresponding coding matrices; the decoder includes an original multi-head attention module and an auxiliary multi-head attention module, whose inputs are the output matrices of the source sentence encoder and the auxiliary sentence encoder, respectively, and the outputs of the two modules are fused to generate a target sentence. At the same time, in the training stage of the translation model, the target sentence is passed to the part-of-speech classifier to generate a part-of-speech sequence. In order to compress the scale of the translation model, the present invention uses a shared encoder to share the training parameters of the two encoders. The introduced part-of-speech classifier is a multi-layer fully connected network, and the number of its hidden states is 128, 32 and 11 respectively, where the number of hidden states of the last layer is equal to the number of part-of-speech tags, and the other parameters of the model are consistent with the original Transformer.

[0048] Step 4: First part of training the translation model

[0049] The source sentence and the auxiliary encoding sequence are simultaneously input into the encoder of the translation model, and after encoding, they are respectively transmitted to the original multi-head attention module and the auxiliary multi-head attention module of the translation model; the part-of-speech tags of the target sentence are composed into an auxiliary decoding sequence, which is input into the decoding end of the translation model as an auxiliary task to participate in the training of the translation model, and the training of the translation model adopts a multi-task training method.

[0050] The first part of the training is to use auxiliary coding sequences to enhance the semantic understanding ability of the model. A sentence in the source language is segmented into a token sequence X (i) ={x1, ..., x n (i) After extracting the content words, we get the auxiliary coding sequence A (i) ={a1, ..., a m (i)}. Among them, The auxiliary encoding sequence is processed in a similar way to the source sentence, e.g. Figure 2 As shown in the figure, the encoding end mainly includes two encoders, which share training parameters. First, the auxiliary encoding sequence is passed into a word vector layer that shares training parameters with the source sentence, and then the auxiliary encoding word vector is passed into the shared encoder, and its output vector is processed by the fused multi-head attention module. The fused multi-head attention module processes the input by changing the query variable from the representation of the source sentence to the representation of the auxiliary encoding sequence, that is, the output vector of the content word encoder, based on the original source sentence-target sentence attention module. This can make the model pay more attention to the words in the auxiliary encoding sequence, that is, all the content words in the corresponding source sentence, because the content words on the encoding end can help the model enhance its understanding of the main semantics of the sentence.

[0051] In this step, the first part of the translation model training is completed, that is, the auxiliary coding sequence is used to enhance the semantic understanding ability of the translation model. The source sentence and the auxiliary coding sequence are respectively passed into the source sentence encoder and the auxiliary sentence encoder of the translation model, and after encoding, they are respectively passed into the original multi-head attention module and the fused multi-head attention module of the translation model. Specifically, the way to process the auxiliary coding sequence is similar to that of the source sentence. The encoding end mainly includes two encoders, namely the source sentence encoder for processing the source sentence and the auxiliary sentence encoder for processing the auxiliary coding sequence. In order to compress the scale of the translation model, the two encoders share training parameters, so they are also called shared encoders.

[0052] The auxiliary coding sequence and the source sentence are passed into the translation model at the same time. Specifically, the auxiliary coding sequence is first passed into a word vector layer that shares training parameters with the source sentence, and then the auxiliary coding word vector is passed into the shared encoder, and its output vector is processed by the fused multi-head attention module. The source sentence is processed similarly, but its output vector is processed by the original multi-head attention module. The fused multi-head attention module processes the input by changing the query variable from the representation of the source sentence to the representation of the auxiliary coding sequence, that is, the output vector of the auxiliary sentence encoder, based on the original source sentence-target sentence attention module. This allows the translation model to pay more attention to the words in the auxiliary coding sequence, that is, all the content words corresponding to the source sentence, because the content words on the encoding end can assist the translation model in enhancing its understanding of the main semantics of the sentence.

[0053] Step 5: Second part of training the translation model

[0054] The second part of the training phase is carried out simultaneously with the first part. The goal of the second part is to use the target part-of-speech sequence to enhance the model's text generation ability. During the training of the decoder, the decoder output matrix H (i) ={h1, ..., h m(i)} Groups are divided according to the word segmentation of the BPE algorithm so that all vectors in the group come from the same word. Specifically, the sentences after word segmentation by the BPE algorithm are composed of several subwords, and each subword is separated by a space. Except for the last subword of the word (called the final subword), the non-final subwords all end with the "@@" symbol, so it is possible to determine which word each subword comes from. When performing part-of-speech tagging in the preprocessing process, the part-of-speech tags also need to be digitized. Although the part-of-speech is a word-level label, it can be extended to the subword level, and all subwords have the same part-of-speech label. However, this cannot distinguish the grouping situation of the BPE algorithm. Therefore, in the process of digitizing the part-of-speech label, the present invention keeps the part-of-speech label of the final subword unchanged, that is, a non-zero number, and the part-of-speech label of the non-final word is represented by a zero. This step is to ensure that the input and output of the part-of-speech classifier are one-to-one. This forms a part-of-speech label sequence T composed of integers. (i) ={t1, ..., t m (i)}, during the training process, this T (i) Will be used as labels for auxiliary tasks.

[0055] See attached Figure 3 , the part-of-speech recognizer and decoder are trained synchronously, in H (i) After all are generated, the part-of-speech recognizer will (i) As input, it will be based on T (i) To group, t j If it is non-zero, it means that the vectors that are consecutively zero before the current position can be averaged to generate a new pseudo-word vector. This vector will pass through the three-layer fully connected network to obtain a generated scalar. And with t j The loss function is calculated to obtain the loss function value of the auxiliary task. For words of different lengths, the probability of "half right and half wrong" phenomenon is also different. Long words with more subwords have a higher probability of generating wrong words. In order to make the classifier pay more attention to these long words, a length coefficient is designed to adjust the influence of words of different lengths. For words with more subwords, the length coefficient is larger. For the jth word, the calculation formula is α j =count j / 10. Among them, count j is the number of subwords of the jth word. In the present invention, the loss of the main task and the auxiliary task are multiplied by their respective coefficients and added together as the final loss function value, and the Adam optimizer is used to optimize the training parameters of the translation model.

[0056] Step 6: Use the translation model for translation

[0057] The sentence to be translated is encoded according to a specific BPE encoding method and then input into the translation model. The output of the decoder in the translation model is used to generate text. First, the output of the decoder is linearly mapped into the score of each word token, and then the text is generated according to a specific generation method. The techniques that can be used here include but are not limited to greedy, sampling, Beam Search, etc. This embodiment uses Beam Search, and the input of the decoder cannot use Teacher Forcing because there is only a reference input during training. Therefore, the decoder uses an autoregressive method to generate text, and it needs to loop multiple times to obtain a complete output sequence. Each decoder input is the sequence generated by the previous loop. The sequence output by the translation model is a sub-word level word token sequence, which needs to be restored to a sentence composed of words according to the restoration method specified by the BPE encoding.

[0058] The test process of this embodiment is implemented on the WMT14 English-German and WMT14 English-French data sets. The two data sets are similar in form and both contain corresponding bilingual corpora. After data processing, they can be used as data sets for zero-resource machine translation. After training and testing on the two data sets, the translation performance of this embodiment is obtained, and the result is 29.10 on the WMT14 English-German test set and 41.15 on the WMT14 English-French test set. The test results show that the translation of this translation model is better than other known machine translation models.

[0059] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A machine translation method using part-of-speech information, characterized in that A translation model consisting of a shared encoder, a decoder and a part-of-speech classifier is used to complete the machine translation of multi-source information. The shared encoder includes a source sentence encoder and an auxiliary sentence encoder, whose inputs are the source sentence and the auxiliary coding sequence, and whose outputs are the corresponding coding matrices; the decoder includes an original multi-head attention module and an auxiliary multi-head attention module, whose inputs are the output matrices of the source sentence encoder and the auxiliary sentence encoder, respectively, and the outputs of the two modules are fused to generate a target sentence. At the same time, in the training stage of the translation model, the target sentence is passed to the part-of-speech classifier to generate a part-of-speech sequence; the machine translation of multi-source information includes the following specific steps: Step 1: Download the English-German and English-French datasets from the official website of the wmt14 dataset, and perform word segmentation and BPE algorithm on each sentence to obtain the preprocessed text. The dataset contains training sets and test sets translated into corresponding languages. Step 2: Use the part-of-speech tagging tool to extract the part-of-speech tags of the source language and target language texts in the training set, as well as the part-of-speech tags of the source language in the test set, divide the words in each source sentence into content words and function words according to their parts of speech, and extract the content words to form an auxiliary coding sequence; Step 3: The source sentence and the auxiliary encoding sequence are simultaneously input into the encoder of the translation model, and after encoding, they are respectively transmitted to the original multi-head attention module and the auxiliary multi-head attention module of the translation model; Step 4: The part-of-speech tags of the target sentence are combined into an auxiliary decoding sequence, and input into the decoding end of the translation model as an auxiliary task to participate in the training of the translation model, and the training of the translation model adopts a multi-task training method; Step 5: After the translation model is trained, part-of-speech information is used to assist machine translation.

2. The machine translation method using part-of-speech information according to claim 1, characterized in that The original multi-head attention module conforms to the following equations (1) to (5): α i =softmax(s(k i ,q)) (1) head i =attention(W i Q Q,W i K K,W i V V) (3) Multi-Head(Q,K,V)=concat(head1,…,head h )W O (4) Original-Attention(H e ,S d )=Multi-Head(H e ,S d ,S d ) (5) The auxiliary multi-head attention module conforms to the following formula (6): Auxiliary-Attention(H a ,S d )=Multi-Head(H a ,S d ,S d ) (6) Where: k i is a key variable in the Attention mechanism; q is the query variable in the Attention mechanism; s is the scoring function; α i is the Attention weight; K is the matrix composed of key variables; X is the matrix composed of value variables; a is the weighted average of value variables, that is, the output of the Attention mechanism, head i represents the i-th head of the multi-head Attention module, W i Q represents the linear transformation matrix corresponding to the query variable, Q represents the matrix composed of query variables, and W i K Represents the linear transformation matrix corresponding to the key variable, W i V Represents the linear transformation matrix corresponding to the value variable, W O represents the linear transformation matrix, H e Represents the encoded representation of the source language, S d represents the decoded representation of the target language, H a Represents auxiliary encoding representation.

3. The machine translation method using part-of-speech information according to claim 1, characterized in that The multi-task training method is to output the translation of the original translation model as the main task, and to simultaneously generate the part-of-speech tag of each word in the corresponding translation as the auxiliary task. The multi-task trains the translation model under the bilingual corpus assisted by the part-of-speech information, so that the translation model can construct reasonable sentences and reduce the error rate at the word level. The implementation of the auxiliary task specifically includes the following steps: Step a1: extracting the parts of speech of the decoding end sentences in the training set to form a decoding end part of speech sequence; Step a2: Input the decoded part-of-speech sequence into the decoder of the translation model and use it as the training label for the auxiliary task; Step a3: The translation model generates all subwords in the same word, passes them through the part-of-speech classifier, compares the translation model output with the training label, and uses the cross entropy loss function to calculate the loss term; Step a4: Perform gradient descent calculation according to the loss function.

Images