A paragraph-level neural machine translation method and system based on masking mechanism

Abstract

The present invention discloses a method and system for passage-level neural machine translation based on a masking mechanism, which belongs to the technical field of natural language processing and language translation. This method uses a masking mechanism to mask part of the information in the sentence to be translated, so that the model is more dependent on contextual information when translating sentences. This method mainly makes improvements to the training strategy and adopts a two-stage training process to better enable the encoder to restore the encoding information, effectively improving the effect of passage-level neural machine translation. The system part includes a data processing module, a source language encoder module, a source language self-encoding decoder module, a target language decoder module and a generator module. Compared with the existing technology, this method adopts a masking and self-encoding strategy in the training strategy, trains a passage-level translation model that is more dependent on context when encoding, makes full use of contextual information, and further improves the effect and quality of passage-level machine translation.

Description

Technical Field

[0001] The present invention relates to a neural machine translation method and system, and in particular to a paragraph-level neural machine translation method and system based on a masking mechanism, belonging to the technical field of natural language processing and language translation. Background Art

[0002] Neural Machine Translation (NMT) is a method that uses deep learning neural networks to obtain mappings between natural languages ​​and is a key task in natural language processing. NMT uses neural network-based technology to achieve more contextually accurate translations, rather than translating fragmented sentences one word at a time. NMT learns and collects information, analyzing it in two phases: encoding and decoding. In the encoding phase, text information from a natural language is input into the machine and then classified into language vectors. Similar words in context are placed in comparable word vectors. In the decoding phase, the vectors are efficiently and seamlessly transmitted to the target language.

[0003] Paragraph-level machine translation, which aims to use computers to automatically translate a paragraph from one language into another, is a very challenging task in machine translation. In recent years, with the rapid development of neural machine translation, paragraph-level neural machine translation has become a hot topic in the field of machine translation. This technology enables the translation to maintain consistency in translation style and accuracy throughout the entire document or in a specific semantic context. With the rapid development of deep learning, end-to-end model architectures can help neural machine translation directly convert source language sentences into target language sentences. Compared to traditional sentence-level neural machine translation, paragraph-level neural machine translation produces higher-quality translations by modeling and utilizing paragraph-level contextual information.

[0004] In the prior art, a multi-encoder approach is usually adopted to solve the problem of paragraph-level translation in neural machine translation. An additional encoder is used to encode the context, and after obtaining the encoded information, it is fused with the current sentence to be translated using an attention mechanism or a gating mechanism. A single encoder is relatively simple. The context and the sentence to be translated are spliced ​​together and sent to the encoder together. The attention mechanism is used to obtain a better encoded representation, which is then sent to the decoder to generate the corresponding translation. However, in the structure of a single encoder, it is necessary to make the sentence to be translated more dependent on the contextual information. For example, tense information, gender information, and even the specific meaning of abbreviations can be obtained from the context to be processed. Since the neural network itself is often regarded as a black box model, it is often difficult to define the degree of dependence on the context during the encoding and translation process. In addition, during the encoding process, the encoding of the source language sentence may not be fully conveyed due to the introduction of additional information.

[0005] In summary, there is an urgent need for a technology that can enable paragraph-level neural machine translation methods to better rely on context during encoding or translation, while ensuring that the original semantic information is not lost. Summary of the Invention

[0006] The purpose of the present invention is to address the problems and shortcomings of the existing technology, and to solve technical problems such as enabling the sentences to be translated to better obtain information from the context and ensure that the original semantic information is not lost during the paragraph-level neural machine translation process. A paragraph-level neural network machine translation method and system based on a masking mechanism is creatively developed.

[0007] The innovation of the present invention is that: first, the context and the randomly masked sentences to be translated are spliced ​​together as the input of the encoder. In common single encoder structures, the spliced ​​sentences are often used directly as the input of the encoder, but this method makes the context itself become a kind of noise, destroying the semantic information of the sentence to be translated, and the encoding information will also be affected, which will reduce the translation effect. In addition, it is necessary to distinguish the context from the sentence to be translated so that the encoder can better help the sentence to be translated obtain information from the context. In response to this situation, the present invention introduces a masked language model task on the one hand, using a mask to randomly replace some words in the sentence to be translated; in addition, it introduces a denoising autoencoder to add constraints to the encoder part, so that the context-based encoding information can not only make full use of the context information, but also better restore the semantic information of the sentence to be translated, and finally generate the target language sentence.

[0008] The present invention is implemented by adopting the following technical solutions.

[0009] A paragraph-level neural machine translation method based on a mask mechanism includes the following steps:

[0010] Step 1: Input the source language sentence and encode it to generate the context-sensitive encoding information Enc(X) through random masking.

[0011] Specifically, step 1 includes the following steps:

[0012] Step 1.1: Sentence X to be translated curr Each word in is replaced by the mask symbol with probability p.

[0013] Preferably, p=0.3, obeying Bernoulli distribution.

[0014] Step 1.2: Select the previous text X of the sentence to be translated prev As the context, the masked sentence to be translated is concatenated with the context to obtain the input sequence X input .

[0015] Step 1.3: Input sequence X input After passing through the embedding layer of the encoder, it is mapped into word vectors word by word, and the embedding layer outputs a sequence Emb(X) consisting of word vectors.

[0016] Step 1.4: Send the sequence Emb(X) to the encoder for encoding to obtain the encoded sequence Enc(X) with context.

[0017] Specifically, the sequence Emb(X) is encoded through the encoder’s multi-head self-attention layer and linear layer, and the process is repeated N times to finally obtain Enc(X).

[0018] Step 2: The encoded sequence Enc(X) obtained by the encoder is sent to the denoising autoencoder decoder with the same structure as the encoder to obtain the denoising autoencoder output sequence Dec Enc (Enc(X)).

[0019] Specifically, this step is the same as the processing in step 1.4, and its purpose is to add a masked language model task to better restore the sentences to be translated whose semantic information may have been destroyed by masking.

[0020] Step 3: Use the encoder output and decoder input to output the decoded sequence Dec(Y).

[0021] Specifically, step 3 includes the following steps:

[0022] Step 3.1: According to the previous article X prev The length of the sentence to be translated X curr The length of the context encoding sequence Enc(X) is cut into pieces, and the context encoding sequence Enc(X) is obtained. prev and the encoding sequence Enc(X) of the sentence to be translated curr .

[0023] Step 3.2: Select the sentence X to be translated in step 1.1 curr The corresponding target language sentence Y tok , sent to the embedding layer of the encoder, mapped word by word into word vectors, and the embedding layer outputs the target language sequence Emb(Y) composed of word vectors.

[0024] Step 3.3: Use the decoder to decode the target language sequence Emb(Y) to obtain a decoded sequence Dec(Y).

[0025] Specifically, the target language sequence representation Emb(Y) passes through the self-attention layer of the decoder, using the multi-head attention layer and the context encoding sequence Enc(X) prev And the sentence to be translated X currThe coding sequence Enc(X) curr The fusion is then passed through the linear layer of the decoder. This process is repeated N times to finally obtain the decoded sequence Dec(Y).

[0026] Step 4: The denoising autoencoder output sequence Dec in step 2 is converted to Enc (Enc(X)) is compared with the concatenated sequence Emb(X) in step 1.3, and the decoded sequence Dec(Y) is compared with the target language reference encoding sequence Ref(Y). The weighted differences are added together as the model loss L1.

[0027] Step 5: Use the gradient descent method according to the model loss and repeat steps 1 to 4 until the model converges to obtain the translation model M.

[0028] Step 6: Based on the translation model M obtained in step 5, continue to use the existing data and train it without masking.

[0029] Specifically, step 6 includes the following steps:

[0030] Step 6.1: Select the sentence to be translated X curr Previous article X prev As the context, and the sentence to be translated X curr Splice with the context to get the input sequence X input ′.

[0031] Step 6.2: Input sequence X input ′ passes through the embedding layer of the encoder and is mapped into word vectors word by word. The embedding layer outputs a sequence Emb(X′) consisting of word vectors.

[0032] Step 6.3: Send the sequence Emb(X′) to the encoder and obtain the encoded sequence Enc(X′) with context after encoding.

[0033] Specifically, the sequence Emb(X′) is encoded through the multi-head self-attention layer and linear layer of the encoder. This process is repeated N times and finally Enc(X′) is obtained.

[0034] Step 7: The process is the same as step 3. The encoder outputs Enc(X′) from step 6.3 and outputs the decoded sequence Dec(Y′).

[0035] Step 8: Compare the decoded sequence Dec(Y′) in step 7 with the target language reference encoding sequence Ref(Y), and use the difference as the model loss L2.

[0036] Step 9: Use the gradient descent method according to the model loss L2, repeat steps 6 to 8 until the model converges to obtain the translation model M′.

[0037] Step 10: Substitute the source language sentences X from the non-training set t 、Previous text X of the source language sentence t-1 , sent to the translation model M′ to obtain the final translation.

[0038] On the other hand, based on the above method, the present invention further proposes a paragraph-level neural machine translation system based on a mask mechanism, which includes a data processing module, a source language encoder module, a source language denoising autoencoder-decoder module, a target language decoder module and a generator module.

[0039] The data processing module is responsible for extracting the information data that needs to be translated and converting it into the corresponding word vector sequence;

[0040] The source language encoder module is used to fuse the paragraph-level information with the encoded information of the source language sentence, and feed them into the source language denoising self-encoder decoder module and the target language decoder module respectively;

[0041] During the first training, the source language denoising autoencoder module performs autoencoding on the output of the source language encoder module.

[0042] The target language decoder module decodes the obtained encoding information to generate decoding information;

[0043] The generator module is responsible for mapping the decoded information into the target vocabulary to generate the translation and present the result to the user.

[0044] Furthermore, the data processing module includes a sample acquisition unit, a pre-processing unit and an embedding layer unit;

[0045] The sample acquisition unit is used to acquire language pair samples from the corpus;

[0046] The preprocessing unit is used to obtain the required data from the corpus (including source language sentences, target language sentences and their corresponding source language context information), and is responsible for constructing the vocabulary and sentence segmentation;

[0047] The embedding layer unit is used to convert the processed text information into a corresponding word vector sequence.

[0048] The source language encoder module, the source language denoising autoencoder / decoder module, and the target language decoder module each include an attention unit and a linear unit. Each attention unit performs the same function: fusing information from the input word vectors or semantic vectors to generate corresponding semantic information. Each linear unit also performs the same function: performing a linear mapping on the input encoded information.

[0049] Among them, the source language encoder module includes a first attention unit and a first linear unit, the source language denoising self-encoder decoder module includes a second attention unit and a second linear unit, and the decoder module includes a third attention unit and a third linear unit.

[0050] The connection relationship of the above modules is:

[0051] The input end of the source language encoder module is connected to the output end of the data processing module;

[0052] The input end of the source language denoising self-encoder / decoder module is connected to the output end of the source language encoder module;

[0053] The input end of the target language decoder module is connected to the output end of the data processing module;

[0054] The input of the target language decoder module is connected to the output of the source language encoder module;

[0055] The input of the generator module is connected to the output of the target language decoder module.

[0056] The connection relationship between the units in the above modules is:

[0057] In the data processing module, the input end of the preprocessing unit is connected to the output end of the sample acquisition unit, and the input end of the embedding layer unit is connected to the output end of the preprocessing unit.

[0058] In the source language encoder module, the source language denoising self-encoder-decoder module and the target language decoder module, the input end of each linear unit is connected to the output end of the attention unit of the module.

[0059] Beneficial effects

[0060] Compared with the existing technology, the present invention has the following advantages:

[0061] The present invention introduces a masked language model task by randomly masking the current sentence to be translated, and at the same time introduces a denoising autoencoder during training, so that the "destroyed" input can obtain information from the context, restore the input information to the greatest extent, and make the sentence to be translated rely more on contextual information, thereby improving the effect and quality of paragraph-level neural machine translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of the method according to the present invention;

[0063] Figure 2 FIG. 1 is a schematic diagram of the architecture of the system according to the present invention. DETAILED DESCRIPTION

[0064] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0065] like Figure 1 As shown in FIG, a paragraph-level neural machine translation method based on a mask mechanism includes the following steps:

[0066] Step 1: Input the source language sentence and generate the context-sensitive encoding information Enc(X) by random masking. Specifically:

[0067] Step 1.1: Sentences to be translated Each word in is replaced by a mask symbol with probability p, following a Bernoulli distribution with p = 0.15.

[0068] Step 1.2: Select the preceding text of the sentence to be translated As the context, the masked sentence to be translated is concatenated with the context to obtain the input sequence

[0069] Step 1.3: Input Sequence After passing through the embedding layer, each word will be mapped to a word vector, and the embedding layer will output a sequence Emb(X) consisting of word vectors.

[0070] Step 1.4: The sequence Emb(X) is fed into the source language encoder and encoded to obtain the context-encoded sequence Enc(X).

[0071] Step 2: The encoded sequence Enc(X) obtained by the source language encoder will be sent to a module with the same structure as the source language encoder to obtain the denoised autoencoder output sequence Dec Enc (Enc(X)).

[0072] Step 3: Use the source language encoder output and the target language decoder input to output the decoded sequence Dec(Y). Specifically:

[0073] Step 3.1: Based on the length of the previous text and the length of the sentence to be translated, split the context-encoded representation Enc(X) in step 1.4 to obtain the context-encoded sequence Enc(X) prev and the encoding sequence Enc(X) of the sentence to be translated curr .

[0074] Step 3.2: Select the target language sentence Y corresponding to the sentence to be translated in step 1.1 tok , sent to the embedding layer, mapped word by word into word vectors, and the embedding layer outputs the target language sequence Emb(Y) composed of word vectors.

[0075] Step 3.3: Use the decoder to decode the target language sequence Emb(Y) to obtain a decoded sequence Dec(Y).

[0076] Step 4: The denoising autoencoder output sequence Dec in step 2 is converted to Enc (Enc(X)) is compared with the concatenated sequence Emb(X) in step 2.3, and the decoded sequence Dec(Y) is compared with the target language reference encoding sequence Ref(Y), and the weighted addition of the differences is used as the model loss L1.

[0077] Step 5: Repeat steps 1-4 using the gradient descent method according to the model loss until the model converges to obtain the translation model M.

[0078] In steps 1.1, 1.2, and 3.1, bilingual parallel sentence pairs with context are obtained and the samples are preprocessed.

[0079] In this example, IWSLT15 data is used, with Chinese as the source language and English as the target language. 200,000 pairs of parallel sentences containing context are extracted from IWSLT15 as training corpus. Each sample E consists of a preceding sentence, a source language sentence to be translated, and a target language sentence, for example:

[0080] Previous sentence: Confucius' family was very poor, but he studied hard and diligently since he was young.

[0081] Source language sentence to be translated: When he was in his twenties, he became a minor official.

[0082] Target language sentence: He became a petty official in his early twenties.

[0083] According to the masking mechanism, the words in the source language sentence to be translated will be randomly replaced using a mask, for example:

[0084] The source language sentence to be translated after masking: When he was in his twenties, [MASK] became a minor official.

[0085] In this example, we first segment the source and target language corpora and construct a vocabulary. For Chinese, we use the Chinese word segmentation tool Jieba (https: / / github.com / fxsjy / jieba) for word segmentation. We then use BPE to perform sub-word encoding on the segmented parallel corpora, generating a sub-word vocabulary and sub-word encodings. Based on the sub-word encodings, we segment the parallel corpora and tokenize them using the sub-word vocabulary.

[0086] In step 1.3, for each sample E, the tokenized source language sentence is mapped through the embedding layer Converted into a sequence of word vectors Emb(X).

[0087] In step 1.4, the sequence Emb(X) consisting of word vectors is fed into the source language encoder for encoding to obtain the encoded sequence Enc(X) with context.

[0088] Specifically, the process of the multi-head self-attention layer in step 1.4 is as follows:

[0089] First, each word vector in the sequence Emb(X) of word vectors is mapped into query value Q, key value K and value V through a matrix:

[0090] Q=W Q ·e (1)

[0091] K=W K ·e (2)

[0092] V=W V ·e (3)

[0093] Among them, W Q 、W K 、W V They represent the mapping parameters used to map query value Q, key value K, and value V respectively; e is the word vector representation of each word.

[0094] In the self-attention mechanism, for each word, the attention score s of the word to all other words is calculated:

[0095]

[0096] Among them, d k is the dimension of K, K T represents the transpose of K, and T represents the matrix transpose.

[0097] Then, use the attention score s as the weight, multiply the corresponding attention score with V and sum them to get the corresponding hidden layer state representation vector

[0098]

[0099] Since the Transformer encoder uses a multi-head attention mechanism, the representations generated by each attention head need to be concatenated here:

[0100]

[0101] Among them, h′ represents the result of splicing the representations produced by different attention heads, Indicates the representation produced by the nth head.

[0102] Finally, we use two fully connected network layers with ReLU as activation function to improve the nonlinear fitting ability of the model:

[0103] Enc layer (X)=FFN(h′)=max(0,max(0,h′W1+b1)W2+b2) (7)

[0104] Among them, Enc layer (X) represents the output of each encoder layer, and the output of the last encoder layer is denoted as Enc(X). FFN(·) represents a linear unit; W1 and W2 represent trainable weights, and b1 and b2 represent the bias terms corresponding to W1 and W2, respectively.

[0105] In step 1.4, after the sequence Emb(X) passes through the self-attention layer, the result of the self-attention layer is represented as SelfAttn(X), and SelfAttn(X) will be used as the input of the fully connected network and output Enc layer (X), and finally get the output of the encoder, which contains the context information Enc(X).

[0106] The calculation process of using the denoising self-encoder decoder to perform denoising self-encoder decoding on the encoder output in step 2 is the same as the process of calculating the encoder result Enc(X) in step 1.4, and Dec is obtained. Enc (Enc(X)).

[0107] In step 3, if Figure 1 As shown, the step of outputting a decoding sequence Dec(Y) using the source language encoder output and the target language decoder input includes:

[0108] In step 3.1, it is necessary to split the context-encoded representation Enc(X) in step 1.4 according to the length of the previous text and the length of the sentence to be translated, and obtain the context-encoded sequence Enc(X) prev and the encoding sequence Enc(X) of the sentence to be translated curr .

[0109] In step 3.2, the process is the same as step 1.3, and the input is the target language sentence Y tok , the output is the target language sequence Emb(Y).

[0110] In step 3.3, the self-attention layer is first used to make Emb(Y) obtain a better information representation. This process is the same as the process of calculating the self-attention result in step 1.4. It will then pass through two layers of multi-head attention layers. The first layer uses Enc(X) currAs the key value K and value V, the output result of the self-attention layer SelfAttn(Y) is used as the query value Q to generate the result DecEncAttn(Y); using the output result of the previous multi-head attention layer as the query value Q, Enc(X) prev As the key value K and the value V, the result DecCtxAttn(Y) is generated. The calculation process of each fully connected layer in the decoder is consistent with the calculation process of the fully connected layer of the encoder in step 1.4, and finally the output result Dec(Y) of the decoder is obtained.

[0111] In step 4, the denoising autoencoder output sequence Dec from step 2 is used Enc The cross entropy loss is calculated between (Enc(X)) and the concatenated sequence Emb(X) in step 2.3, and the cross entropy loss is calculated between the decoded sequence Dec(Y) and the target language reference encoding sequence Ref(Y). The two are weightedly added together to form the model loss L1:

[0112] L1=α·Loss(Emb(X),Dec Enc (Enc(X)))+(1-α)Loss(Ref(Y),Dec(Y)) (8)

[0113] Among them, α is a hyperparameter used to control the loss of the two parts for the overall training of the model, and Loss represents the cross entropy loss function.

[0114] Repeat steps 1-4 using gradient descent based on the model loss until the model converges, resulting in the translation model M. Since the training process involves random replacement of words in the source language sentence being translated, this differs from the process of using the model to generate translations. Therefore, a second phase of training is proposed, using the existing data without masking to continue training the resulting translation model M. The model's loss function L2 is as follows:

[0115] L2=Loss(Ref(Y),Dec(Y′)) (9)

[0116] The generation process of Dec(Y′) is consistent with that in the first stage, and the mask replacement process is no longer performed on the source language sentence to be translated.

[0117] The last vector of the target language decoding sequence is input into the generator, mapped to the vocabulary through the Softmax function, and the prediction for the next word is obtained. The above steps are repeated until the decoding generates a sentence end marker or the sentence length reaches the maximum length limit, at which point the generation is terminated.

[0118] According to another aspect of the present invention, a paragraph-level neural machine translation system based on a masking mechanism is proposed. Figure 2As shown, the system includes a data processing module, a source language encoder module, a source language denoising autoencoder / decoder module, a target language decoder module, and a generator module. The data processing module is responsible for extracting the data to be translated and converting it into a corresponding word vector sequence. The source language encoder module is used to fuse the encoded information of the source language sentence and the corresponding context information, and feed it into the source language denoising autoencoder / decoder module and the target language decoder. The source language denoising autoencoder / decoder module decodes the obtained encoded information to restore the source language information. The target language decoder decodes the obtained encoded information to generate decoded information. The generator module is responsible for mapping the decoded information to the target end's vocabulary, generating a translation, and presenting the result to the user.

[0119] Furthermore, the data processing module includes: a sample acquisition unit, a preprocessing unit, and an embedding layer unit, wherein:

[0120] A sample acquisition unit, used to acquire language pair samples from the corpus;

[0121] The preprocessing unit is used to obtain the required data from the corpus, including source language sentences, target language sentences and their corresponding source language context information, and is responsible for building a vocabulary and sentence segmentation;

[0122] The embedding layer unit is used to convert the processed text information into a corresponding word vector sequence.

[0123] The source language encoder module, the source language denoising autoencoder / decoder module, and the target language decoder module each include an attention unit and a linear unit. Each attention unit functions identically, fusing information from the input word vectors or semantic vectors to generate corresponding semantic information. Each linear unit also functions identically, performing a linear mapping of the input encoded information. The source language encoder module includes a first attention unit and a first linear unit, the source language denoising autoencoder / decoder module includes a second attention unit and a second lookahead unit, and the decoder module includes a third attention unit and a third linear unit.

[0124] In the sample acquisition unit, Chinese and English of IWSLT 15 can be selected as the source language and target language respectively.

[0125] In the preprocessing unit, the source language corpus and the language corpus are segmented and a vocabulary is constructed. Then, BPE is used to encode the vocabulary, and the sentences are split based on the vocabulary obtained by BPE encoding. The specific method is described above.

[0126] In the embedding layer, a sentence can be converted into a sequence of word vector representations through mapping. The specific method is as described above.

[0127] In each attention unit, the self-attention mechanism is used to calculate the correlation between words in the same sentence, and the multi-head attention mechanism is used to integrate context information. The specific methods are described above.

[0128] In each linear unit, the sequence information is converted into corresponding encoding and decoding sequences.

[0129] Those skilled in the art will understand that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art will understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A paragraph-level neural machine translation method based on a masking mechanism, characterized by: The following steps are involved: Step 1: Input the source language sentence and encode it to generate the context-sensitive encoding information Enc(X) by random masking. Step 1.1: Sentence X to be translated curr Each word in is replaced by a mask symbol with probability p; Step 1.2: Select the previous text X of the sentence to be translated prev As the context, the masked sentence to be translated is concatenated with the context to obtain the input sequence X input ; Step 1.3: Input sequence X input After passing through the embedding layer of the encoder, it is mapped into word vectors word by word, and the embedding layer outputs a sequence Emb(X) consisting of word vectors; Step 1.4: Send the sequence Emb(X) to the encoder for encoding to obtain the encoded sequence Enc(X) with context; The sequence Emb(X) is encoded by the encoder’s multi-head self-attention layer and linear layer, and the process is repeated N times to obtain Enc(X). Step 2: The encoded sequence Enc(X) obtained by the encoder is sent to the denoising autoencoder decoder with the same structure as the encoder to obtain the denoising autoencoder output sequence Dec Enc (Enc(X)); Step 3: Use the encoder output and decoder input to output the decoded sequence Dec(Y); Step 3.1: According to the previous article X prev The length of the sentence to be translated X curr The length of the context encoding sequence Enc(X) is cut into pieces, and the context encoding sequence Enc(X) is obtained. prev and the encoding sequence Enc(X) of the sentence to be translated curr ; Step 3.2: Select the sentence X to be translated in step 1.1 curr The corresponding target language sentence Y tok , sent to the embedding layer of the encoder, mapped word by word into word vectors, and the embedding layer outputs the target language sequence Emb(Y) composed of word vectors; Step 3.3: Use the decoder to decode the target language sequence Emb(Y) to obtain a decoded sequence Dec(Y); The target language sequence representation Emb(Y) passes through the self-attention layer of the decoder, using the multi-head attention layer and the context encoding sequence Enc(X) prev And the sentence to be translated X curr The coding sequence Enc(X) curr The fusion is performed and then passes through the linear layer of the decoder; this process is repeated N times to finally obtain the decoded sequence Dec(Y); Step 4: The denoising autoencoder output sequence Dec in step 2 is converted to Enc Compare (Enc(X)) with the concatenated sequence Emb(X) from step 1.3, compare the decoded sequence Dec(Y) with the target language reference encoding sequence Ref(Y), and add the weighted differences as the model loss L1; Step 5: Use the gradient descent method according to the model loss and repeat steps 1 to 4 until the model converges to obtain the translation model M; Step 6: Based on the translation model M obtained in step 5, continue to use the existing data and train it without masking; Step 6.1: Select the sentence to be translated X curr Previous article X prev As the context, and the sentence to be translated X curr Splice with the context to get the input sequence X input '; Step 6.2: Input sequence X input 'After passing through the embedding layer of the encoder, it is mapped into word vectors word by word, and the embedding layer outputs a sequence Emb(X') consisting of word vectors; Step 6.3: Send the sequence Emb(X') to the encoder and obtain the encoded sequence Enc(X') with context after encoding; The sequence Emb(X') is encoded by the encoder's multi-head self-attention layer and linear layer. This process is repeated N times, and finally Enc(X') is obtained. Step 7: Same as step 3, the encoder outputs Enc(X') from step 6.3 and outputs the decoded sequence Dec(Y'); Step 8: Compare the decoded sequence Dec(Y') from step 7 with the target language reference encoding sequence Ref(Y), and use the difference as the model loss L2; Step 9: Use the gradient descent method based on the model loss L2 and repeat steps 6 to 8 until the model converges to obtain the translation model M'; Step 10: Substitute the source language sentences X from the non-training set t 、Previous text X of the source language sentence t-1 , sent to the translation model M' to obtain the final translation.

2. A paragraph-level neural machine translation method based on a masking mechanism as claimed in claim 1, characterized in that: In step 1.4, the process of the multi-head self-attention layer is as follows: First, each word vector in the sequence Emb(X) of word vectors is mapped into query value Q, key value K and value V through a matrix: Q=W Q ·e (1) K=W K e (2) V / W V and (3) Among them, W Q 、W K 、W V They represent the mapping parameters used to map query value Q, key value K, and value V respectively; e is the word vector representation of each word; In the self-attention mechanism, for each word, the attention score s of the word to all other words is calculated: Among them, d k is the dimension of K, K T represents the transpose of K, and T represents the matrix transpose; Then, use the attention score s as the weight, multiply the corresponding attention score with V and sum them to get the corresponding hidden layer state representation vector Concatenate the representations produced by each attention head: Among them, h' represents the result of splicing the representations produced by different attention heads, represents the representation produced by the nth head; Finally, we use two fully connected network layers with ReLU as activation function to improve the nonlinear fitting ability of the model: Enc layer (X)=FFN(h')=max(0,max(0,h'W1+b1)W2+b2) (7) Among them, En clayer (X) represents the output of each layer of encoder, and the output of the last layer of encoder is expressed as Enc(X); FFN(·) represents a linear unit; W1 and W2 represent trainable weights, and b1 and b2 represent the bias terms corresponding to W1 and W2 respectively; In step 1.4, after the sequence Emb(X) passes through the self-attention layer, the result of the self-attention layer is represented as SelfAttn(X), and SelfAttn(X) will be used as the input of the fully connected network and output Enc layer (X), and finally get the output of the encoder, which contains the context information Enc(X).

3. The method for paragraph-level neural machine translation based on a masking mechanism according to claim 1, characterized in that: The model loss L1 is as follows: L1=α·Loss(Emb(X),Dec Enc (Enc(X)))+(1-α)Loss(Ref(Y),Dec(Y)) (8) Among them, α is a hyperparameter used to control the loss of the two parts for the overall training of the model, and Loss represents the cross entropy loss function; The model loss L2 is as follows: L2=Loss(Ref(Y),Dec(Y')) (9).

Images