A latent variable based document-level neural machine translation method and system

Abstract

The application relates to a kind of based on latent variable's paragraph-level neural machine translation method, belong to natural language processing application technical field.First in the encoder-decoder neural machine translation model, respectively, the source language sentence and its last sentence are encoded, and the latent variable is generated by the attention mechanism fusion coding information.The distribution of latent variable is considered as a kind of Gaussian distribution, which contains the context information required by paragraph-level translation.The latent variable is mapped to the shallow paragraph relationship distribution by the conversion matrix through the discriminator pre-trained on the paragraph relationship analysis task.Finally, the discriminator is applied to the training process of paragraph-level neural machine translation model, the generation of latent variable is guided, and the sentence of target language is finally generated.The application utilizes latent variable to represent the information required in paragraph-level neural machine translation, and guides the generation process of latent variable by introducing the discriminator, so as to improve the effect and quality of paragraph-level neural machine translation.

Description

Technical Field

[0001] This invention relates to a text-level neural machine translation method based on latent variables, belonging to the field of natural language processing application technology. Background Technology

[0002] Document-level machine translation is an important branch of neural machine translation and has long attracted much attention. Modern machine translation methods are mostly based on encoder-decoder models, using end-to-end learning methods to directly input source language sentences and corresponding target language sentences into the model, which can then learn the correspondence on its own.

[0003] Most existing machine translation methods translate individual sentences, often neglecting contextual information. Document-level machine translation, on the other hand, focuses on expanding the translation scope to the entire document, fully utilizing contextual information to improve the overall coherence and flow of the translation.

[0004] Most existing text-level machine translation methods encode the context of the source language and then feed it into the encoder or decoder through different mechanisms. These methods provide richer contextual information for translating the current sentence, that is, using a wider range of context to semantically encode (or decode) the current word (position). This approach can directly and effectively disambiguate word meanings. However, because this semantic matching lacks explicit or implicit guidance, it is difficult to better help the learning model discover other textual elements hidden beneath the surface word combinations; it is merely a text-level translation that expands the range of available information. In fact, for text-level machine translation, it is more necessary to actively guide the model in semantic modeling and purposefully learn such textual connections, rather than simply relying on general, poorly interpretable semantic encoding and mapping. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing machine translation methods, such as ignoring contextual information and lacking explicit or implicit guidance in semantic matching, which makes it difficult for learning models to discover other textual elements hidden beneath surface word combinations. This invention creatively proposes a text-level neural machine translation method based on latent variables. In the modeling process, this method utilizes latent variables to represent the information needed for text-level neural machine translation, and introduces a discriminator to guide the generation process of latent variables, thereby improving the performance of text-level neural machine translation.

[0006] The innovation of this invention lies in the following: First, in the encoder-decoder neural machine translation model, the source language sentence and its preceding sentence are encoded separately, and the encoded information is fused through an attention mechanism to generate latent variables. Here, the distribution of the latent variables is considered to be a Gaussian distribution, containing the contextual information required for discourse-level translation. Although parallel corpora are not lacking, parallel corpora with discourse element annotations are missing. To address this, a discriminator is introduced. A discriminator pre-trained on a discourse relation analysis task maps the latent variables to a shallow discourse relation distribution through a transformation matrix. Finally, this discriminator is applied to the training process of the discourse-level neural machine translation model to guide the generation of latent variables and ultimately generate sentences in the target language.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] A text-level neural machine translation method based on latent variables includes the following steps:

[0009] Step 1: Select a discourse-level relation corpus (such as the CDTB Chinese Discourse-Level Relation Corpus) as the training set for the discourse relation analysis task. Train the discriminator based on the training set to obtain the discriminator.

[0010] Step 2: Encode the context sentence and the source language sentence, generate encoded information, and generate latent variables.

[0011] Specifically, it includes the following steps:

[0012] Step 2.1: Convert the context sentence and the source language sentence into a sequence of word vector representations through the embedding layer.

[0013] Step 2.1.1: Process the training corpus into samples containing context samples, source language sentences, and target language sentences.

[0014] Step 2.1.2: Use different word segmentation tools to segment the source language corpus and the target language corpus according to different languages.

[0015] Step 2.1.3: Encode the training corpus using Byte Pair Encoding (BPE) to obtain a vocabulary, and then perform sentence segmentation on the training corpus based on the vocabulary;

[0016] Step 2.1.4: Obtain the word vector representation sequence through embedding layer mapping.

[0017] Step 2.2: Encode the context sentence and the source language sentence using an encoder. During the source language sentence encoding process, a multi-head attention mechanism is used to fuse the encoded information E(X) of the context sentence. t ).

[0018] Specifically, an encoder is used to encode the word vector representation sequences of the context sentence and the source language sentence to obtain the encoded information E(X). t ).

[0019] Step 2.3: Utilize the encoded information E(X) t This guides the generation of prior latent variable z.

[0020] Specifically, the encoded information E(X) t The latent variable z is obtained by sampling the generated result as input to the latent variable generation network.

[0021] Step 3: Use a discriminator to guide the generation of posterior latent variables The decoder generates the target language sequence E(Y). t The translation model M is obtained by training based on the total loss.

[0022] Specifically, the target language encoding sequence E(Y) t ) and reference code R(Y t Compare the prior latent variable z with the posterior latent variable z. A comparison is then made. The differences are then weighted and summed to obtain the model loss. Based on the model loss, gradient descent is used to repeat steps 2.2 to 3 until convergence, yielding the translation model M.

[0023] Step 4: Input source language sentences that do not belong to the training set and the context of the source language sentences into the trained translation model. After the corresponding latent variables are generated, they are used together with the encoded information to generate target language sentences.

[0024] Beneficial effects

[0025] Compared with existing technologies, this invention focuses on learning the text associations in text-level machine translation. It proposes to model the context required for text-level machine translation using latent variables. The latent variable modeling method guided by the discriminator effectively utilizes contextual information and improves the quality of text-level machine translation.

[0026] On the English-Chinese translation corpus WIT3, experiments have demonstrated that the discriminator-guided latent variable modeling method can effectively improve the performance of discourse-level neural machine translation, and that this modeling method outperforms the end-to-end modeling method based on the Transformer model. Attached Figure Description

[0027] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the system architecture according to an embodiment of the present invention. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings.

[0030] Take the following sentence as an example, which includes the context sentence, the source language sentence, and the target language sentence:

[0031] Contextual sentence: The conference received more than 7,700 valid contributions.

[0032] Source language sentence: Finally, 1,150 papers were accepted.

[0033] Target language sentence: 1150 papers were accepted in the end.

[0034] like Figure 1 As shown, a document-level neural machine translation method based on latent variables includes the following steps:

[0035] Step 1: Select a discourse-level relation corpus (such as the CDTB Chinese Discourse-Level Relation Corpus) as the training set for the discourse relation analysis task. Train the discriminator based on the training set to obtain the discriminator.

[0036] Specifically, the CDTB corpus can be used as a task for discourse relation analysis, and a discriminator D can be trained.

[0037]

[0038] Here, Pretrain represents the pre-trained model. These represent different training passages in the corpus dataset and their corresponding labels.

[0039] Step 2: Encode the context sentence and the source language sentence to generate encoded information and generate latent variables.

[0040] Specifically, it includes the following steps:

[0041] Step 2.1: Convert the context sentence and the source language sentence into their word vector representation sequence through the embedding layer.

[0042] Specifically, a corpus of bilingual parallel sentence pairs with context was obtained, and the samples were preprocessed. For example, WIT3 Chinese and English were selected as the source and target languages, respectively. 160,000 high-quality parallel sentence pairs with context were extracted from the WIT3 corpus as training data. Each sample E includes a context sentence, a source language sentence, and a target language sentence.

[0043] For example, in the context of the sentence: I met a guy who had been an Air Force general and had shot down enemy planes in battle.

[0044] Source language sentence: And he is now a one-star general.

[0045] Target language sentence: He's now a one-star General.

[0046] First, segment the source language corpus and the language corpus separately, and construct a vocabulary. The Chinese word segmentation tool Jieba (https: / / github.com / fxsjy / jieba) can be used for Chinese word segmentation; English, due to its natural spaces, does not require segmentation.

[0047] Then, BPE is used for encoding to obtain the vocabulary.

[0048] Then, for each sample E, the sentence is split according to the vocabulary obtained from BPE encoding.

[0049] Finally, the sentence is converted into a sequence of word vector representations through embedding layer mapping.

[0050] Step 2.2: Encode the context sentence and the source language sentence separately using an encoder. During the source language sentence encoding process, the encoded information of the context sentence is fused using a multi-head attention mechanism.

[0051] Specifically, after obtaining the word vector representation sequence, the context sentence and source language sentence in sample E are fed into the encoder for encoding to obtain the source language sentence representation and latent variables that incorporate context information.

[0052] First, each word vector in the vector representation sequence of the context sentence is mapped to a query value Q, a key value K, and a value V using a matrix:

[0053] Q = W Q ·e (2)

[0054] K = W K ·e (3)

[0055] V = W V ·e (4)

[0056] Among them, W Q W K W V represents the query value Q, key value K, and value V, respectively; e represents the input word vector.

[0057] Then, for each word, its attention score *s* relative to all other words is calculated using a self-attention mechanism:

[0058]

[0059] Where, d k Let K be the dimension. T represents the transpose.

[0060] Then, using the attention score as the weight, the corresponding attention score is multiplied by V and summed to obtain the corresponding hidden layer state representation vector.

[0061]

[0062] Where s represents the attention score.

[0063] Here, the representations generated by each attention head need to be concatenated:

[0064]

[0065] Where h′ represents the concatenated hidden layer state representation vector. This is represented by the representation generated by the nth head.

[0066] Finally, two linear layers with ReLU activation function are used:

[0067] h=FFN(h′)=max(0,h′W1+b1)W2+b2 (8) where FFN(·) represents a linear unit, W1 and W2 represent the training weights, and b1 and b2 represent the bias terms corresponding to W1 and W2, respectively. This yields the encoding sequence of the context sentence.

[0068] The source language sentence is encoded using a self-attention layer as described above.

[0069] The multi-head attention mechanism fuses the encoded information of the context sentence, similar to the self-attention layer encoding mechanism. Here, Q and K are obtained by mapping words in the context sentence, while V is obtained by mapping words in the source language sentence. Similarly, two linear layers using ReLU as the activation function are used to obtain the source language sentence encoding information incorporating the context information.

[0070] Step 2.3: Generate latent variables using encoded information.

[0071] Specifically, the source language sentence encoding information obtained in step 2.2, which incorporates contextual information, is used to generate the mean μ and variance σ through a latent variable generation network, and then the latent variable z is obtained by sampling:

[0072]

[0073] in, This represents the encoding variable for the i-th word in the k-th sentence. This represents the encoding variable for the j-th word in the (k-1)-th sentence.

[0074] Step 3: Use a discriminator to guide the generation of latent variables.

[0075] Specifically, for each word y in the target language sentence t In the generation process, the discriminator obtained in step 1 is used to discriminate the latent variables.

[0076] The discrimination result obtained by the discriminator is converted into a confidence score c with an interval range of (0,1). t For a sequence of sentences of length T in the target language, calculate its sum of confidence c. t :

[0077] c t =Confidence(D(z),y t (10)

[0078] Where D(z) represents the discriminator's output for the latent variable z, and y t This represents the t-th word in the target language sentence.

[0079]

[0080] Where C is the value in the interval (0,T).

[0081] The difference between C and T is used as the loss at the discriminator's D terminal. The weighted result Included in the final total loss middle:

[0082]

[0083] in, This represents the latent variables generated by the posterior network. λ represents the model output, Y represents the reference translation, and λ represents the model output. D This represents hyperparameters.

[0084] As described above, even without labeled passage information, the discriminator can still guide the generation of latent variable z, thereby improving the final performance of the model.

[0085] Step 4: Output the target language sentence based on the context sentence, the source language sentence, and the latent variables.

[0086] The steps involved in decoding the target language sentence include:

[0087] Step 4.1: Using an embedding layer, the generated target language sentence is converted into its word vector representation sequence.

[0088] Step 4.1 is processed in the same way as step 2.1, resulting in a sequence of word vector representations of the target language sentence.

[0089] Step 4.2: Decode the target language sentence information using a decoder. During the decoding process, a multi-head attention mechanism is used to fuse the encoded information of the context sentence and the source language sentence.

[0090] Step 4.3: Generate a prediction of the next word in the target language sentence using a generator.

[0091] In step 4.2, the vector representation sequence of the target language sentence is first encoded using a self-attention layer, using the same method as the self-attention layer encoding in step 2.2. The first word of the target language sentence is... <bos>This serves as a representation of the sentence's beginning. By adding the latent variable vector to its representation and employing a self-attention mechanism, latent variable information can be fused during the target language sentence encoding process.

[0092] Then, the encoded information of the context sentence is fused through a multi-head attention mechanism. The principle is the same as the self-attention layer encoding mechanism. Q and K are derived from the mapping of the encoded information obtained in step 2.2, and V is derived from the mapping of words in the target language sentence.

[0093] Finally, similar to step 2.2, the target language decoding sequence is obtained through two linear layers activated by ReLU. The last vector of the target language decoding sequence is input into the generator, and mapped to the vocabulary through the Softmax function to obtain the prediction for the next word. The above steps are repeated until the sentence end marker is generated. <eos>Or the sentence length reaches the maximum length limit, the generation is terminated.

[0094] The method of the present application effectively utilizes context information through the discriminator-guided latent variable modeling method, and the semantic information such as conference and contributions is contained in the latent variable, so that the papers in the source language sentence can be accurately translated into papers, and paper or other incorrect translations will not be generated. Compared with the baseline model, the method improves the BLEU value by 1.2 on the same size corpus, and improves the quality of the document-level machine translation.

[0095] In another aspect, the present application provides a document-level neural machine translation system based on a latent variable model, which comprises a data processing module, an encoder module, a discriminator module, a decoder module and a generator module.As shown in the figure. Figure 2

[0096] The data processing module is responsible for extracting the data to be translated and converting it into a corresponding word vector sequence.

[0097] The data processing module comprises a sample acquisition unit, a preprocessing unit and an embedding layer unit. The sample acquisition unit is used to acquire language pair samples in the corpus; the preprocessing unit is used to acquire the required data from the corpus, including source language sentences, target language sentences and their corresponding source language context information, and is responsible for building a word table and sentence segmentation; the embedding layer unit is used to convert the processed text information into a corresponding word vector sequence.

[0098] In the sample acquisition unit, Chinese and English of WIT3 can be selected as the source language and the target language respectively.

[0099] In the preprocessing unit, the source language corpus and the language corpus are segmented respectively, and a word table is constructed. Then BPE is used for encoding to obtain a word table, and the sentence is split according to the word table obtained by BPE encoding.

[0100] In the embedding layer unit, the sentence is converted into a word vector representation sequence through mapping. The embedding layer unit comprises an encoding embedding layer and a decoding embedding layer.

[0101] The encoder module is used to encode the source language and the context information, and guide the generation of the latent variable in combination with the discriminator, and send the obtained encoding information and the latent variable to the decoder.

[0102] The encoder module comprises a context encoder module, an original text encoder module and a translation text encoder module. The context encoder module comprises a first attention unit and a first linear unit; the original text encoder module comprises a second attention unit, a context attention unit and a second linear unit; and the translation text encoder module comprises a third attention unit and a third linear unit.

[0103] The functions of the first attention unit, the second attention unit, and the third attention unit are all to fuse information from the input word vectors or semantic vectors to generate corresponding semantic information.

[0104] The functions of the first linear unit, the second linear unit, and the third linear unit are all to perform linear mapping on the input encoded information.

[0105] The context attention unit is responsible for integrating the encoding results of the context into the current encoding process.

[0106] The decoder module generates decoded information based on the obtained encoding information and latent variables.

[0107] The decoder module consists of multiple decoder layers stacked together (e.g., 6 decoders), each layer containing an attention unit, a source-translation attention unit, and a linear unit.

[0108] The role of the source-translation attention unit is to integrate the output of the source encoder into the decoding process of the translation.

[0109] The attention unit and linear unit function the same as the attention unit and linear unit in the encoder.

[0110] The generator module is responsible for mapping the decoded information to the target vocabulary, generating the translation, and presenting the result to the user.

[0111] Among all attention units, self-attention is used to calculate the relevance between words in the sentence, while multi-head attention is used to fuse contextual information.

[0112] In all linear units, the sequence information is processed to obtain the corresponding encoded and decoded sequences.

[0113] The discriminator module is responsible for outputting the discourse relation to which the current sentence pair belongs. Pre-training can use the CDTB corpus as the discourse relation analysis task and train the discriminator module accordingly.

[0114] The connection relationships of the above components are as follows:

[0115] In the data processing module, the sample acquisition unit is connected to the preprocessing unit, which in turn is connected to the embedding layer unit. The embedding layer unit, as the last unit in the data processing module, outputs a word vector matrix, which is then fed into the context encoder module, the original text encoder module, and the translated text encoder module.

[0116] The context encoder module is connected to the data processing module and the original text encoder module, respectively. The original text encoder module is connected to the data processing module, the context encoder module, the discriminator module, and the decoder module, respectively.

[0117] The original text encoder module outputs the original text encoded sequence and latent variables; the encoded sequence is input into the decoder module; the latent variables are output to the discriminator module and the decoder module respectively; the decoder module outputs the decoded sequence; and the generator module generates the translated sentence based on the decoded sequence.

[0118] In the context encoder module, original text encoder module, translated text encoder module, and decoder module, the attention unit is connected to the linear unit, and the output of the attention unit is directly fed into the linear unit.< / eos> < / bos>

Claims

1. A document-level neural machine translation method based on latent variables, characterized in that, Includes the following steps: Step 1: Select a discourse-level relation corpus as the training set for the discourse relation analysis task; The discriminator is obtained by training the training set; Step 2: Encode the context sentence and the source language sentence, generate encoded information, and generate latent variables; Step 2.1: Convert the context sentence and the source language sentence into a sequence of word vector representations through the embedding layer; Step 2.1.1: Process the training corpus into samples containing contextual samples, source language sentences, and target language sentences; Step 2.1.2: Use different word segmentation tools to segment the source language corpus and the target language corpus according to different languages; Step 2.1.3: Encode the training corpus using Byte-Pair Encoding (BPE) to obtain a vocabulary, and then perform sentence segmentation on the training corpus based on the vocabulary; Step 2.1.4: Obtain the word vector representation sequence through embedding layer mapping; Step 2.2: Encode the context sentence and the source language sentence using an encoder; during the source language sentence encoding process, use a multi-head attention mechanism to fuse the encoded information of the context sentence; Step 2.3: Use encoded information to guide the generation of prior latent variables; Step 3: Use the discriminator to guide the generation of posterior latent variables that contain the contextual information required for document-level translation and follow a Gaussian distribution. Use the decoder to generate the target language sequence and train the translation model based on the total loss. Step 4: Input source language sentences that do not belong to the training set and the context of the source language sentences into the trained translation model. After generating the corresponding prior latent variables, they are used together with the encoded information to generate target language sentences.

2. The document-level neural machine translation method based on latent variables as described in claim 1, characterized in that, In step 1, the training method for discriminator D is as follows: Here, Pretrain represents the pre-trained model. These represent different training passages in the corpus dataset and their corresponding labels.

3. The document-level neural machine translation method based on latent variables as described in claim 1, characterized in that, In step 2.2, after obtaining the word vector representation sequence, the context sentence and source language sentence in the sample are sent to the encoder for encoding to obtain the source language sentence representation and latent variables that incorporate context information; First, each word vector in the vector representation sequence of the context sentence is mapped to a query value Q, a key value K, and a value V using a matrix: Q=W Q ·e (2) K=W K e (3) V / W V ·e (4) Among them, W Q W K W V These represent the query value Q, key value K, and value V, respectively; e represents the input word vector. Then, for each word, its attention score *s* relative to all other words is calculated using a self-attention mechanism: Where, d k Let K be the dimension; T represents the transpose; Then, using the attention score as the weight, the corresponding attention score is multiplied by V and summed to obtain the corresponding hidden layer state representation vector. Where s represents the attention score; Here, the representations generated by each attention head need to be concatenated: Where h' represents the concatenated hidden layer state representation vector. This is represented as the representation generated by the nth head; Finally, two linear layers with ReLU activation function are used: h=FFN(h')=max(0,h'W1+b1)W2+b2 (8) Where FFN(·) represents a linear unit, W1 and W2 represent the training weights, and b1 and b2 represent the bias terms corresponding to W1 and W2, respectively; thus, the encoding sequence of the context sentence is obtained. Following the above method, the source language sentence is encoded through a self-attention layer; The encoding information of the context sentence is fused through a multi-head attention mechanism, which is the same as the self-attention layer encoding mechanism. Q and K are obtained by mapping words in the context sentence, and V is obtained by mapping words in the source language sentence. Similarly, the encoding information of the source language sentence with context information is obtained through two linear layers with ReLU as the activation function.

4. The document-level neural machine translation method based on latent variables as described in claim 1, characterized in that, In step 2.3, the source language sentence encoding information obtained in step 2.2, which incorporates contextual information, is used to generate the mean μ and variance σ through a latent variable generation network, and then the latent variable z is obtained by sampling. in, This represents the encoding variable for the i-th word in the k-th sentence. This represents the encoding variable for the j-th word in the (k-1)-th sentence.

5. The document-level neural machine translation method based on latent variables as described in claim 1, characterized in that, In step 3, the target language encoding sequence E(Y) is... t ) and reference code R(Y t Compare the prior latent variable z with the posterior latent variable z. Make comparisons; then, sum the differences in a weighted manner to obtain the model loss; based on the model loss, use the gradient descent method to repeat steps 2.2 to 3 until convergence, and obtain the translation model M; For each word y in the target language sentence t In the generation process, the discriminator obtained in step 1 is used to discriminate the latent variables; The discrimination result obtained by the discriminator is converted into a confidence score c with an interval range of (0,1). t For a sequence of sentences of length T in the target language, calculate its total confidence score c. t : c t =Confidence(D(z),y t ) (10) Where D(z) represents the discriminator's output for the latent variable z, and y t This represents the t-th word in the target language sentence; Where C is the value in the interval (0,T); The difference between C and T is used as the loss at the discriminator's D terminal. The weighted result Included in the final total loss middle: in, This represents the latent variables generated by the posterior network. λ represents the model output, Y represents the reference translation, and λ represents the model output. D This represents hyperparameters.

6. The document-level neural machine translation method based on latent variables as described in claim 1, characterized in that, Step 4 involves decoding the target language sentence, including the following steps: Step 4.1: Convert the generated target language sentence into its word vector representation sequence through the embedding layer; Step 4.2: Decode the target language sentence information using a decoder; during the decoding process, fuse the encoded information of the context sentence and the source language sentence using a multi-head attention mechanism; Step 4.3: Generate a prediction of the next word in the target language sentence using a generator; In step 4.2, the vector representation sequence of the target language sentence is first encoded using a self-attention layer, where the first word of the target language sentence is... <bos> The latent variable vector is added to its representation as the beginning of the sentence; the latent variable information is then fused into the target language sentence encoding process through a self-attention mechanism.< / bos> Then, the encoded information of the context sentence is fused through a multi-head attention mechanism; Finally, the target language decoding sequence is obtained through two linear layers with ReLU activation function; the last vector of the target language decoding sequence is input into the generator, and mapped to the vocabulary through the Softmax function to obtain the prediction for the next word; Repeat the above steps until the sentence end marker is generated. <eos> The generation process will terminate when the sentence length reaches the maximum length limit.< / eos>

Images