Multi-mode Mongolian-Chinese neural machine translation method fusing image information

Abstract

The invention provides a multi-mode Mongolian-Chinese neural machine translation method fusing image information, which comprises the following steps of: training to obtain a Mongolian text generation image model, inputting a source sentence into the Mongolian text generation image model to generate a corresponding semantic image, and inputting the source sentence into an encoder to obtain a representation form HL of the source sentence in a translation process; and fusing the image representation # imgabs0 # into a source sentence by using an attention mechanism to obtain a source representation # imgabs1 #, inputting the source representation # imgabs2 # into a decoder to calculate and obtain a time-dependent context vector of the decoder for predicting target translation, and finally outputting a Chinese text. According to the method, image information is fused to construct a multi-mode-based encoder-decoder framework, and the sentence-image pair with Mongolian description is combined, so that the image information and semantic information of sentences can be better utilized, and the translation fluency and translation quality of Mongolian-Chinese neural machine translation are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine translation, and in particular to a multimodal Mongolian-Chinese neural machine translation method integrating image information. Background Art

[0002] Machine translation studies how to use computers to automatically convert the semantics expressed in the source language into the same semantics in the target language. It is a major topic in the field of artificial intelligence and natural language processing. With the development of global economic globalization, exchanges and cooperation between countries are becoming more and more frequent, so translation between different languages ​​is receiving more and more attention. There are usually two translation methods, manual translation and machine translation. Manual translation is very effective, but it is time-consuming and labor-intensive and cannot adapt to the timeliness of language communication; machine translation is convenient, fast and timely, but it requires a lot of time to train the translation model in the early stage, and cannot guarantee the quality of translation.

[0003] In the current research on machine translation, in addition to translation between different texts, data in other modalities such as images and videos can also be used for translation, and the translation system can fuse this information together to further improve the quality of translation; modality refers to the appearance and existence of an event, and multimodality is the combination of two or more modalities. However, multimodal fusion technology for deep learning improves the performance of the model by transforming and fusing data in text, images, voice, video, etc., and the popularity of modal consistency and deep learning has promoted the development of this technology. The framework adopted by the classic neural machine translation model (NMT) is a sequence-to-sequence translation model based on encoding and decoding. The input is the source language word sequence X = (x1, x2, x3, ..., xm), and the output is the target language word sequence Y = (y1, y2, y3, ..., yn). The NMT model hopes to learn the model with the maximum probability P(Y|X) of translating X into Y, so as to learn the conditional probability distribution of the training set data. Based on the encoding and decoding end-to-end machine translation model framework, the source language sentence is encoded at the encoding end of the model, and the target language sentence is decoded using LSTM at the decoding end of the model. At the same time, the attention mechanism is introduced so that the decoding end can obtain the alignment information between the implicit state at that moment and the source language text when decoding and predicting the next target word. However, in the conversion-based translation system, the source input information plays a very critical role. From the perspective of minority languages, the research on Mongolian natural language processing started late and the relatively complex morphological changes of Mongolian itself have made the research progress of Mongolian-Chinese translation relatively slow. There are also phenomena of "over-translation" and "under-translation" in the translation process. The lack of parallel corpus makes it difficult to achieve high-quality translations. Summary of the invention

[0004] In view of the problem that Mongolian-Chinese machine translation in the prior art lacks parallel predictions and is difficult to achieve high-quality translation, the present invention proposes a multimodal Mongolian-Chinese neural machine translation method that integrates image information. A multimodal neural machine translation model is adopted, which mainly takes image and text modal information as input to realize end-to-end neural network modeling, and realizes translation from source language to target language by integrating image information to assist text.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] In the first aspect, the present invention proposes a multimodal Mongolian-Chinese neural machine translation method integrating image information, wherein a Mongolian text generation image model is trained, a source sentence is input into the Mongolian text generation image model to generate a corresponding semantic image, and during the translation process, the source sentence is input into an encoder to obtain a representation form H of the source sentence. L , and use the attention mechanism to represent the image Fusion to the source sentence to get the source representation Represent the source The input is fed into the decoder to calculate the dependent temporal context vector that the decoder uses to predict the target translation, and finally outputs Chinese text.

[0007] In the second aspect, the Mongolian text generation image model is formed by the following method:

[0008] Step 1: collect and organize data sets, use existing Mongolian-Chinese parallel corpus and image data sets, and perform preprocessing and feature extraction on the data in the data sets;

[0009] Step 2: Preprocess the Mongolian sentence-image pair through the topic-image conversion method to obtain a topic-image lookup table, and then preprocess the input source sentence to obtain the topic words, so as to retrieve the corresponding image for each topic word in the lookup table;

[0010] Step 3: Input Mongolian sentence features and image features into the AttnGAN network to train a Mongolian text-to-image generation model, and generate Mongolian images based on the Mongolian text-to-image generation model to obtain Mongolian semantic images.

[0011] Furthermore, the subject-image conversion method in step 1 performs preprocessing including:

[0012] Step 1.1, the present invention organizes the existing bilingual corpus and performs data cleaning on the bilingual corpus to improve the corpus quality, and then uses the Chinese word segmentation tool Jieba to segment the original data set;

[0013] Step 1.2, perform BPE subword processing on Mongolian and Chinese after jieba segmentation;

[0014] Step 1.3: According to the image content, the deep attention multimodal similarity model is used to select images that are more related to the text to serve as Mongolian sentence-image pair data, and the images obtained after the preliminary screening are described in corresponding Mongolian.

[0015] In step 1.4, the extracted image features are fed into a feedforward neural network, and then the attention mechanism is used to focus on the image information. After that, the image information is fused with the encoded text information through a gated residual network.

[0016] Furthermore, the step 2 specifically includes:

[0017] Step 2.1, convert the existing sentence-image pairs into a topic-image lookup table, and extract the topic words of the sentences by word frequency inverse document frequency;

[0018] Step 2.2: Select the keywords that are most consistent with the image as the new image description T = t1, t2, ..., t w} is used to input sentences. After preprocessing, each filtered sentence T is paired with an image e, and each word t i ∈T is the topic word of image e. After processing the entire corpus, a topic-image lookup table Q is formed, in which each topic word t i Will be paired with the image.

[0019] Furthermore, the method of extracting the topic words of a sentence by inverse document frequency is as follows: given a source input sentence X with a length of L = {x1, x2, ... x i} and its paired image are first filtered by the stop word list, then the filtered sentences are processed as file g, and finally the word x in g is calculated. i TI i,j Value, calculated as:

[0020]

[0021] Among them i,j Represents word x i The number of occurrences in the input sentence g, |G| represents the total number of source language sentences in the training data, |j:x i ∈g| means that the training data contains word x i The number of source sentences.

[0022] Furthermore, the step 3 specifically includes:

[0023] Step 3.1: Train a deep attention multimodal similarity model (DAMSM) through the corpus to select images generated by the attention-generated adversarial network that are more relevant to the text;

[0024] Step 3.2, build and implement a Mongolian text generation image model based on AttnGAN.

[0025] Furthermore, the step 3.1 includes: using the image encoder in DAMSM to learn the local characteristics of different sub-regions of the image and fuse their overall features; using the text encoder to encode the source language sentence using a bidirectional LSTM; and using an attention-based graphic matching score method to measure whether the sentences in the image are consistent.

[0026] Furthermore, in the translation process, the source sentence X = {x1, x2, ..., x I The input encoder learns the representation H of the source sentence L , and then transform the image E = {e1, e2, ..., e m} Input the pre-trained ResNet, and then go to the feed-forward layer to learn the representation textM of the source image, and then use the attention mechanism to attach the image representation to the text representation. The calculation method is as follows: where {K M ,V M} is the value in M ​​from the learned source image representation.

[0027] Furthermore, the source represents λ∈[0,1], where W λ and U λ is the model parameter, fusion H L and To learn an effective source representation

[0028] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multimodal Mongolian-Chinese neural machine translation method is implemented.

[0029] Compared with the prior art, the present invention integrates image information to construct a multimodal encoder-decoder framework, and improves the integration of image information between the encoder and the decoder. Combining sentence-image pairs with Mongolian descriptions can better utilize image information and sentence semantic information. At the same time, the present invention can combine the language features and syntactic features of Mongolian to effectively improve the translation fluency and translation quality of Mongolian-Chinese neural machine translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the present invention.

[0031] Figure 2 Generate image model for Mongolian text based on AttnGAN. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] like Figure 1 As shown, the present invention is a multimodal Mongolian-Chinese neural machine translation method integrating image information, which mainly obtains a Mongolian text generation image model through training, inputs the source sentence into the Mongolian text generation image model to generate the corresponding semantic image, and in the translation process, inputs the source sentence into the encoder to obtain the representation form H of the source sentence. L , and use the attention mechanism to represent the image Fusion to the source sentence to get the source representation Represent the source The input is fed into the decoder to calculate the dependent temporal context vector that the decoder uses to predict the target translation, and finally outputs Chinese text.

[0034] Specifically, the method includes the following steps. Among them:

[0035] Step 1: Corpus preprocessing: collect and organize data sets, use existing Mongolian-Chinese parallel corpora and image data sets to preprocess and extract features from the data sets.

[0036] Specifically include:

[0037] Step 1.1, the present invention organizes the existing bilingual corpus and performs data cleaning on the bilingual corpus to improve the corpus quality, and then uses the Chinese word segmentation tool Jieba to segment the original data set.

[0038] Step 1.2, perform BPE subword processing on Mongolian and Chinese after jieba word segmentation. BPE (byte pair encoding) is a method for processing text at subword level granularity. This method uses controllable segmentation granularity to segment low-frequency words and keep high-frequency words intact to alleviate the data sparsity problem. Due to the scarcity of Mongolian-Chinese parallel corpus, there will be some low-frequency words in the Chinese data. After using BPE subword processing, it is beneficial to the translation of the model.

[0039] Step 1.3: According to the image content, the Deep Attention Multimodal Similarity Model (DAMSM) is used to select images that are more related to the text as the data of Mongolian sentence-image pairs, and the images obtained after the preliminary screening are described in corresponding Mongolian.

[0040] In step 1.4, the extracted image features are fed into a feedforward neural network, and then the attention mechanism is used to focus on the image information. After that, the image information is fused with the encoded text information through a gated residual network, and feature extraction is performed to facilitate the translation of subsequent models.

[0041] Step 2, preprocess the Mongolian sentence-image pairs through the topic-image conversion method to obtain a topic-image lookup table, and then preprocess the input source sentence (Mongolian) to obtain the topic words, so as to retrieve the corresponding image for each topic word in the lookup table.

[0042] Specifically include:

[0043] In step 2.1, the existing sentence-image pairs are converted into a topic-image lookup table, and the “topic” words of the sentences are extracted through term frequency-inverse document frequency (TF-IDF).

[0044] First, given a source input sentence X = {x1, x2, ... x i} and its paired image are first filtered by the stop word list, then the filtered sentences are processed as file g, and finally the word x in g is calculated. i TI i,j Value, calculated as: Among them i,j Represents word x i The number of occurrences in the input sentence g, |G| represents the total number of source language sentences in the training data, |j:x i ∈g means that the training data contains word x i The number of source sentences.

[0045] Select the top TF-IDF words with high w values ​​(high fit with the image) as the new image description T = t1, t2, ..., t w} is used to input sentence X. After preprocessing, each filtered sentence T is paired with an image e, and each word t i ∈T is the topic word of image e. After processing the entire corpus, a topic-image lookup table Q is formed, in which each topic word t i Will be paired with the image.

[0046] Step 3: Input Mongolian sentence features and image features into the AttnGAN network to train a Mongolian text-to-image generation model, and generate Mongolian images based on the Mongolian text-to-image generation model to obtain Mongolian semantic images.

[0047] Step 3.1, train a deep attention multimodal similarity model (DAMSM) through the corpus, screen out images that are more related to text generated by the attention-generated adversarial network, use the image encoder in DAMSM to learn the local characteristics of different sub-regions of the image, and fuse the overall features; use the text encoder to encode the source language sentences using a bidirectional LSTM; and use the attention-based graphic matching score method to measure whether the sentences in the image are consistent.

[0048] Specifically, the present invention measures the similarity between images and texts at the word level to find out the slight loss. The present invention focuses on using the image encoder, text encoder and attention-based graphic matching score method in DAMSM.

[0049] The image encoder is an Inception-v3 model based on ImageNet, which combines images with semantic vector convolutional neural networks. Through the intermediate layer network of CNN, the local characteristics of different sub-regions can be learned and their overall features can be integrated. The image encoder generates a 299*299 high-definition image and extracts the local feature matrix. Each column is a feature vector of an image sub-region, 768 is the localization unit vector dimension, and 289 is the number of image sub-regions. To measure the similarity between images and text, it is necessary to ensure the consistency of the feature size of text and images, and make the image feature conversion conform to the size of the text vector. The calculation formula is: v = Wf, Where v is the feature after image feature conversion represents the global vector of the image, The work extracts global features from the last layer of the Inception-v3 network and uses it as the overall feature.

[0050] The text encoding under the DAMSM model is a bidirectional long short-term memory network based on the semantic vector description of the text. In the bidirectional LSTM, each word has two hidden states, and each state has an implicit state. There is one in each direction, so its two hidden states are connected together to represent the meaning of the word. For example, the forward LSTM receives the word vector of each word and the implicit vector output by the previous unit from left to right according to the word order, and obtains the output sequence. The reverse LSTM receives the input of the word sequence and the implicit output of the previous unit from right to left to obtain the output sequence. The final output of the encoder at each timestamp is the concatenation of the forward and backward implicit vectors of each word. The feature matrix of all words is given by Indicates that D is the dimension of the word vector and T is the number of words. At the same time, the last hidden bidirectional LSTM state is connected as the global sentence vector. express.

[0051] The attention-based graph matching score method measures whether the sentence in the image is consistent based on the attention model between the image and the text. First, the similarity matrix of the sub-regions in the image and all the word pairs in the sentence are calculated using the calculation method formula: s = e T v, where Represents the inner product of a word and a sub-region, and is used to measure similarity. In order to improve the efficiency of training, the similarity of the same sub-region and word is standardized, and the calculation formula is: Then, this method is used to perform vector operations on each word in the regional context, and the regional context vector c i is a dynamic representation, which involves i words in a sentence. This method is calculated by the sum of the visual vectors of all regions, as shown in the formula: In this method, γ1 represents the proportion of the relevant sub-regions so that each word has the corresponding regional contextual visualization information, thereby enabling a measure of the association between vocabulary and visual information.

[0052] The difference is measured by defining the cosine distance, which is calculated as follows:

[0053]

[0054] By analyzing the text level γ2, we can see the similarity between the overall image and the text. The calculation method is as follows:

[0055] Step 3.2, build and implement a Mongolian text generation image model based on AttnGAN.

[0056] In step 3.2, combine Figure 2 As shown, the present invention uses an attention-based generative adversarial network model, which has m generators (G0, G1, ... G m-1 ), so the generator follows the hidden state (h0,h1,…h m-1 ) as input, and is divided into three stages of generation: F0, F1, and F2. F0 generates a quarter-resolution image (64*64*3) based on the overall characteristics, F1 generates an image with half resolution (128*128*3) through the attention mechanism, and F2 generates an image with the same resolution as the target (256*256*3) through the attention mechanism. The text in the F0 stage is extracted through the text encoder to extract sentence features and word features After dimensionality reduction, it is converted into a conditional vector c. The conditional vector c is combined with random noise z~N(0,1) that conforms to the Gaussian distribution, and through a series of upsampling operations, an implicit eigenvalue is obtained, in which the basic information of the position and target of the image is already contained.

[0057] Among them, the calculation method for generating the first image feature is as follows:

[0058] The calculation method for the subsequent image generation process is as follows:

[0059]

[0060] Among them, F attn It is to generate context vectors for all sub-regions.

[0061] The input here is the word matrix vector e and the previous implicit features After the word vector is transformed by a product, the dimension of the entire connection layer is changed to dimension, then the word vector e satisfies This method is consistent with the feature dimension of the image, which facilitates the calculation of similarity by the inner product operation in the later stage. Each column in h represents a sub-region of the image, and the jth sub-region is represented by all the word vectors in the sentence. Therefore, the corresponding word vector should have a larger weight, while the weight of irrelevant word vectors is smaller. The weighted result of the word vector of each sub-region is the context vector. Each sub-region and the final context vector are weighted and calculated as follows:

[0062]

[0063] Among them, β j,i represents the attention that the i-th word attracts when generating image region j, c jRepresents a context vector, which is the jth sub-region. The final objective function of the module is calculated as follows:

[0064]

[0065] The three stages F0, F1, and F2 eventually refine the granularity of the image gradually to generate an image based on Mongolian text.

[0066] Finally, the model is trained. Using the trained model, the source sentence is input into it for prediction. First, the source sentence X = {x1, x2, ..., x I The input encoder learns the representation H of the source sentence L Then the image E={e1,e2,...,e m} Input the pre-trained ResNet and then to the feed-forward layer to learn the representation of the source image textM∈R m×2048 .

[0067] The image representation is attached to the text representation using the attention mechanism, and the source sentence representation is as follows: where {K M ,V M} is the value in the learned source image representation M. The goal of NMT is to generate a target sentence with the same meaning as the source sentence, rather than a series of pictures. In other words, in translation prediction, image information is an auxiliary means, so the source image representation and the source sentence representation are fused to obtain a final source representation Source Representation λ∈[0,1], where W λ and U λ is the model parameter, fusion H L and To learn an effective source representation It is then input into the decoder to learn the dependent temporal context vector for predicting the target translation, and finally decoded by the decoder to obtain the same vector representation as the source sentence.

[0068] Finally, the present invention can achieve the work of translating Mongolian into image information by generating an image model through the trained Mongolian text, so as to use Mongolian image information to assist text information in translating Mongolian to Chinese. The present invention integrates image information to construct an encoder-decoder framework based on multimodality, and improves the image information fusion between the encoder and the decoder. In combination with sentence-image pairs described in Mongolian, the image information and the semantic information of the sentence can be better utilized. At the same time, the present invention can combine the language characteristics and syntactic features of Mongolian to effectively improve the translation fluency and translation quality of Mongolian-Chinese neural machine translation.

Claims

1. A multimodal Mongolian-Chinese neural machine translation method integrating image information, characterized in that: A Mongolian text-generated image model is trained, and the source sentence is input into the Mongolian text-generated image model to generate the corresponding semantic image. During the translation process, the source sentence is input into the encoder to obtain the representation form H of the source sentence. L , and use the attention mechanism to represent the image Fusion to the source sentence to get the source representation Represent the source The input is fed into the decoder to calculate the dependent temporal context vector that the decoder uses to predict the target translation, and finally outputs Chinese text.

2. The multimodal Mongolian-Chinese neural machine translation method according to claim 1, characterized in that: The Mongolian text generation image model is formed by the following method: Step 1: collect and organize data sets, use existing Mongolian-Chinese parallel corpus and image data sets, and perform preprocessing and feature extraction on the data in the data sets; Step 2: Preprocess the Mongolian sentence-image pair through the topic-image conversion method to obtain a topic-image lookup table, and then preprocess the input source sentence to obtain the topic words, so as to retrieve the corresponding image for each topic word in the lookup table; Step 3: Input Mongolian sentence features and image features into the AttnGAN network to train a Mongolian text-to-image generation model, and generate Mongolian images based on the Mongolian text-to-image generation model to obtain Mongolian semantic images.

3. The multimodal Mongolian-Chinese neural machine translation method according to claim 2, characterized in that: The subject-image conversion method in step 1 performs preprocessing including: Step 1.1, the present invention organizes the existing bilingual corpus and performs data cleaning on the bilingual corpus to improve the corpus quality, and then uses the Chinese word segmentation tool Jieba to segment the original data set; Step 1.2, perform BPE subword processing on Mongolian and Chinese after jieba segmentation; Step 1.3: According to the image content, the deep attention multimodal similarity model is used to select images that are more related to the text to serve as Mongolian sentence-image pair data, and the images obtained after the preliminary screening are described in corresponding Mongolian. In step 1.4, the extracted image features are fed into a feedforward neural network, and then the attention mechanism is used to focus on the image information. After that, the image information is fused with the encoded text information through a gated residual network.

4. The multimodal Mongolian-Chinese neural machine translation method according to claim 2, characterized in that: The step 2 specifically includes: Step 2.1, convert the existing sentence-image pairs into a topic-image lookup table, and extract the topic words of the sentences by word frequency inverse document frequency; Step 2.2: Select the keywords that are most consistent with the image as the new image description T = t1, t2, ..., t w } is used to input sentences. After preprocessing, each filtered sentence T is paired with an image e, and each word t i ∈T is the topic word of image e. After processing the entire corpus, a topic-image lookup table Q is formed, in which each topic word t i Will be paired with the image.

5. The multimodal Mongolian-Chinese neural machine translation method according to claim 4, characterized in that: The method of extracting the topic words of a sentence by word frequency inverse document frequency is as follows: given a source input sentence X with a length of L = {x1, x2, ... x i } and its paired image are first filtered by the stop word list, then the filtered sentences are processed as file g, and finally the word x in g is calculated. i TI i,j Value, calculated as: Among them i,j Represents word x i The number of occurrences in the input sentence g, |G| represents the total number of source language sentences in the training data, |j:x i ∈g| means that the training data contains word x i The number of source sentences.

6. The multimodal Mongolian-Chinese neural machine translation method according to claim 5, characterized in that: The step 3 specifically includes: Step 3.1: Train a deep attention multimodal similarity model (DAMSM) through the corpus to select images generated by the attention-generated adversarial network that are more relevant to the text; Step 3.2, build and implement a Mongolian text generation image model based on AttnGAN.

7. The multimodal Mongolian-Chinese neural machine translation method according to claim 6, characterized in that: The step 3.1 includes: using the image encoder in DAMSM to learn the local characteristics of different sub-regions of the image and fuse them into overall features; using the text encoder to encode the source language sentence using a bidirectional LSTM; and using an attention-based graphic matching score method to measure whether the sentences in the image are consistent.

8. The multimodal Mongolian-Chinese neural machine translation method according to claim 1, characterized in that: In the translation process, the source sentence X = {x1, x2, ..., x I The input encoder learns the representation H of the source sentence L , and then transform the image E = {e1, e2, ..., e m } Input the pre-trained ResNet, and then go to the feed-forward layer to learn the representation textM of the source image, and then use the attention mechanism to attach the image representation to the text representation. The calculation method is as follows: where {K M ,V M } is the value in M ​​from the learned source image representation.

9. The multimodal Mongolian-Chinese neural machine translation method according to claim 8, characterized in that: Source Representation λ∈[0,1], where W λ and U λ is the model parameter, fusion H L and To learn an effective source representation 10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the multimodal Mongolian-Chinese neural machine translation method as described in any one of claims 1-9.

Images