A multimodal machine translation method based on pre-training

Abstract

The present invention provides a pre-training-based multimodal machine translation method, comprising the following steps: processing a multimodal machine translation dataset, Multi30k, comprising two input parts: a text description and an image; constructing a Vision Transformer model, processing the image input, and obtaining an image encoding representation; inheriting the pre-training parameters of Huggingface, and constructing an mBart model based on the Transformer structure; the text encoding module of the mBart model processes the text input to obtain a text encoding representation; designing a prompt template, connecting the image encoding representation and the text encoding representation, and feeding the result into the encoder module of the mBart model; fixing the pre-training parameters of the mBart model, introducing the KL divergence into the loss function calculation, and fine-tuning the model. By inheriting the parameters of a large-scale text pre-training model and fine-tuning the image encoding representation module, the present invention effectively applies large-scale text pre-training knowledge to multimodal machine translation, improving the performance of multimodal machine translation and providing ideas for subsequent research on other multimodal tasks.

Description

Technical Field

[0001] The present invention relates to a multimodal machine translation technology, in particular to a multimodal machine translation method based on pre-training. Background Art

[0002] Machine translation is the process of converting one natural language into another using computers. A traditional task in natural language processing, machine translation has evolved through rule-based approaches, entity-based methods, statistical machine translation, and now neural machine translation. Based on the Transformer model framework, which has been the mainstream in recent years, numerous researchers have optimized and improved it. Today, machine translation technology has reached considerable maturity and is now used in daily life.

[0003] Multimodal machine translation is a multimodal task that incorporates image information corresponding to text into the traditional machine translation process. Unlike machine translation, which has developed over decades, multimodal machine translation, as an extension of traditional neural machine translation, has only been around for a few years. Current multimodal machine translation tasks aim to utilize image information that matches bilingual text to supplement additional information and enhance translation, making them a cross-modal and cross-disciplinary research task.

[0004] For the image portion of multimodal machine translation, the limited size of existing multimodal machine translation datasets makes it difficult to train an image feature extractor from scratch. Existing methods are based on pre-trained models, such as ResNet and RCNN, to extract image features and thus achieve sufficient representation of image content. For the text portion, the current data size is sufficient to train an excellent translation model from scratch. Therefore, existing discussions often overlook the impact of the text module as a core component on multimodal machine translation. In fact, many translation errors originate from the text translation itself, so images are needed to provide additional correct information to supplement it.

[0005] Since most current multimodal machine translation methods are limited to the extraction of modal information and the improvement of modal interaction structures, the performance improvement of such methods is very limited when the text is not restricted. In other words, the current methods cannot solve the problem of sufficient text itself. Based on the current problems, the pre-training method of the image part provides inspiration. For multimodal machine translation with scarce data, large-scale pre-training can further improve the translation quality and solve the problems of the text itself. However, the problem of scarce multimodal machine translation data is also a bottleneck for joint pre-training of bilingual and images. Although some researchers have performed pre-training by expanding the multimodal machine translation dataset, the role of pre-training has been proved to a certain extent. However, the problem of data scarcity makes the direct pre-training method only applicable to the case where the text in the current test set is simple, and it is difficult to handle more complex tasks.

[0006] To address these issues, considering the powerful text representation capabilities of the large-scale text pre-trained model mBart, migrating it to the field of multimodal machine translation can fully leverage the pre-trained model's text representation capabilities, overcome performance bottlenecks caused by data scarcity, and further improve the performance of multimodal machine translation. However, using text pre-trained models to enhance multimodal machine translation also presents challenges: how to integrate image modules into the text-pretrained mBart model, how to align image and text representations, and how to fine-tune the model. These issues make the current application of text pre-trained models in multimodal machine translation immature and require further exploration. Summary of the Invention

[0007] In view of the problems that existing multimodal machine translation technologies are mostly limited to the extraction of modal information and the improvement of modal interaction structure, and the performance improvement is very limited, and the attempts to introduce the advantages of large-scale text pre-training models into the field of multimodal machine translation are still immature, the technical problem to be solved by the present invention is to provide a multimodal machine translation method based on pre-training and explore the application of large-scale text pre-training models in multimodal machine translation tasks.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] A multimodal machine translation method based on pre-training, comprising the following steps:

[0010] 1) Processing multimodal machine translation datasets, which contain two parts of input data: text description and corresponding images;

[0011] 2) Build a Vision Transformer model to process image input data and obtain image encoding representation;

[0012] 3) Using existing pre-trained model parameters, a pre-trained model based on the Transformer structure is constructed. The text input data passes through the encoding layer of the pre-trained model to obtain the text encoding representation;

[0013] 4) Design a prompt template, connect the image encoding representation and the text encoding representation, and feed it into the encoder module of the pre-trained model;

[0014] 5) Fix the pre-trained model's pre-trained parameters, introduce KL divergence into the loss function calculation, fine-tune the model, and input the fine-tuned model into images and text to achieve multimodal machine translation.

[0015] In step 1), training data is obtained from a public multimodal machine translation dataset. The image data is divided into fixed-size sub-images, each of which is called a patch. The dimensions are then adjusted to a suitable input size for the model. The encoding of each patch is obtained, that is, the image embedding vector, which is specifically:

[0016] 101) Take input (batch, c, h, w), each patch resolution is p1*p2, and the specific process of dividing it into sub-pictures: First, each picture is divided into (h / p1)*(w / p2) small blocks, that is, from (batch, c, p1*(h / p1), p2*(w / p2)) to (batch, c, (h / p1)*(w / p2), p1*p2), and then converted into (batch, (h / p1)*(w / p2), p1*p2*c), which is equivalent to dividing it into (h / p1)*(w / p2) patches, and the dimension of each patch is p1*p2*c; the implementation of this process is calculated by the following formula:

[0017] x=rearrange(img,′bc(h p1)(w p2)→b(hw)(p1 p2 c)′)

[0018] Among them, the rearrange function is an operator of the einops library, p1 and p2 are the number of pixels of the height and width of a patch, c is the number of channels, b is the number of batches, batch represents the batch of data during model training, and h and w are the number of pixels of the height and width of the image respectively.

[0019] 102) After dividing the sub-images, the image embedding vector is obtained and its dimension is adjusted. The image embedding vector is adjusted to the required length through a fully connected layer and used as the input of the model.

[0020] In step 2), the Vision Transformer model is constructed to process the image embedding vector input and obtain the image encoding representation, specifically:

[0021] 201) Use 2D sincos encoding to encode the position information. The encoding formula is:

[0022]

[0023]

[0024] Among them, PE is a two-dimensional matrix, the rows represent sub-pictures, the columns represent position vectors; pos represents the position of the sub-picture in the picture; d model Represents the dimension of the position vector; i represents the position of the position vector;

[0025] 202) The image embedding vector and the two-dimensional position encoding vector are concatenated to obtain an input vector, which is fed into the encoder structure of the VisionTransformer model to obtain the image encoding representation.

[0026] In step 4), a prompt template is designed, which connects the image encoding representation and the text encoding representation and is fed into the encoder module of the mBart model. Specifically:

[0027] 401) Design a prompt template, encode the image as X, encode the source language text as Y, encode the target language text as Z, and encode the prompt template as "[prompt 1]", "[prompt 2]", and "[prompt 3]". Concatenate them to obtain the encoder input encoding representation "[prompt 1]:X[prompt 2]:Y";

[0028] 402) encoding the text portion using the position encoding module of the pre-trained model to obtain a text position code, and encoding each dimension of the image portion as if it were the same position to obtain an image position code; concatenating the text position code and the image position code, adding the result to the input code representation, and feeding the result to the encoder module of the pre-trained model to perform joint representation learning of the image and text;

[0029] 403) Connect the prompt template and the target language text encoding to obtain the decoder input encoding representation "[prompt 3]:Z", which is sent to the decoder module of the pre-trained model for autoregressive generation of the target language.

[0030] In step 5), the pre-training parameters of the pre-training model are fixed, and the KL divergence between the text encoding representation and the image encoding representation is calculated. The loss L KL Compared with the original cross entropy loss L of the pre-training model CE Add them together to get the new loss L, and fine-tune the model; the loss formula is:

[0031] L=L CE +αL KL

[0032]

[0033]

[0034] Among them, p(·) represents the normalized text encoding representation, q(·) represents the normalized image encoding representation, and x i represents the i-th dimension of the distribution, L CE is the cross entropy loss, L KL is the KL divergence loss.

[0035] The present invention has the following beneficial effects and advantages:

[0036] 1. This paper utilizes a large-scale text pre-training model to effectively transfer pre-trained text knowledge to the field of multimodal machine translation. Using KL divergence loss to align image and text representations, it successfully integrates the image module into the text pre-training model, improving the performance of multimodal machine translation.

[0037] 2. At the same time, the present invention encodes the image into a text-like word string through the image encoding method of the Vision Transformer model, which can better align the image representation and text representation while obtaining powerful image feature extraction capabilities.

[0038] 3. This invention successfully applies large-scale text pre-training models to the field of multimodal machine translation, providing ideas for subsequently utilizing the knowledge of text pre-training models and aligning different modal information in other multimodal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the model structure in the method of the present invention;

[0040] Figure 2 This is a diagram illustrating the image division method of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings.

[0042] This paper addresses the fact that traditional multimodal machine translation techniques are mostly limited to extracting modal information and improving modal interaction structures, resulting in very limited performance improvements. This paper proposes a pre-trained multimodal machine translation method. This method successfully applies a large-scale text pre-training model to the field of multimodal machine translation. By extracting image encoding representations through the VisionTransformer model, designing hint templates, and introducing the KL divergence loss to align images and text, this method not only achieves effective multimodal machine translation but also opens up new possibilities for subsequent multimodal research.

[0043] The method of the present invention comprises the following steps:

[0044] 1) Processing the Multi30k multimodal machine translation dataset, which contains two input parts: text description and corresponding image;

[0045] 2) Build a Vision Transformer model to process image input and obtain image encoding representation;

[0046] 3) Using the pre-trained model parameters of Huggingface, we construct the mBart model based on the Transformer structure. The text input passes through the Embedding layer of the mBart model to obtain the text encoding representation;

[0047] 4) Design a prompt template, connect the image encoding representation and the text encoding representation, and feed it into the encoder module of the mBart model;

[0048] 5) Fix the pre-training parameters of the mBart model, introduce the KL divergence into the loss function calculation, and fine-tune the model.

[0049] In step 1), we obtain training data from the public multimodal machine translation dataset Multi30k, divide the image into fixed-size sub-images, each sub-image is called a patch, and then adjust its dimensions to a suitable input size for the model to obtain the encoding of each patch, that is, the image embedding vector. This process is as follows Figure 2 As shown, specifically:

[0050] 101) Take the input (batch, 3, 384, 384) with each patch resolution of 32*32 as an example. The specific process of dividing into sub-images is as follows: First, each image is divided into 12*12 small blocks, that is, from (batch, 3, 32*12, 32*12) to (batch, 3, 12*12, 32*32), and then converted to (batch, 12*12, 32*32*3), which is equivalent to dividing it into 12*12 patches, and the dimension of each patch is 32*32*3. This process can be implemented by the following calculation:

[0051] x=rearrange(img,′bc(hp1)(w p2)→b(hw)(p1 p2 c)′)

[0052] Among them, the rearrange function is an operator of the einops library, p1 and p2 are the number of pixels of the height and width of a patch, c is the number of channels, b is the number of batches, batch represents the batch of data during model training, and h and w are the number of pixels of the height and width of the image respectively.

[0053] 102) After dividing the image into sub-images, the image embedding vector is obtained and its dimension needs to be adjusted. Here, the image embedding vector is resized to 1024 through a fully connected layer and used as the input of the model.

[0054] In step 2), the Vision Transformer model is constructed to process the image embedding vector input to obtain the image encoding representation, specifically:

[0055] 201) Use 2D sincos encoding to encode the position information. The encoding formula is:

[0056]

[0057]

[0058] Among them, PE is a two-dimensional matrix, the rows represent sub-pictures, the columns represent position vectors; pos represents the position of the sub-picture in the picture; d model Represents the dimension of the position vector; i represents the position of the position vector;

[0059] 202) The image embedding vector and the two-dimensional position encoding vector are concatenated to obtain an input vector, which is fed into the encoder structure of VisionTransformer to obtain the image encoding representation.

[0060] In step 3), the pre-trained parameters of Huggingface are used to build the mBart model based on the Transformer structure. After the text data passes through the Embedding layer of the mBart model, the text encoding representation is obtained. This process is as follows Figure 1 shown.

[0061] In step 4), a prompt template is designed, the image encoding representation and the text encoding representation are connected, and the template is fed into the encoder module of the mBart model. This process is as follows: Figure 1 As shown, specifically:

[0062] 401) Design a prompt template, represent the image encoding as X, the source language text encoding as Y, and the target language text encoding as Z. The corresponding prompt templates are "IMG", "EN", and "DE", and the encoder input encoding representation is "IMG:XEN:Y" is obtained by concatenating them.

[0063] 402) Encode the text portion using the positional encoding module of mBart to obtain a text positional encoding, and encode each dimension of the image portion as if it were the same position to obtain an image positional encoding. Concatenate the text positional encoding and the image positional encoding, add them to the input encoding representation, and feed them into the encoder module of the mBart model to learn a joint representation of the image and text.

[0064] 403) Connect the prompt template and the target language text encoding to obtain the decoder input encoding representation "DE:Z", which is sent to the decoder module of the mBart model for autoregressive generation of the target language.

[0065] In step 5), the pre-trained parameters of the mBart model are fixed, and the KL divergence between the text encoding representation and the image encoding representation is calculated to obtain the loss L KL Compared with the original cross entropy loss L of the mBart modelCE Add them together to get the new loss L, and fine-tune the model. The loss formula is:

[0066] L=L CE +αL KL

[0067]

[0068]

[0069] Among them, L CE is the cross entropy loss, L KL is the KL divergence loss, p(·) represents the normalized text encoding representation, q(·) represents the normalized image encoding representation, and x i represents the i-th dimension of the distribution.

[0070] This embodiment takes the dataset Multi30k as an example and applies the pre-trained multimodal machine translation model to the English-German-2016 task and the English-French-2016 task of Multi30k. The Multi30k dataset contains 31,014 images and corresponding bilingual texts, which constitute the data for training and testing. It can be found from Table 1 that the method proposed in the present invention has a significant improvement in BLEU value compared with the Transformer model for text translation and the traditional multimodal machine translation model that designs image and text interaction modules. It proves that the method proposed in the present invention can effectively utilize image information and pre-trained text knowledge to improve the performance of multimodal machine translation.

[0071]

[0072] Table 1 Comparison of experimental results

[0073] The present invention proposes a multimodal machine translation method based on pre-training, which can transfer the knowledge of the large-scale text pre-training model mBart to the field of multimodal machine translation and enhance translation performance. This method obtains image representation by extracting image features using the VisionTransformer model. By designing a prompt template and introducing KL divergence loss, the image representation and text representation are aligned to solve the problem of multimodal information mismatch under text pre-training. Using the text pre-training model to process multimodal tasks verifies the text representation ability of the large-scale text pre-training model and the importance of multimodal information alignment. At the same time, the pre-trained multimodal machine translation method also provides new thinking for future tasks in other multimodal fields.

Claims

1. A multimodal machine translation method based on pre-training, characterized by comprising the following steps: 1) Processing multimodal machine translation datasets, which contain two parts of input data: text description and corresponding images; 2) Build a Vision Transformer model to process image input data and obtain image encoding representation; 3) Using existing pre-trained model parameters, a pre-trained model based on the Transformer structure is constructed. The text input data passes through the encoding layer of the pre-trained model to obtain the text encoding representation; 4) Design a prompt template, connect the image encoding representation and the text encoding representation, and feed it into the encoder module of the pre-trained model; 5) Fix the pre-trained model's pre-trained parameters, introduce KL divergence into the loss function calculation, fine-tune the model, and input the fine-tuned model into images and text to achieve multimodal machine translation; In step 4), a prompt template is designed, which connects the image encoding representation and the text encoding representation and is fed into the encoder module of the mBart model. Specifically: 401) Design a prompt template, encode the image as X, encode the source language text as Y, encode the target language text as Z, and encode the prompt template as "[prompt 1]", "[prompt 2]", and "[prompt 3]", and concatenate them to obtain the encoder input encoding representation "[prompt 1]:X[prompt 2]:Y"; 402) encoding the text portion using the position encoding module of the pre-trained model to obtain a text position encoding, and encoding each dimension of the image portion as if it were the same position to obtain an image position encoding; concatenating the text position encoding and the image position encoding, adding the result to the input encoding representation, and feeding the result to the encoder module of the pre-trained model to perform joint representation learning of the image and text; 403) Connect the prompt template and the target language text encoding to obtain the decoder input encoding representation "[prompt 3]:Z", which is sent to the decoder module of the pre-trained model for autoregressive generation of the target language.

2. The pre-training-based multimodal machine translation method according to claim 1, characterized in that: In step 1), training data is obtained from a public multimodal machine translation dataset. The image data is divided into fixed-size sub-images, each of which is called a patch. The dimensions are then adjusted to a suitable input size for the model. The encoding of each patch is obtained, that is, the image embedding vector, which is specifically: 101) Take input (batch, c, h, w), each patch resolution is p1*p2, and the specific process of dividing it into sub-pictures: First, each picture is divided into (h / p1)*(w / p2) small blocks, that is, from (batch, c, p1*(h / p1), p2*(w / p2)) to (batch, c, (h / p1)*(w / p2), p1*p2), and then converted into (batch, (h / p1)*(w / p2), p1*p2*c), which is equivalent to dividing it into (h / p1)*(w / p2) patches, and the dimension of each patch is p1*p2*c; the implementation of this process is calculated by the following formula: x=rearrange(img, ′ b c(h p1)(w p2)→b(h w)(p1 p2 c) ′ ) Among them, the rearrange function is an operator of the einops library, p1 and p2 are the number of pixels of the height and width of a patch, c is the number of channels, b is the number of batches, batch represents the batch of data during model training, h and w are the number of pixels of the height and width of the image respectively; 102) After dividing the sub-images, the image embedding vector is obtained and its dimension is adjusted. The image embedding vector is adjusted to the required length through a fully connected layer and used as the input of the model.

3. The pre-training-based multimodal machine translation method according to claim 1, wherein: In step 2), the Vision Transformer model is constructed to process the image embedding vector input to obtain the image encoding representation, specifically: 201) Use 2D sincos encoding to encode the position information. The encoding formula is: Among them, PE is a two-dimensional matrix, the rows represent sub-pictures, the columns represent position vectors; pos represents the position of the sub-picture in the picture; d model Represents the dimension of the position vector; i represents the position of the position vector; 202) The image embedding vector and the two-dimensional position encoding vector are concatenated to obtain an input vector, which is fed into the encoder structure of the VisionTransformer model to obtain the image encoding representation.

4. The pre-training-based multimodal machine translation method according to claim 1, wherein: In step 5), the pre-training parameters of the pre-training model are fixed, and the KL divergence between the text encoding representation and the image encoding representation is calculated. The loss L KL Compared with the original cross entropy loss L of the pre-training model CE Add them together to get the new loss L, and fine-tune the model; the loss formula is: L=L CE +αL KL Among them, p(·) represents the normalized text encoding representation, q(·) represents the normalized image encoding representation, and x i represents the i-th dimension of the distribution, L CE is the cross entropy loss, L KL is the KL divergence loss.

Images