Multi-modal neural machine translation method for scene self-adaption based on visual fantasy guidance

Abstract

The invention provides a multi-modal neural machine translation method for scene self-adaption based on visual fantasy guidance, and a scene self-adaption translation function is realized by fusing different modal features and continuously reducing differences among modals by utilizing visual fantasy as guidance. According to the method, an asymmetric alternative updating strategy is introduced, so that the modal gap is reduced, and the characterization capability of the multi-modal data is improved. And finally, establishing a cross-modal adaptive fusion mechanism to effectively fuse the two modals. According to the method, the multi-modal representation learning ability of the model in a complex and changeable image-text environment can be effectively improved, and it is ensured that the model can still maintain high machine translation performance even in a noisy data or plain text scene in the reasoning stage. According to the method, the optimal score superior to that of other prior art is achieved on most sets, it is proved that the method shows outstanding stability and consistency in various scenes, especially in a noise environment, and the machine translation performance is powerfully improved.

Description

Technical Field

[0001] The present invention provides a multimodal neural machine translation method for scene adaptation based on visual fantasy guidance, which has multimodal information to promote the machine translation performance of domain texts and belongs to the technical field of natural language processing. Background Art

[0002] Multimodal neural machine translation aims to translate sentences from a source language to a target language using additional input from other modalities, such as images. The use of supplementary modality input distinguishes multimodal neural machine translation from traditional neural machine translation. Many studies have shown that incorporating additional visual information can significantly improve machine translation performance. Most existing MNMT methods rely on large-scale, high-quality image-text data pairs. Training MNMT models typically requires both semantically consistent images and source-target text data as input. Unfortunately, due to the inherent modality gap between text and images, obtaining semantically aligned multimodal data is usually costly. In addition, semantically aligned images may not always match the source-target language.

[0003] Most previous MNMT studies assume that visual information is available. These methods focus on addressing the image-augmented machine translation challenge by improving machine translation performance by integrating visual information into text. While these methods are effective, they are limited by the availability of visual data in both training and inference steps. As a result, most of the above MNMT methods either suffer from significant performance degradation or fail to work at all. Building an adaptive machine translation framework that accommodates both image- and image-free scenarios remains unexplored.

[0004] Capturing the similarities between NMT and MNMT is crucial for developing an adaptive machine translation framework that can generalize to both image- and image-free scenarios. Assume that we can generate visual illusion images corresponding to the source language through cross-modal mapping. In this context, NMT can be viewed as a specific type of multimodal machine translation performed with the help of virtual visual illusions. Therefore, target sentence generation is a sequence-to-sequence task that takes the source language and its corresponding visual illusion as input within the MNMT-like framework. Therefore, this visual illusion can be viewed as the key similarity that connects the boundary between MNMT and NMT. ​​Using a suitable virtual visual generation module generated from the source text is reasonable for developing an adaptive machine translation framework that can adapt to both image- and image-free scenarios. Summary of the invention

[0005] In view of the above problems, the present invention proposes a multimodal neural machine translation method based on visual fantasy guidance for scene adaptation. The present invention is suitable for multimodal neural machine translation in scenes with and without images, and can effectively bridge the gap between text and visual modalities and improve the ability to represent modal features. The method proposed in the present invention can translate in both scenes with and without images, and effectively narrow the gap between text and visual modalities.

[0006] The technical solution of the present invention is: a multimodal neural machine translation method based on visual fantasy guidance for scene adaptation, and the specific steps of the multimodal neural machine translation method based on visual fantasy guidance for scene adaptation are as follows:

[0007] Step 1, obtaining bilingual parallel text data, image data corresponding to sentences, and obtaining visual fantasy image data;

[0008] Step 2, get the features of text, original image and visual fantasy image;

[0009] Step 3: Feed the features of text, original image and visual fantasy image into a module with multiple layers of Transformer.

[0010] Step 4: Construct a modal fusion method guided by visual illusions and impose consistency constraints in the visual space to improve the representation ability of multimodal data;

[0011] Step 5, construct an asymmetric alternating update method to narrow the modal gap;

[0012] Step 6: Establish a cross-modal adaptive fusion mechanism to effectively fuse text and visual modalities;

[0013] Step 7: Send the target sentence and the fused features to the traditional Transformer decoding end for decoding.

[0014] The specific steps of Step 1 are as follows:

[0015] Step 1.1, get the multimodal machine translation dataset Multi30K. The dataset used is obtained from a public dataset.

[0016] Step 1.2, use the pre-trained model of Stable Diffusion text to generate images, encode the text, and obtain the visual fantasy image with the highest relevance to the text;

[0017] Step 1.3: Perform byte pair encoding (BPE) preprocessing on the text to obtain the vocabulary and text data after subword segmentation.

[0018] The specific steps of Step 2 are as follows:

[0019] Step 2.1, use Resnet-101 to encode the original image and the visual fantasy image to obtain the corresponding original image features and visual fantasy image features;

[0020] Step 2.2, use the Embed & encoding module in Transformer to encode the text to obtain text representation;

[0021] Step 2.3. Use the Fairseq open source system to complete the design of a multimodal machine translation model based on text and visual modalities.

[0022] The specific steps of Step 2.3 are as follows:

[0023] Step 2.3.1, modify the data reading program based on the Fairseq open source system, and add the Transformer Encoder module of image features;

[0024] Step 2.3.2, add a visual encoder to the encoder part of the Transformer, and use the image features as Query, Key, and Value for attention mechanism calculation;

[0025] The specific steps of Step 3 are as follows:

[0026] Step 3.1, send the text features into the stacked multi-layer Transformer to fully represent the text features;

[0027] Step 3.2, send the original image features into the stacked multi-layer Transformer to fully characterize the original image features;

[0028] Step 3.3, send the visual illusion image features into the stacked multi-layer Transformer to fully represent the visual illusion features;

[0029] The specific steps of Step 4 are as follows:

[0030] Step 4.1, perform text-visual fantasy and original vision-visual fantasy interactions on the fully characterized text, original visual features and visual fantasy features obtained in Step 3;

[0031] Step 4.2, perform multi-layer iterative interaction on the interaction results of text-visual fantasy and original vision-visual fantasy;

[0032] Step 4.3, constrain the consistency of original visual features and visual illusion features to improve the adaptive ability of the model;

[0033] The specific steps of Step 4.1 are as follows:

[0034] Step 4.1.1, perform the following operations on the fully characterized text representation and visual fantasy image representation:

[0035] H l =β1·H x +(1-β1)·H v

[0036] H x =H x +H l

[0037] Among them, β1 is a learnable hyperparameter used to balance text features and visual illusion features, H x and H v They are text features and visual illusion features. l Hybrid multimodal features representing text and its corresponding visual illusion features.

[0038] The fully characterized original image representation and visual illusion representation are operated as follows:

[0039] H m =β2·H a +(1-β2)·H v

[0040] H a =H a +H m

[0041] Among them, β2 is a learnable hyperparameter used to balance the original image and visual illusion features, H a and H v are the source image and the visual illusion feature respectively. m It is the feature obtained by interacting the original image and its corresponding visual illusion feature.

[0042] The specific steps of Step 4.3 are as follows:

[0043] Step 4.3.1. Calculate the KL divergence between the original image features and the visual hallucination image features:

[0044]

[0045] Where N is the number of samples, H a and H v They are original image features and visual illusion image features respectively.

[0046] The specific steps of Step 5 are as follows:

[0047] Step 5.1, calculate the similarity between text-original image, text-visual illusion image and original image-visual illusion image respectively:

[0048]

[0049]

[0050]

[0051] Where D(,) represents the Kullback-Leibler (KL) divergence. and They represent the features of the original image, visual fantasy image and text obtained in the last layer respectively.

[0052] Step 5.2, calculate the minimum similarity between text-original image, text-visual illusion image, and original image-visual illusion image:

[0053] l min =min{l1,l2,l3}

[0054] Step 5.3. Calculate the asymmetric loss value between features:

[0055]

[0056] Where D(,) represents the Kullback-Leibler (KL) divergence and sg() represents the stopping gradient. and They represent the features of the source image, visual illusion and text from the last layer respectively.

[0057] The specific steps of Step 6 are as follows:

[0058] Step 6.1, respectively send text features and visual features into the cross-modal adaptive fusion method:

[0059] Ω=Sigmoid(W Ω H v +U Ω H x )

[0060] H g =H x +ΩHv

[0061] Where W Ω and U Ω is a trainable model parameter. The final output H g Directly fed into our target sentence decoder to predict the translation.

[0062] The final training goal is,

[0063] L=L trans +λloss img +γloss triple

[0064] where λ and γ are hyperparameters controlling the visual space consistency constraint loss and the asymmetric alternating update loss.

[0065] The beneficial effects of the present invention are:

[0066] 1. The present invention proposes a multimodal transformer with progressive fusion guided by visual illusion to solve the problem of machine translation with visual scene adaptation. The present invention can perform machine translation in both scenes with and without images, and by using asymmetric loss strategy and consistency constraints, it reduces the difference between modalities, enhances the representation ability of modalities, improves the adaptability of scenes, and further enhances the translation effect of text;

[0067] 2. The present invention constructs a cross-modal fusion strategy guided by vision, which effectively narrows the modal gap between text and vision and improves the representation ability of modal features;

[0068] 3. The present invention adopts an asymmetric loss update strategy, which effectively shortens the difference between modalities and improves the adaptability of the model.

[0069] 3. This paper adopts a cross-modal adaptive fusion strategy, guided by the text modality, to selectively fuse image features. Visual experiments have proved the effectiveness of this strategy.

[0070] 4. The method of the present invention is implemented based on the Fairseq open source system, which is relatively stable and has strong text processing capabilities, which provides a strong guarantee for multimodal machine translation;

[0071] 5. The method proposed in the present invention has achieved SOTA results on the test sets of multiple public data sets. Visual experiments and case analysis have demonstrated the effectiveness and superiority of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a flow chart of the present invention;

[0073] Figure 2 A detailed flowchart of the multimodal neural machine translation method for scene adaptation based on visual fantasy guidance proposed by the present invention; DETAILED DESCRIPTION

[0074] The present invention will be further described below in conjunction with the accompanying drawings.

[0075] Aiming at the problems faced by multimodal machine translation due to visual adaptation problems in multimodal neural machine translation, the inherent modality gap between text and image, and the high cost of collecting semantically aligned multimodal data, the present invention proposes a multimodal machine translation method based on visual scene adaptation guided by visual fantasy, which uses visual fantasy as an auxiliary input to bridge the gap between text and visual modalities, constructs a text-visual modality fusion guided by visual fantasy to enhance the representation ability of domain text, improves the scene adaptation ability of the model, uses an asymmetric alternating update mechanism to shorten the difference between modalities, and then uses a multimodal adaptive fusion mechanism to effectively fuse the two modalities.

[0076] like Figure 1-2 As shown, the present invention proposes a multimodal neural machine translation method for visual scene adaptation based on visual fantasy guidance, and the technical solution adopted is:

[0077] Step 1. Obtain multimodal machine translation data and use the Stable Diffusion text-image generation model to generate visual fantasy images based on the text; perform byte pair encoding (BPE) preprocessing on the text.

[0078] In Step 1, directly obtain the original data of the multimodal machine translation data Multi30K from the public download link of the Multi30K dataset, totaling 30,000 records. Divide the obtained data into training set, validation set, and test set.

[0079] Step 2: Use the convolutional neural network Resnet-101 to extract features from the images and visual illusion images in the multimodal machine translation dataset to obtain feature vectors of the two images, and preprocess the text at the same time;

[0080] In Step 2, the pre-trained convolutional neural network Resnet-101 is used to extract features from the images of the Multi-30k dataset and the visual illusion images generated by the Stable Diffusion model.

[0081] The specific steps of Step 2 are as follows:

[0082] Step 2.1, use Resnet-101 to encode the original image to obtain the corresponding image features;

[0083] Use Resnet-101 to encode the visual illusion image and obtain the corresponding image features;

[0084] Step 2.2, use the Embed & encoding module in Transformer to encode the text to obtain text representation;

[0085] Step 2.3. Use the Fairseq open source system to complete the design of a multimodal machine translation model based on text and visual modalities.

[0086] The specific steps of Step 2.3 are as follows:

[0087] Step 2.3.1, modify the data reading program based on the Fairseq open source system, and add the Transformer Encoder module of image features;

[0088] Step 2.3.2, add a visual encoder to the encoder part of the Transformer, and use the image features as Query, Key, and Value for attention mechanism calculation;

[0089] Step 3: Send the features of text, original image and visual fantasy image into the module with multi-layer Transformer for full representation. The specific steps are as follows:

[0090] Step 3.1, send the text features into the stacked multi-layer Transformer to fully represent the text features;

[0091] Step 3.2, send the original image features into the stacked multi-layer Transformer to fully characterize the original image features;

[0092] Step 3.3, send the visual illusion image features into the stacked multi-layer Transformer to fully represent the visual illusion features;

[0093] Step 4: Construct a modal fusion method guided by visual illusion and perform consistency constraints in the visual space to improve the representation ability of multimodal data. The specific steps of Step 4 are:

[0094] Step 4.1, perform text-visual fantasy and original vision-visual fantasy interactions on the fully characterized text, original visual features and visual fantasy features obtained in Step 3;

[0095] Step 4.2, perform multi-layer iterative interaction on the interaction results of text-visual fantasy and original vision-visual fantasy;

[0096] Step 4.3, constrain the consistency of the original visual features and the visual illusion features;

[0097] The specific steps of Step 4.1 are as follows:

[0098] Step 4.1.1, perform the following operations on the fully characterized text representation and visual fantasy image representation:

[0099] H l =β1·H x +(1-β1)·H v

[0100] H x =H x +H l

[0101] Among them, β1 is a learnable hyperparameter used to balance text features and visual illusion features, H x and H v They are text features and visual illusion features. l Hybrid multimodal features representing text and its corresponding visual illusion features.

[0102] The fully characterized original image representation and visual illusion representation are operated as follows:

[0103] H m =β2·H a +(1-β2)·H v

[0104] H a =H a +H m

[0105] Among them, β2 is a learnable hyperparameter used to balance the original image and visual illusion features, H a and H v are the source image and the visual illusion feature respectively. m It is the feature obtained by interacting the original image and its corresponding visual illusion feature.

[0106] The specific steps of Step 4.3 are as follows:

[0107] Step 4.3.1. Calculate the KL divergence (Kullback-Leibler Divergence) between the original image features and the visual hallucination image features:

[0108]

[0109] Where N is the number of samples, H a and H v They are original image features and visual illusion image features respectively.

[0110] Step 5: construct an asymmetric alternating updating method to narrow the modal gap; the specific steps of Step 5 are as follows:

[0111] Step 5.1, calculate the similarity between text-original image, text-visual illusion image and original image-visual illusion image respectively:

[0112]

[0113]

[0114]

[0115] Where D(,) represents the Kullback-Leibler (KL) divergence. and They represent the features of the original image, visual fantasy image and text obtained in the last layer respectively.

[0116] Step 5.2, calculate the minimum similarity between text-original image, text-visual illusion image, and original image-visual illusion image:

[0117] l min =min{l1,l2,l3}

[0118] Step 5.3. Calculate the asymmetric loss value between features:

[0119]

[0120] where D(,) represents the Kullback-Leibler (KL) divergence and sgO represents the stopping gradient. and They represent the features of the source image, visual illusion and text from the last layer respectively.

[0121] Step 6: Establish a cross-modal adaptive fusion mechanism to effectively fuse text and visual modalities;

[0122] The specific steps of Step 6 are as follows:

[0123] Step 6.1, respectively send text features and visual features into the cross-modal adaptive fusion method:

[0124] Ω=Sigmoid(W Ω H v +U Ω H x )

[0125] H g =H x +ΩH v

[0126] Where W Ω and U Ω is a trainable model parameter. The final output H g Directly fed into our target sentence decoder to predict the translation.

[0127] The final training goal is,

[0128] L=L trans +λloss img +γloss triple

[0129] where λ and γ are hyperparameters controlling the visual space consistency constraint loss and the asymmetric alternating update loss.

[0130] Step 7: Send the target sentence and the fused features to the traditional Transformer decoding end for decoding.

[0131] In order to prove the effectiveness of the proposed method, a large number of experiments were conducted on the Multi-30k dataset, where the details of the Multi-30k dataset are shown in Table 1. In order to accurately evaluate the performance of the method in the experiment, BLUE and METEOR are used as evaluation indicators. These two indicators can objectively and accurately reflect the effect of machine translation. The experimental results are shown in Tables 2 and 3.

[0132] Table 1 Multi-30k dataset details

[0133]

[0134]

[0135] Table 2 Comparison of the model corresponding to the proposed method and the existing SOTA model in the English-German task

[0136]

[0137] For the ablation experiment of Multi30K, the results are shown in 3;

[0138] Table 3 Performance of multimodal machine translation models on the Multi30K test set

[0139]

[0140] The results in Tables 2 and 3 both show that the multimodal neural machine translation method based on visual illusion guidance for scene adaptation proposed in the present invention has achieved SOTA results on the Multi-30k dataset. The results in Table 2 show that this method still has strong robustness and high translation performance when facing multiple scenes, and exceeds the existing SOTA methods. Through the performance comparison on the test set, it can be found that the progressive fusion strategy guided by visual illusion can greatly improve the translation performance and multi-scene adaptability of the model; Table 3 shows that after removing the model corresponding to the method of the present invention, the model performance has been greatly degraded, which proves the effectiveness and superiority of this method;

[0141] This paper proposes a visual illusion-guided scene adaptation strategy to address the visual adaptation challenge in MNMT. First, cross-modal mixing is introduced in the text with the help of modality-independent visual illusions. This mechanism can promote the model to learn multimodal representations by hierarchically integrating information between images and texts. In addition, consistency constraints are added between the original visual information and the virtual visual illusions, thereby enhancing visual adaptability in various noisy scenes. This method gradually bridges the modality gap with the support of visual illusions by adopting an asymmetric alternating update strategy. This strategy can be applied to various scenarios and realizes a higher-performance multimodal machine translation system for multiple scenarios.

[0142] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A multimodal neural machine translation method based on visual fantasy guidance for scene adaptation, characterized in that: The following steps are involved: Step 1, obtaining bilingual parallel text data, image data corresponding to sentences, and obtaining visual fantasy image data; Step 2, obtain the features of text, original image and visual fantasy image; Step 3: The features of text, original image and visual fantasy image are fed into a module with multi-layer Transformer for full representation; Step 4: Construct a modal fusion method guided by visual illusion and impose consistency constraints in the visual space; Step 5, construct an asymmetric alternating update method to narrow the modal gap; Step 6: Establish a cross-modal adaptive fusion mechanism to effectively fuse text and visual modalities; Step 7: Send the target sentence and the fused features to the traditional Transformer decoding end for decoding.

2. The method for scene-adaptive multimodal neural machine translation based on visual fantasy guidance as claimed in claim 1, characterized in that: The Step 1 includes: Step 1.1, obtain the multimodal machine translation dataset Multi30K, the dataset used is obtained from the public dataset; Step 1.2, use the pre-trained model of Stable Diffusion text to generate images, encode the text, and obtain the visual fantasy image with the highest relevance to the text; Step 1.3: Perform BPE preprocessing on the text to obtain the vocabulary and text data after sub-word segmentation.

3. The method for scene-adaptive multimodal neural machine translation based on visual fantasy guidance as claimed in claim 1, characterized in that: The Step 2 includes: Step 2.1, use Resnet-101 to encode the original image and the visual fantasy image to obtain the corresponding original image features and visual fantasy image features; Step 2.2, use the Embed & encoding module in Transformer to encode the text to obtain text representation; Step 2.

3. Use the Fairseq open source system to complete the design of a multimodal machine translation model based on text and visual modalities.

4. The method for scene-adaptive multimodal neural machine translation based on visual fantasy guidance as claimed in claim 3, characterized in that: The specific steps of Step 2.3 are as follows: Step 2.3.1, modify the data reading program based on the Fairseq open source system, and add the Transformer Encoder module of image features; Step 2.3.

2. Add a visual encoder to the encoder part of the Transformer and use the image features as Query, Key, and Value for attention mechanism calculation.

5. The method for scene-adaptive multimodal neural machine translation based on visual fantasy guidance as claimed in claim 1, characterized in that: The specific steps of Step 3 are as follows: Step 3.1, send the text features into the stacked multi-layer Transformer to fully represent the text features; Step 3.2, send the original image features into the stacked multi-layer Transformer to fully characterize the original image features; Step 3.3, send the visual fantasy image features into the stacked multi-layer Transformer to fully represent the visual fantasy features.

6. The method for scene-adaptive multimodal neural machine translation based on visual fantasy guidance as claimed in claim 1, characterized in that: The specific steps of Step 4 are as follows: Step 4.1, perform text-visual fantasy and original vision-visual fantasy interactions on the fully characterized text, original visual features and visual fantasy features obtained in Step 3; Step 4.2, perform multi-layer iterative interaction on the interaction results of text-visual fantasy and original vision-visual fantasy; Step 4.3: Constrain the consistency of the original visual features and the visual illusion features.

7. The method for scene-adaptive multimodal neural machine translation based on visual fantasy guidance as claimed in claim 6, characterized in that: The specific steps of Step 4.1 are as follows: Step 4.1.1, perform the following operations on the fully characterized text representation and visual fantasy image representation: H l =β1·H x +(1-β1)·H v H x =H x +H l Among them, β1 is a learnable hyperparameter used to balance text features and visual illusion features, H x and H v They are text features and visual illusion features, H l Hybrid multimodal features representing text and its corresponding visual illusion features; The fully characterized original image representation and visual illusion representation are operated as follows: H m =β2·H a +(1-β2)·H v H a =H a +H m Among them, β2 is a learnable hyperparameter used to balance the original image and visual illusion features, H a and H v are the source image and visual illusion features, respectively, m It is the feature obtained by interacting the original image and its corresponding visual illusion feature.

8. The method for scene-adaptive multimodal neural machine translation based on visual fantasy guidance as claimed in claim 6, characterized in that: The specific steps of Step 4.3 are as follows: Step 4.3.

1. Calculate the KL divergence between the original image features and the visual hallucination image features: Where N is the number of samples, H a and H v They are original image features and visual illusion image features respectively.

9. The method for scene-adaptive multimodal neural machine translation based on visual fantasy guidance as claimed in claim 1, characterized in that: The specific steps of Step 5 are as follows: Step 5.1, calculate the similarity between text-original image, text-visual illusion image and original image-visual illusion image respectively: Where D(,) represents the KL divergence, and Respectively represent the features of the original image, visual illusion image and text obtained in the last layer; Step 5.2, calculate the minimum similarity between text-original image, text-visual illusion image, and original image-visual illusion image: <h2 style=";text-align:left;direction:ltr">l<h2 style=";text-align:left;direction:ltr"> min <h2 style=";text-align:left;direction:ltr"> = min{l1, l2, l3} Step 5.

3. Calculate the asymmetric loss value between features: Where D(,) represents KL divergence, sg() represents the stop gradient, and They represent the features of the source image, visual illusion and text from the last layer respectively.

10. The method for scene-adaptive multimodal neural machine translation based on visual illusion guidance according to claim 1, characterized in that: The specific steps of Step 6 are as follows: Step 6.1: Send text features and visual features into the cross-modal adaptive fusion mechanism respectively: Ω=Sigmoid(W Ω H v +U Ω H x ) H g =H x +ΩH v Where W Ω and U Ω is a trainable model parameter, and the final output is H g Directly input into the target sentence decoder to predict the translation; The final training goal is, L=L trans +λloss img +γloss tripie where λ and γ are hyperparameters controlling the visual space consistency constraint loss and the asymmetric alternating update loss.

Images