A multi-modal neural machine translation method based on visual illusion guidance for scene adaptation

Abstract

The application provides a multi-modal neural machine translation method based on visual imagination guidance for scene adaptation, which realizes scene adaptive translation function by fusing different modal characteristics and continuously reducing the difference between modes by using visual imagination as guidance. The application introduces an asymmetric alternating update strategy to reduce the modal gap and improve the representation ability of multi-modal data. Finally, a cross-modal adaptive fusion mechanism is established to effectively fuse the two modes. The application can effectively improve the multi-modal representation learning ability of the model in a complex and variable image-text environment, ensuring that it can maintain high machine translation performance even in the reasoning stage when facing noisy data or pure text scenes. The application has achieved optimal scores better than other prior art on most sets, proving its outstanding stability and consistency in various scenes, especially in noisy environments, and effectively improving the performance of machine translation.

Description

TECHNICAL FIELD

[0001] The application provides a multi-modal neural machine translation method based on visual hallucination guidance for scene adaptation, which has multi-modal information to promote the machine translation performance of field text, and belongs to the technical field of natural language processing. BACKGROUND

[0002] Multi-modal neural machine translation aims to translate sentences from a source language into a target language using additional input from other modalities, such as images. Multi-modal neural machine translation is distinguished from traditional neural machine translation using supplementary modal input. Many studies have shown that adding extra visual information can significantly improve machine translation performance. Most existing MNMT methods rely on large-scale, high-quality image-text data pairs. Training an MNMT model usually requires providing semantically aligned images and source language-target text data as input at the same time. Unfortunately, due to the inherent modality gap between text and images, it is often costly to obtain semantically aligned multi-modal data. In addition, semantically aligned images may not always be available to match the source language-target language.

[0003] Most previous MNMT research has assumed that visual information is available. These methods focus on improving machine translation performance by integrating visual information into text, thereby addressing the image-enhanced machine translation challenge. While these methods are effective, they are limited by the availability of visual data in the training and inference steps. Therefore, most of the above MNMT methods either perform significantly worse or completely fail to work. Building an adaptive machine translation framework that simultaneously adapts to image and no-image scenarios has not been explored.

[0004] Capturing the similarities between NMT and MNMT is crucial for developing an adaptive machine translation framework that can be generalized to image and no-image scenarios. Suppose we can generate a visual hallucination image corresponding to the source language through cross-modal mapping. In this case, NMT can be viewed as a specific type of multi-modal machine translation that is carried out with the help of virtual visual hallucination. Therefore, target sentence generation is a sequence-to-sequence task that takes the source language and its corresponding visual hallucination as input within the MNMT framework. Therefore, this visual hallucination can be seen as a key similarity that bridges the boundaries between MNMT and NMT. It is reasonable to use a suitable virtual visual generation module generated from source text to develop an adaptive machine translation framework that adapts to image and no-image scenarios. SUMMARY

[0005] To solve the above problems, the application provides a multi-modal neural machine translation method based on visual fantasy guidance for scene adaptation, which is suitable for multi-modal neural machine translation in image and non-image scenes, can effectively bridge the gap between text and visual modalities, and improve the modal feature representation capability.

[0006] The technical solution of the application is a multi-modal neural machine translation method based on visual fantasy guidance for scene adaptation, and the specific steps of the multi-modal neural machine translation method based on visual fantasy guidance for scene adaptation are as follows:

[0007] Step 1, obtaining bilingual parallel text data, image data corresponding to the sentence, and obtaining visual fantasy image data;

[0008] Step 2, obtaining features of text, original image and visual fantasy image;

[0009] Step 3, sending the features of text, original image and visual fantasy image into a module with multiple layers of Transformer;

[0010] Step 4, constructing a modal fusion method guided by visual fantasy, and performing consistency constraint on the visual space to improve the representation capability of multi-modal data;

[0011] Step 5, constructing an asymmetric alternating update method for reducing the modal gap;

[0012] Step 6, establishing a cross-modal adaptive fusion mechanism to effectively fuse the two modalities of text and vision;

[0013] Step 7, sending the target sentence and the fused features to the traditional Transformer decoding end for decoding.

[0014] The specific steps of the step Step 1 are as follows:

[0015] Step 1.1, obtaining a multi-modal machine translation dataset Multi30K. The used dataset is obtained from a public dataset.

[0016] Step 1.2, using a Stable Diffusion text-to-image pre-training model to encode the text and obtain the visual fantasy image with the highest relevance to the text;

[0017] Step 1.3, performing byte pair encoding (BPE) preprocessing operation on the text to obtain a word table and text data after subword segmentation.

[0018] The specific steps of the step Step2 are as follows:

[0019] Step2.1, using Resnet-101 to encode the original image and the visual fantasy image to obtain corresponding original image features and visual fantasy image features;

[0020] Step2.2, using the Embed&encoding module in the Transformer to encode the text to obtain the text representation;

[0021] Step2.3, using the Fairseq open source system to complete the design of the multi-modal machine translation model based on the text and the visual modal.

[0022] The specific steps of the step Step2.3 are as follows:

[0023] Step2.3.1, modifying the Fairseq open source system, modifying the data reading program, and adding the Transformer Encoder module of the image feature;

[0024] Step2.3.2, adding a visual encoder to the encoder part of the Transformer, and taking the image feature as Query, Key and Value to perform attention mechanism calculation;

[0025] The specific steps of the step Step3 are as follows:

[0026] Step3.1, sending the text feature into the stacked multi-layer Transformer to make the text feature fully represented;

[0027] Step3.2, sending the original image feature into the stacked multi-layer Transformer to make the original image feature fully represented;

[0028] Step3.3, sending the visual fantasy image feature into the stacked multi-layer Transformer to make the visual fantasy feature fully represented;

[0029] The specific steps of the step Step4 are as follows:

[0030] Step4.1, the text, the original visual feature and the visual fantasy feature obtained by Step3 are respectively subjected to text-visual fantasy and original visual-visual fantasy interaction;

[0031] Step4.2, the interaction results of text-visual fantasy and original visual-visual fantasy are subjected to multi-layer iteration interaction;

[0032] Step 4.3: Apply consistency constraints to the original visual features and visual illusion features to improve the model's adaptability;

[0033] The specific steps of Step 4.1 are as follows:

[0034] Step 4.1.1: Perform the following operations on the fully represented text representation and visual fantasy image representation:

[0035] H l =β1·H x +(1-β1)·H v

[0036] H x =H x +H l

[0037] Where β1 is a learnable hyperparameter used to balance textual features and visual illusion features, H x and H v These are textual features and visual illusion features, respectively. H l A mixed multimodal feature representing text and its corresponding visual fantasy features.

[0038] The fully characterized original image representation and visual illusion representation are then subjected to the following operations:

[0039] H m =β2·H a +(1-β2)·H v

[0040] H a =H a +H m

[0041] Where β2 is a learnable hyperparameter used to balance the features of the original image and the visual illusion, H a and H v These are the source image and visual illusion features, respectively. H m It is a feature obtained by interacting with the original image and its corresponding visual illusion features.

[0042] The specific steps of Step 4.3 are as follows:

[0043] Step 4.3.1: Calculate the KL divergence between the original image features and the visual illusion image features:

[0044]

[0045] where N denotes the number of samples, H a and H v are the features of the original image and the visual hallucination image, respectively.

[0046] The specific steps of the step Step5 are as follows:

[0047] Step5.1, the similarity between the text-original image, text-visual hallucination image and original image-visual hallucination image is calculated respectively:

[0048]

[0049]

[0050]

[0051] where D(,) denotes the Kullback-Leibler (KL) divergence. and denote the features of the original image, the visual hallucination image and the text obtained by the last layer, respectively.

[0052] Step5.2, the minimum similarity between the text-original image, text-visual hallucination image and original image-visual hallucination image is calculated:

[0053] l min =min{l1, l2, l3}

[0054] Step5.3, the asymmetric loss value between the features is calculated:

[0055]

[0056] where D(,) denotes the Kullback-Leibler (KL) divergence, and sg() denotes the stop gradient. and denote the features of the original image, the visual hallucination and the text obtained by the last layer, respectively.

[0057] The specific steps of the step Step6 are as follows:

[0058] Step6.1, the text features and visual features are respectively sent into the cross-modal adaptive fusion method:

[0059] Ω=Sigmoid(W Ω H v +U Ω H x )

[0060] H g =H x +ΩHv

[0061] where W Ω and U Ω are trainable model parameters. The final output H g is directly input into our target sentence decoder to predict the translation.

[0062] The final training objective is,

[0063] L = L trans + lambda * loss img + gamma * loss triple

[0064] where lambda and gamma are hyperparameters that control the visual space consistency constraint loss and the asymmetric alternating update loss.

[0065] The beneficial effects of the present application are:

[0066] 1. The present application proposes a multi-modal transformer with visual fantasy guidance and progressive fusion to solve the problem of machine translation in visual scene adaptation. The present application can perform machine translation in both image and non-image scenes, and by using an asymmetric loss strategy and consistency constraints, it reduces the inter-modal difference, enhances the representation ability of the modal, improves the scene adaptation ability, and further enhances the translation effect of the text.

[0067] 2. The present application constructs a visual-guided cross-modal fusion strategy, which effectively narrows the modal gap between text and vision and improves the representation ability of modal features.

[0068] 3. The present application adopts an asymmetric loss update strategy, which effectively shortens the inter-modal difference and improves the model's adaptive ability.

[0069] 3. The present application adopts a cross-modal adaptive fusion strategy to selectively fuse image features guided by the text modal, and visual experiments prove the effectiveness of this strategy.

[0070] 4. The method of the present application is implemented based on the Fairseq open source system, which is relatively stable and has strong text processing ability, providing a strong guarantee for multi-modal machine translation.

[0071] 5. The method proposed by the present application achieves SOTA effect on multiple public dataset test sets, and visual experiments and case analysis prove the effectiveness and superiority of the method. BRIEF DESCRIPTION OF DRAWINGS

[0072] Fig. 1 is the flowchart in the present application;

[0073] Fig. 2 The detailed flowchart of the multi-modal neural machine translation method based on visual imagination guidance for scene adaptation proposed by the present application is shown in the figure; DETAILED DESCRIPTION

[0074] The present application will be further described below in conjunction with the accompanying drawings.

[0075] The present application aims at the problems of multi-modal neural machine translation, such as visual adaptation, inherent modal gap between text and image, high cost of collecting multi-modal data with semantic alignment, etc. A multi-modal machine translation method based on visual imagination guidance for visual scene adaptation is proposed, which uses visual imagination as auxiliary input to make up for the gap between text and visual modal, constructs text-visual modal fusion guided by visual imagination to enhance the representation ability of domain text, improves the scene adaptation ability of the model, uses an asymmetric alternating update mechanism to shorten the difference between the two modalities, and then uses a multi-modal adaptive fusion mechanism to effectively fuse the two modalities.

[0076] As shown in Figs. 1-2 The technical scheme adopted by the multi-modal neural machine translation method based on visual imagination guidance for visual scene adaptation proposed by the present application is as follows:

[0077] Step 1, obtain multi-modal machine translation data and generate visual imagination images from text using a Stable Diffusion text-image generation model; perform byte pair encoding (BPE) preprocessing operation on the text.

[0078] In Step 1, the original data of the multi-modal machine translation data Multi30K is directly obtained from the download link of the Multi30K data set, a total of 30000. The obtained data is divided into training set, validation set and test set.

[0079] Step 2, use a convolutional neural network Resnet-101 to extract features from the images and visual imagination images in the multi-modal machine translation data set, thereby obtaining the feature vectors of the two kinds of images, and simultaneously pre-process the text;

[0080] In Step 2, the pre-trained convolutional neural network Resnet-101 is used to extract features from the images of the Multi-30k data set and the visual imagination images generated by the Stable Diffusion model.

[0081] The specific steps of Step 2 are as follows:

[0082] Step2.1, encode the original image using Resnet-101 to obtain the corresponding image features;

[0083] Encode the visual fantasy image using Resnet-101 to obtain the corresponding image features;

[0084] Step2.2, use the Embed&encoding module in Transformer to encode the text to obtain the text representation;

[0085] Step2.3, use the Fairseq open source system to complete the design of the multi-modal machine translation model based on text and visual modal.

[0086] The specific steps of the step Step2.3 are as follows:

[0087] Step2.3.1, modify the data reading program based on the Fairseq open source system, and add the Transformer Encoder module of image features;

[0088] Step2.3.2, add a visual encoder to the encoder part of the Transformer, and use the image features as Query, Key and Value for attention mechanism calculation;

[0089] Step3, send the text, original image and visual fantasy image features into the module with multi-layer Transformer for sufficient representation, and the specific steps are as follows:

[0090] Step3.1, send the text features into the stacked multi-layer Transformer to make the text features fully represented;

[0091] Step3.2, send the original image features into the stacked multi-layer Transformer to make the original image features fully represented;

[0092] Step3.3, send the visual fantasy image features into the stacked multi-layer Transformer to make the visual fantasy features fully represented;

[0093] Step4, construct a modal fusion method guided by visual fantasy, and perform consistency constraint in the visual space to improve the representation ability of multi-modal data, and the specific steps of Step4 are as follows:

[0094] Step4.1, interact the text-visual fantasy and original visual-visual fantasy features obtained in Step3 respectively;

[0095] Step4.2、the text-visual fantasy, the original visual-visual fantasy interaction result is carried out multi-layer iteration interaction;

[0096] Step4.3、the original visual feature and visual fantasy feature are carried out consistency constraint;

[0097] The specific steps of the step Step4.1 are as follows:

[0098] Step4.1.1、the text representation after being fully represented and the visual fantasy image representation are carried out as follows:

[0099] H l =β1·H x +(1-β1)·H v

[0100] H x =H x +H l

[0101] Wherein, β1 is a learnable hyperparameter, for balancing text feature and visual fantasy feature, H x And H v Text feature and visual fantasy feature respectively.H l H m It is the mixed multi-modal feature of text and its corresponding visual fantasy feature.

[0102] The original image representation after being fully represented and the visual fantasy representation are carried out as follows:

[0103] H m =β2·H a +(1-β2)·H v

[0104] H a =H a +H m

[0105] Wherein, β2 is a learnable hyperparameter, for balancing original image and visual fantasy feature, H a And H v Source image and visual fantasy feature respectively.H m It is the feature of the interactive original image and its corresponding visual fantasy feature.

[0106] The specific steps of the step Step4.3 are as follows:

[0107] Step4.3.1、the KL divergence (Kullback-Leibler Divergence) between the original image feature and the visual fantasy image feature is calculated:

[0108]

[0109] Where N represents the number of samples, H a and H v These are the original image features and the visual illusion image features, respectively.

[0110] Step 5: Construct an asymmetric alternating update method to reduce the modal gap; the specific steps of Step 5 are as follows:

[0111] Step 5.1: Calculate the similarity between text and original image, text and visual illusion image, and original image and visual illusion image respectively:

[0112]

[0113]

[0114]

[0115] Where D(,) represents the Kullback-Leibler (KL) divergence. and These represent the features of the original image, visual illusion image, and text obtained from the last layer, respectively.

[0116] Step 5.2: Calculate the minimum similarity between text and original image, text and visual illusion image, and original image and visual illusion image:

[0117] l min =min{l1, l2, l3}

[0118] Step 5.3: Calculate the asymmetric loss value between features:

[0119]

[0120] Where D(,) represents the Kullback-Leibler (KL) divergence, and sgO represents the stopping gradient. and These represent the features of the source image, visual illusion, and text that emerge from the last layer, respectively.

[0121] Step 6: Establish a cross-modal adaptive fusion mechanism to effectively fuse text and visual modalities;

[0122] The specific steps of Step 6 are as follows:

[0123] Step 6.1: Feed the text features and visual features into the cross-modal adaptive fusion method respectively:

[0124] Ω = Sigmoid(W) Ω H v +U Ω H x )

[0125] H g =H x +ΩH v

[0126] Among them W Ω and U Ω These are the trainable model parameters. The final output H g We input the data directly into our target sentence decoder to predict the translation.

[0127] The ultimate training goal is,

[0128] L = L trans +λloss img +γloss triple

[0129] λ and γ are hyperparameters that control the visual spatial consistency constraint loss and the asymmetric alternating update loss.

[0130] Step 7: Send the target sentence and the fused features to the traditional Transformer decoder for decoding.

[0131] To demonstrate the effectiveness of the proposed method, extensive experiments were conducted on the Multi-30k dataset, details of which are shown in Table 1. In order to accurately evaluate the method's performance, BLUE and METEOR were used as evaluation metrics. These two metrics can objectively and accurately reflect the effectiveness of machine translation. The experimental results are shown in Tables 2 and 3.

[0132] Table 1. Details of the Multi-30k dataset

[0133]

[0134]

[0135] Table 2 compares the model corresponding to the method of this invention with the existing state-of-the-art (SOTA) model in the English-German task.

[0136]

[0137] The results of the ablation experiment for Multi30K are shown in Figure 3.

[0138] Table 3 Performance of the multimodal machine translation model on the Multi30K test set

[0139]

[0140] The results of Table 2 and Table 3 both show that the multi-modal neural machine translation method based on visual illusion guidance for scene adaptation proposed in the application achieves the SOTA effect on the Multi-30k data set, and the results of Table 2 show that when facing multiple scenes, the method still has strong robustness and high translation performance, and exceeds the existing SOTA method. Through the performance comparison on the test set, it can be found that the progressive fusion strategy guided by visual illusion can greatly improve the translation performance of the model and the adaptability to multiple scenes; Table 3 shows that the performance of the model is greatly degraded after removing the model corresponding to the method of the application, which proves the effectiveness and superiority of the method;

[0141] The scene adaptation strategy guided by visual illusion proposed in the application solves the visual adaptation challenge in MNMT. First, cross-modal mixing is introduced in the text with the aid of modality-independent visual illusion. This mechanism can promote the model to learn multi-modal representation by hierarchically integrating information between images and text. In addition, a consistency constraint is added between the original visual information and the virtual visual illusion, thereby enhancing the visual adaptability in various noisy scenes. The method gradually bridges the modal gap with the support of visual illusion by adopting an asymmetric alternating update strategy. This strategy can be applied to various scenes, and a multi-modal machine translation system with higher performance for multiple scenes is realized.

[0142] The specific embodiments of the application are described in detail above in combination with the drawings, but the application is not limited to the above-mentioned embodiments, and various changes can be made within the knowledge possessed by those skilled in the art without departing from the purpose of the application.

Claims

1. A multimodal neural machine translation method based on visual imagination guidance for scene adaptation, characterized in that, Includes the following steps: Step 1: Obtain bilingual parallel text data, raw image data, and visual illusion image data; Step 2: Extract features from the text, original image, and visual illusion image; Step 3: Feed the features of the text, original image, and visual illusion image into a module with multiple Transformers for full representation; Step 4: Construct a modal fusion method guided by visual imagination and impose consistency constraints in visual space; Step 5: Construct an asymmetric alternating update method to reduce the modal gap; Step 6: Establish a cross-modal adaptive fusion mechanism to effectively fuse text and visual modalities; Step 7: Send the target sentence and the fused features to the traditional Transformer decoder for decoding; The specific steps of Step 4 are as follows: Step 4.1: Perform text-visual fantasy and original image-visual fantasy interactions on the fully characterized text, original image features and visual fantasy image features obtained in Step 3 respectively; Step 4.2: Iterate the interaction results of text-visual fantasy and original image-visual fantasy in multiple layers; Step 4.3: Apply consistency constraints to the original image features and the visual illusion image features; The specific steps of Step 5 are as follows: Step 5.1: Calculate the similarity between text and original image, text and visual illusion image, and original image and visual illusion image respectively: ； ； ； in Denotes KL divergence, , and These represent the features of the original image, visual illusion image, and text obtained from the last layer, respectively. Step 5.2: Calculate the minimum similarity between text and original image, text and visual illusion image, and original image and visual illusion image: ； Step 5.3: Calculate the asymmetric loss value between features: ； in Denotes KL divergence, This indicates that the gradient has stopped. , and These represent the features of the original image, visual illusion image, and text that emerge from the last layer, respectively.

2. The multimodal neural machine translation method based on visual imagination guidance for scene adaptation as described in claim 1, characterized in that: Step 1 includes: Step 1.1: Obtain the Multi30K multimodal machine translation dataset. The dataset used is obtained from a public dataset. Step 1.2: Use the Stable Diffusion text-to-image pre-trained model to encode the text and obtain the visual fantasy image with the highest relevance to the text. Step 1.3: Perform BPE preprocessing on the text to obtain the vocabulary and the text data after sub-word segmentation.

3. The multimodal neural machine translation method based on visual imagination guidance for scene adaptation as described in claim 1, characterized in that: Step 2 includes: Step 2.1: Use ResNet-101 to encode the original image and the visual illusion image to obtain the corresponding original image features and visual illusion image features; Step 2.2: Use the Embed & encoding module in Transformer to encode the text and obtain text features; Step 2.3: Use the Fairseq open-source system to complete the design of a multimodal machine translation model based on text and visual modalities.

4. The multimodal neural machine translation method based on visual imagination guidance for scene adaptation as described in claim 3, characterized in that: The specific steps of Step 2.3 are as follows: Step 2.3.1: Modify the Fairseq open-source system by modifying the data input program and adding a Transformer Encoder module for image features; Step 2.3.2: Add a visual encoder to the encoder part of the Transformer, and use image features as Query, Key and Value for attention mechanism calculation.

5. The multimodal neural machine translation method based on visual imagination guidance for scene adaptation as described in claim 1, characterized in that: The specific steps of Step 3 are as follows: Step 3.1: Feed the text features into a stacked multi-layer Transformer to fully represent the text features; Step 3.2: Feed the original image features into a stacked multi-layer Transformer to fully represent the original image features; Step 3.3: Feed the visual fantasy image features into a stacked multi-layer Transformer to fully represent the visual fantasy image features.

6. The multimodal neural machine translation method based on visual imagination guidance for scene adaptation as described in claim 1, characterized in that: The specific steps of Step 4.1 are as follows: Step 4.1.1: Perform the following operations on the fully represented text representation and visual fantasy image representation: ； ； in, It is a learnable hyperparameter used to balance text features and visual illusion image features. and These are text features and visual fantasy image features. Hybrid multimodal features representing text and its corresponding visual fantasy image features; The fully characterized original image representation and the visual illusion image representation are then subjected to the following operations: ； ； in, It is a learnable hyperparameter used to balance features of the original image and features of the visual illusion image. and These are the original image features and the visual illusion image features, It is a feature obtained by interacting with the original image features and their corresponding visual fantasy image features.

7. The multimodal neural machine translation method based on visual imagination guidance for scene adaptation as described in claim 6, characterized in that: The specific steps of Step 4.3 are as follows: Step 4.3.1: Calculate the KL divergence between the original image features and the visual illusion image features: ； in, Indicates the number of samples. and These are the original image features and the visual illusion image features, respectively.

8. The multimodal neural machine translation method based on visual imagination guidance for scene adaptation as described in claim 7, characterized in that: The specific steps of Step 6 are as follows: Step 6.1: Feed the text features and visual fantasy image features into the cross-modal adaptive fusion mechanism respectively: ； ； in and These are the trainable model parameters, and the final output. The text is directly input into the target sentence decoder to predict the translation. The ultimate training goal is to ； in and It is a hyperparameter that controls the visual spatial consistency constraint loss and the asymmetric alternating update loss.

Images