Multimodal neural machine translation method based on double-level fusion of image-text features

Abstract

The present application relates to a multi-modal neural machine translation method based on image-text feature two-stage fusion, and belongs to the technical field of natural language processing.The present application comprises: using text features and image features of aligned input at the encoding end, then generating context representation of text from different modes through a self-attention mechanism, and screening image semantic representation using an Attention+Gating mechanism;finally, a two-stage fusion mechanism is used to realize fusion and alignment of images and text, and the output of the fourth layer (the last layer) is used as the hidden layer output of the encoder, and then a Transformer decoder is used to realize word-by-word translation of sentences.The present application uses a Multi30K data set to train the model.The experimental results show that, on two language pairs and three test sets, the experimental results of the present application reach the best values at present.

Description

Technical Field

[0001] This invention relates to a multimodal neural machine translation method based on two-level fusion of image and text features, belonging to the field of natural language processing technology. Background Technology

[0002] Multimodal neural machine translation (MMT) (Calixto et al., 2017; Huang et al., 2016; Barrault et al., 2018) aims to improve the performance of traditional plain text neural machine translation (NMT) by extracting information from other modalities. Similar to traditional plain text neural machine translation, MMT tasks have significant research value and wide applications in the field of machine translation, such as translating multimedia news and online products (Zhou et al., 2018). However, compared to traditional neural machine translation, MMT is better suited to handling corpora with semantic ambiguity and sentence incompleteness (Huang et al., 2016; Li et al., 2022; Frank et al., 2018).

[0003] Images can serve as parallel corpora to text, supplementing or enhancing the semantic information of sentences, such as... Figure 4 In the example, the source language sentence is "a gray and white dog jumping over standing water in the sand." The matching image contains entities that correspond to the noun phrases "dog," "water," and "sand" in the text. Even if the sentence is incomplete, lacking noun phrases corresponding to entities in the image, the source sentence can still be accurately translated using the matching image. Furthermore, there are numerous spatial relationships between the entities in the image. This semantic representation of spatial relationships between entities is a fine-grained representation of the image. In multimodal neural machine translation models, the spatial semantic representation of images plays a crucial role in enhancing the understanding of the sentence's contextual semantics. Spatial semantics is typically represented by global image features. Introducing global image features helps explore the connection between the spatial semantic representations of text and images, improving the quality of the model's translated sentences.

[0004] Currently, existing work on integrating image visual information into multimodal machine translation models mainly focuses on designing reasonable multimodal fusion models to fully utilize the information provided by images. For example, as initial input information in sequence-sequence coding architectures, (Huang et al., 2016; Calixto et al., 2017) initialize the hidden state of the RNN encoder by acquiring global features of the image. (Yao and Wan, 2020; Nishihara et al., 2020) performs a simple concatenation and fusion of text feature vectors and image feature vectors, and then uses a Transformer encoder to achieve semantic fusion and alignment of image and text information. Other studies (Yin et al., 2020; Huang et al., 2016; Zhang et al., 2020; Lin et al., 2020; Li et al., 2022) have used gating mechanisms to filter out visual semantic information that is irrelevant to the text while incorporating the visual information of the image into the text feature vector, thereby conducting image-text cross-modal semantic interaction to guide the generation of the text hidden state.

[0005] However, images often contain multiple entity objects, and the spatial relationships between these objects and their interrelationships are crucial. Previous methods directly encode the acquired entity objects into an input sequence and utilize traditional self-attention or gating mechanisms to align image-text cross-modal semantic information, performing feature-level fusion at the entity object and noun phrase level. This ignores the relationships between objects in each image and makes it difficult to explore fine-grained variations, thus impacting subsequent image-text feature fusion. Furthermore, due to the semantic gap between visual and textual modalities, directly using the hidden states generated by visual-text cross-modal fusion as encoded states interferes with decoding. This paper proposes a multimodal neural machine translation method based on a two-level fusion of image and textual features to perform cross-modal alignment of text and image semantic information. The aligned feature vectors are then fused to generate a feature vector that fuses visual-text cross-modal semantics. Since text and vision are two different modalities, directly using the feature vector generated by fusing these two modalities to generate encoded hidden states is inappropriate. The multimodal neural machine translation encoder should be text-driven; therefore, we designed two different gating mechanisms. A secondary fusion of visual feature semantics and textual information semantics is achieved by introducing a gating unit based on sentence context semantics. This enables the encoding of textual feature vectors guided by visual information. Summary of the Invention

[0006] To address the shortcomings of some existing methods, this invention provides a multimodal neural machine translation method based on two-level fusion of image and text features, which can improve the quality of multimodal machine translation and enhance the effective utilization of visual features.

[0007] The technical solution of this invention is: a multimodal neural machine translation method based on two-level fusion of image and text features, the specific steps of which are as follows:

[0008] Step 1: Prepare the dataset by preprocessing the public dataset Multi30K.

[0009] Step 2: The processed data is represented by source sentences and visual representations; images and text are encoded using an attention mechanism; cross-modal alignment of images and text is performed; cross-modal semantic fusion of images and text is performed; and two-level image-text fusion based on gating is performed.

[0010] Step 3: The last layer output of the encoder is sent to the traditional decoder for decoding; both the decoder and the encoder are stacked with 4 layers of model.

[0011] As a further aspect of the present invention, the specific steps of Step 1 are as follows:

[0012] The Multi30K dataset was downloaded from the WMT2018:mltimodel-task1 international translation competition, with the training, validation, and test sets containing 29k, 1014, and 1000 text-image pairs, respectively. The model was also evaluated using the WMT17 test set containing 1000 text-image pairs and the fuzzy MSCOCO test set containing 461 text-image pairs. The preprocessed sentence pairs were then used directly through byte-pair encoding segmentation and 10,000 merging operations.

[0013] As a further aspect of the present invention, Step 2, which involves performing source sentence and visual representation on the processed data, includes:

[0014] For text input, the source sentence text feature vector X is processed using an embedding layer with positional embedding. k X is obtained by embedding. kemd For visual information, pre-trained ResNet-101 and Faster R-CNN are used to obtain the grid and region features Z of the image. g Z r After the embedding layer, feature vectors X with the same dimensions as the text feature vectors are generated. gemd With X remd The specific embedding details are as follows:

[0015] X kemd =Embed(X) k )

[0016] X gemd =Embed(Z) g )

[0017] X remd =Embed(Z) r ).

[0018] As a further aspect of the present invention, Step 2, encoding images and text using an attention mechanism specifically includes: using a multi-head self-attention mechanism to encode the feature vector X output from the embedding layer. kemd ,X gemd With X remd Perform self-attention encoding to generate hidden states H respectively. t H g H r :

[0019] H t =MultiHead(X kemd ,X kemd ,X kemd )

[0020] H g =MultiHead(X gemd ,X gemd ,X gemd )

[0021] H r =MultiHead(X remd ,X remd ,X remd )

[0022] Among them, X kemd To use an embedding layer with positional embeddings to analyze the feature vector X of the source sentence text k The feature vector obtained by embedding; X gemd With X remd For visual information, we use pre-trained ResNet-101 and Faster R-CNN to obtain grid and region feature vectors of the image, respectively.

[0023] As a further aspect of the present invention, Step 2, specifically includes cross-modal alignment of images and text, including:

[0024] Design a text-guided visual gating unit to select the image target most relevant to the sentence and align the sentence to the object in the image; use the text-guided visual gating unit to consider local image features while employing an attention mechanism to focus on the image's grid features to achieve sentence alignment with the entire image; specific details are as follows:

[0025] G t,r =ρ⊙H t +W t H r

[0026] ρ = Sigmoid(W) r,t H r )

[0027] Among them, W t W r,t These are the model parameters of the gating unit. ρ represents the correlation between the image and text, with values ​​between 0 and 1. A value of 1 indicates the highest correlation, and 0 indicates no correlation. Then, the generated G... t,r The data is fed into a cross-attention mechanism to obtain the image spatial semantic representation while aligning the sentence with the entire image; where K and V are H g Q is G t,r ;

[0028] H t,r,g =MultiHead(G t,r H g H g )

[0029] Among them, H t H g H r These represent the multi-head self-attention mechanism applied to the feature vector X output from the embedding layer. kemd ,X gemd With X remd Hidden states generated through self-attention encoding; X kemd To use an embedding layer with positional embeddings to analyze the feature vector X of the source sentence text k The feature vector obtained by embedding; X gemd With X remd For visual information, pre-trained ResNet-101 and Faster R-CNN are used to obtain the grid and region feature vectors of the image, respectively.

[0030] After the above processing, the semantic representation H of the entity object with spatial location relationship is obtained. t,r,g Next, this semantic representation is input into the next processing unit.

[0031] As a further aspect of the present invention, Step 2, specifically includes the following: Image-text cross-modal semantic fusion.

[0032] An attention mechanism is used to fuse image feature vectors and text feature vectors, effectively eliminating interfering information; H t Set to Q,H t,r,g Set to K and V;

[0033] C t,r,g =MultiHead(H t H t,r,g H t,r,g ).

[0034] Among them, H t,r,g H is a semantic representation of entity objects with spatial location relationships. t This indicates that a multi-head self-attention mechanism is used to process the feature vector X output from the embedding layer. kemd Hidden states generated through self-attention encoding; X kemd To use an embedding layer with positional embeddings to analyze the feature vector X of the source sentence text k The feature vector obtained by embedding.

[0035] As a further aspect of the present invention, Step 2, which involves gating-based two-level image-text fusion of images and text, specifically includes:

[0036] (1) Visual perception is used as a text gating initialization variable;

[0037] For text feature vectors that incorporate visual information, they are used as initial vectors and incorporated into the encoding of text semantics. Visual initialization gating units are designed to guide the reasoning of semantic relationships within sentences and assist in the generation of hidden sentence states.

[0038] HF t =g(H t )

[0039]

[0040]

[0041]

[0042]

[0043] Here, g(.) is a linear function. All are visual initialization gating parameters, γ t , ζ t It is the activation function, γ t Its function is to check whether the generation of the hidden state of a sentence requires supplementary visual information. When it is 0, it indicates that the semantic context of the sentence is complete, and the text feature vector alone can complete the encoding; when it is 1, it indicates that the semantic information of the sentence is insufficient, and image features are introduced to supplement and enhance the semantic information of the sentence. t Responsible for updating the text feature vector;

[0044] Among them, C t,r,g =MultiHead(H t H t,r,g H t,r,g ), H t,r,g H is a semantic representation of entity objects with spatial location relationships. t This indicates that a multi-head self-attention mechanism is used to process the feature vector X output from the embedding layer. kemd Hidden states generated through self-attention encoding; X kemd To use an embedding layer with positional embeddings to analyze the feature vector X of the source sentence text k The feature vector obtained by embedding;

[0045] Similar to the traditional Transformer encoder model architecture, a position feedforward network (FFN) is used to represent the fused feature vector output as follows:

[0046]

[0047] Finally, the decoder is implemented using the Transformer decoder standard to translate the source language sentence;

[0048] (2) Update the text state using visual information related to the text;

[0049] By designing a gating unit that can continuously update and supplement text information, this gating unit acquires information related to another modality from text features and visual features; thus realizing the updating of text feature vectors and semantic enhancement.

[0050]

[0051]

[0052] in This involves updating the text gating parameters, where σ is the activation function and λ is the modality correlation matrix, used to update the text hidden state. Similar to the visual initialization gating in (1) above, a position feedforward network (FFN) is used to represent the encoder output as:

[0053]

[0054] Similarly, the decoder is implemented using the Transformer decoder standard.

[0055] The beneficial effects of this invention are:

[0056] 1. This invention effectively improves machine translation performance by fusing image targets and spatial information between targets;

[0057] 2. This invention designs two different types of gating units based on visual information-guided text feature vector encoding to control the secondary fusion and semantic interaction between visual feature vectors and text feature vectors, exploring how visual information should be disseminated within text. To our knowledge, this invention is the first attempt to perform secondary fusion of image feature vectors and textual semantic information.

[0058] 3. Experimental results show that the two gating fusion ideas of this invention have achieved good results in English-German and English-French machine translation tasks on the Multi30k dataset, reaching the current state-of-the-art values. Attached Figure Description

[0059] Figure 1 This is a general diagram of the model in this invention;

[0060] Figure 2 In this invention, the visual initialization text state gating is used, where σ represents the Sigmoid activation function.

[0061] Figure 3 In the text self-updating gating graph of this invention, σ represents the Sigmoid activation function;

[0062] Figure 4 This is an example of multimodal machine translation in this invention;

[0063] Figure 5 For the comparison of ablation experimental results of the method in this invention: the bold and italic parts are the focus of the comparison. Detailed Implementation

[0064] Example 1: As Figures 1-5 As shown, a multimodal neural machine translation method based on two-level fusion of image and text features is described. The specific steps of the method are as follows:

[0065] Step 1: Prepare the dataset by preprocessing the public dataset Multi30K.

[0066] Step 2: The processed data is represented by source sentences and visual representations; images and text are encoded using an attention mechanism; cross-modal alignment of images and text is performed; cross-modal semantic fusion of images and text is performed; and two-level image-text fusion based on gating is performed.

[0067] Step 3: The last layer output of the encoder is sent to the traditional decoder for decoding; both the decoder and the encoder are stacked with 4 layers of model.

[0068] As a further aspect of the present invention, the specific steps of Step 1 are as follows:

[0069] The Multi30K dataset was downloaded from the WMT2018:mltimodel-task1 international translation competition, with the training, validation, and test sets containing 29k, 1014, and 1000 text-image pairs, respectively. The model was also evaluated using the WMT17 test set containing 1000 text-image pairs and the fuzzy MSCOCO test set containing 461 text-image pairs. The preprocessed sentence pairs were then used directly through byte-pair encoding (BPE) segmentation and 10,000 merging operations.

[0070] As a further aspect of the present invention, Step 2, which involves performing source sentence and visual representation on the processed data, includes:

[0071] For text input, the source sentence text feature vector X is processed using an embedding layer with positional embedding. k X is obtained by embedding. kemd For visual information, pre-trained ResNet-101 and Faster R-CNN are used to obtain the grid and region features Z of the image. g Z r After the embedding layer, feature vectors X with the same dimensions as the text feature vectors are generated. gemd With X remd The specific embedding details are as follows:

[0072] X kemd =Embed(X) k )

[0073] X gemd =Embed(Z) g )

[0074] X remd =Embed(Z) r ).

[0075] As a further aspect of the present invention, Step 2, encoding images and text using an attention mechanism specifically includes: In multimodal machine translation (MMT), focusing on the intra-class relationships of modalities is crucial. Only when a modality has well fused its own modal information will it generate less noise when fusing inter-class relationships between modalities. The multi-head self-attention mechanism in the Transformer encoding architecture is a better mechanism for focusing on the intra-class relationships of modalities. Therefore, the multi-head self-attention mechanism is used to encode the feature vector X output after the embedding layer. kemd ,X gemd With X remd Perform self-attention encoding to generate hidden states H respectively. t H g H r :

[0076] H t =MultiHead(X kemd ,X kemd ,X kemd )

[0077] H g =MultiHead(X gemd ,X gemd ,X gemd )

[0078] H r =MultiHead(X remd ,X remd ,X remd )

[0079] Among them, X kemd To use an embedding layer with positional embeddings to analyze the feature vector X of the source sentence text k The feature vector obtained by embedding; X gemd With X remd For visual information, we use pre-trained ResNet-101 and Faster R-CNN to obtain grid and region feature vectors of the image, respectively.

[0080] As a further aspect of the present invention, Step 2, specifically includes cross-modal alignment of images and text, including:

[0081] An image often contains multiple entities, but not all detected entities are relevant to the source sentence. Using an attention mechanism might introduce semantic visual information unrelated to the sentence, affecting subsequent work. Therefore, a text-guided visual gating unit is designed to select the image targets most relevant to the sentence and align the sentence to the objects in the image. In particular, the ablation experiments in Table 3 show that using a reasonable gating mechanism to focus on the image's regional features yields better results than using an attention mechanism. However, focusing solely on regional features in the image will result in the target object's features lacking interaction with the overall image's grid features, failing to consider the fine-grained relationships between visual feature vectors. To eliminate this problem, a text-guided visual gating unit is used to consider local image features while simultaneously using an attention mechanism to focus on the image's grid features, achieving alignment between the sentence and the entire image. Specific details are as follows:

[0082] G t,r =ρ⊙H t +W t H r

[0083] ρ = Sigmoid(W) r,t H r )

[0084] Among them, W t W r,t These are the model parameters of the gating unit. ρ represents the correlation between the image and text, with values ​​between 0 and 1. A value of 1 indicates the highest correlation, and 0 indicates no correlation. Then, the generated G... t,r The data is fed into a cross-attention mechanism to obtain the image spatial semantic representation while aligning the sentence with the entire image; where K and V are H g Q is G t,r ;

[0085] H t,r,g =MultiHead(G t,r H g H g )

[0086] Among them, H t H g H r These represent the multi-head self-attention mechanism applied to the feature vector X output from the embedding layer. kemd ,X gemd With X remd Hidden states generated through self-attention encoding; X kemd To use an embedding layer with positional embeddings to analyze the feature vector X of the source sentence text k The feature vector obtained by embedding; X gemd With X remd For visual information, pre-trained ResNet-101 and Faster R-CNN are used to obtain the grid and region feature vectors of the image, respectively.

[0087] After the above processing, the semantic representation H of the entity object with spatial location relationship is obtained. t,r,g Next, this semantic representation is input into the next processing unit.

[0088] As a further aspect of the present invention, Step 2, specifically includes the following: Image-text cross-modal semantic fusion.

[0089] Due to modal differences, directly using the image hidden state obtained under text guidance to guide cross-modal text generation is difficult and will generate a large amount of interference information due to modal differences. Therefore, an attention mechanism is used to fuse image feature vectors and text feature vectors to effectively eliminate interference information; H t Set to Q,H t,r,g Set to K and V;

[0090] C t,r,g =MultiHead(H t H t,r,gH t,r,g ).

[0091] Among them, H t,r,g H is a semantic representation of entity objects with spatial location relationships. t This indicates that a multi-head self-attention mechanism is used to process the feature vector X output from the embedding layer. kemd Hidden states generated through self-attention encoding; X kemd To use an embedding layer with positional embeddings to analyze the feature vector X of the source sentence text k The feature vector obtained by embedding.

[0092] As a further aspect of the present invention, Step 2, which involves gating-based two-level image-text fusion of images and text, specifically includes:

[0093] (1) Visual perception is used as a text gating initialization variable;

[0094] For text feature vectors that incorporate visual information, they are used as initial vectors and integrated into the encoding of text semantics, through design such as... Figure 2 The visual initialization gating unit shown guides the reasoning of semantic relationships within sentences and assists in the generation of hidden sentence states; the ablation experiments in Tables 3 and 4 verify that the idea of ​​this invention is reasonable and effective. This is the first attempt to apply the attention + visual initialization gating method in an encoder architecture.

[0095] HF t =g(H t )

[0096]

[0097]

[0098]

[0099]

[0100] Here, g(.) is a linear function. All are visual initialization gating parameters, γ t , ζ t It is the activation function, γ t Its function is to check whether the generation of the hidden state of a sentence requires supplementary visual information. When it is 0, it indicates that the semantic context of the sentence is complete, and the text feature vector alone can complete the encoding; when it is 1, it indicates that the semantic information of the sentence is insufficient, and image features are introduced to supplement and enhance the semantic information of the sentence. t Responsible for updating the text feature vector;

[0101] Among them, Ct,r,g =MultiHead(H t H t,r,g H t,r,g ), H t,r,g H is a semantic representation of entity objects with spatial location relationships. t This indicates that a multi-head self-attention mechanism is used to process the feature vector X output from the embedding layer. kemd Hidden states generated through self-attention encoding; X kemd To use an embedding layer with positional embeddings to analyze the feature vector X of the source sentence text k The feature vector obtained by embedding;

[0102] Similar to the traditional Transformer encoder model architecture, a position feedforward network (FFN) is used to represent the fused feature vector output as follows:

[0103]

[0104] Finally, the decoder is implemented using the Transformer decoder standard to translate the source language sentence;

[0105] (2) Update the text state using visual information related to the text;

[0106] In MMT, text plays a dominant role, and the principle of visual guidance for text is applied. This is addressed by designing a gating unit that continuously updates and supplements textual information. This gating unit acquires information related to another modality from both textual and visual features, enabling the updating of textual feature vectors and semantic enhancement. For detailed structure, see [link to detailed structure]. Figure 3 ;

[0107]

[0108]

[0109] in This involves updating the text gating parameters, where σ is the activation function and λ is the modality correlation matrix, used to update the text hidden state. Similar to the visual initialization gating in (1) above, a position feedforward network (FFN) is used to represent the encoder output as:

[0110]

[0111] Similarly, the decoder is implemented using the Transformer decoder standard.

[0112] To illustrate the effectiveness of this invention, the following experiments were conducted:

[0113] The Multi30K dataset was used, with training, validation, and test sets containing 29k, 1014, and 1000 text-image pairs, respectively. Additionally, the effectiveness of the proposed method was evaluated using the WMT17 test set (containing 1000 text-image pairs) and the fuzzy MSCOCO test set (containing 461 text-image pairs). Parameter settings are shown in Table 1 below.

[0114] Table 1: Model Parameter Settings

[0115]

[0116] Finally, the average of the checkpoints from the last 15 models was used as the model parameters for this invention, and the metrics BLEU and METEOR were used to evaluate the model's translation performance. During training, if the BLEU score failed to improve the model's translation validation data for 10 consecutive times, training was stopped. Experimental results on three test sets of English-German language pairs are as follows: Figure 4 As shown.

[0117] Table 2: Experimental results in the English→German (En→De) multimodal translation task

[0118]

[0119] First, the multimodal neural machine translation method (MNMT) based on dual-level fusion of image and text features proposed in this invention can effectively guide the encoding of text feature vectors through visual information, with text information being the primary modality in the model of this invention. Both models designed in this invention can effectively fuse visual information into the text, assisting in the generation of hidden states from the text feature vectors.

[0120] Secondly, the model proposed in this invention has been designed in terms of graph-text cross-modal semantic alignment, graph-cross-modal fusion and semantic interaction, and the rationality of the design of each unit module has been explored in the ablation experiment.

[0121] Finally, influenced by the conclusions of (Wu et al., 2021), a suitable number of parameters is crucial for a model. Therefore, this invention uses different model parameters to conduct experiments on its model. The results are shown in Table 2. It is clear that when using the Transformer-Tiny parameters based on Wu et al. (2021), even after continuously adjusting parameters such as decay rate and learning rate to achieve the best performance under those parameters, it still does not perform as well as the model based on Transformer-Small parameters, and there is even a significant difference. Inspired by (Chen et al., 2021), we believe this may be because the number of Transformer-Tiny parameters is too small, while our training mechanism is different, the model is more complex, and the number of model parameters is relatively large. Therefore, it underfits on the Multi30K dataset. This underfitting is not only due to the size of the data but also related to the complexity of the model. Therefore, Transformer-Small is used as the baseline for discussing the multimodal machine translation model (MNMT) of this invention.

[0122] Ablation experiments were conducted on image features. Firstly, the impact of semantic interactions between image region features, grid features, and text feature vectors on the model is a key research point of this invention. The use of image grid features and region features was investigated. By removing grid features and region features and randomly initializing visual features, the importance of visual feature-level and fine-grained processing in the field of multimodal machine translation was explored. The experimental results are shown in Table 3, where Remove grid features, Remove region features, Random grid feature, and Random region feature represent the removal of grid features, region features, and randomized grid and region features, respectively.

[0123] Table 3: Experimental Study on Ablation of Image Features (En→De)

[0124]

[0125] For multimodal neural machine translation models, utilizing only the grid features or region features of an image as the input for the model's image semantic representation can improve the translation quality of text. However, observing rows 1, 3, and 4 in Table 3 reveals that removing the grid or region features of the image leads to a certain decline in experimental results on the Test 2016 and Test 2017 test sets. This is especially true when region features are removed, resulting in a significant drop in the Test 2016 results. This indicates that visual information can indeed improve the quality of translation. However, the reason for the severe decline in data results when region features are removed may be that the drawback of only establishing the semantic relationships of global features while ignoring the semantic relationships of region features is more detrimental to multimodal machine translation models. The features of entity objects play a crucial role in guiding the text feature vector, and exploring the fine-grained processing of vision may have a gilding effect on the model.

[0126] The model of this invention has a good learning ability for visual information. Even if grid features or region features are replaced with randomly shuffled visual features, the model can still make full use of the semantic representation of visual content. In particular, it can be found that in the model of this invention, removing region features and replacing region features with random region features has a greater impact on model performance than processing grid features. In addition, the data in Tables 2 and 3 show that the BULE and METEOR scores on the MSCOCO test set indicate that visual features are also helpful for the translation of ambiguous sentences, and the more abundant the visual information, the better the disambiguation effect. Ablation experiments on the components of the method: To explore the impact of processing the semantic relationship of visual content, interactive attention mechanism, and vision-based text generation gating on the overall model performance, we investigated the impact of each part on our experimental performance by replacing the text-guided visual gating mechanism with attention, and removing interactive attention and vision-based text generation gating units. The experimental results are shown in Table 4. In Table 4, Remove GATA represents the method of removing GATA gating, and Remove Transit Attention represents the method of removing fused attention. "gat replace with Attention" means replacing GATA gating with an attention mechanism.

[0127] Table 4: Ablation experiments on model components (En-De)

[0128]

[0129] Text-guided visual gating mechanism. As shown in rows 1 and 4 of Table 4, when we replace the text-guided visual gating unit with an attention mechanism, the experimental results on the Test 2016 test set show a slight decrease. This indicates that in the field of multimodal machine translation, text-guided gating units are better at learning the relationships between region features compared to using attention to learn the relationships between entities in an image.

[0130] The effectiveness of the image-text cross-modal attention fusion unit is evident. Rows 1 and 3 in Table 4 show that using the attention mechanism for image-text cross-modal fusion helps visual information better guide the text feature vector to generate hidden states, reducing interference caused by different modalities.

[0131] The importance of the image-text cross-modal fusion gating unit. In experiments, removing this component resulted in a significant performance drop, approaching two percentage points, which may indicate that in multimodal machine translation models, using attention mechanisms for cross-modal fusion of already aligned image-text feature vectors does not fully leverage the potential of visual semantics in multimodal models. A more reasonable approach would be to integrate the processed visual information into the text semantics, guiding the generation of hidden text states, with the visual information playing a supporting role.

[0132] For image-text cross-modal gating fusion units, it is crucial to employ a suitable image-text cross-modal fusion unit for secondary fusion. The rationality and superiority of the image-text cross-modal gating fusion unit design were explored by replacing the visual initialization gating unit (GATA) with its image-text cross-modal fusion unit. The experimental results are shown in Table 5.

[0133] Table 5: Ablation experiments on the image-text cross-modal gated fusion unit (En-De)

[0134]

[0135] In Table 5, "GATA replace with Attention" means replacing the GATA method with attention, "GATA replace with LSTM-gating" means replacing the GATA gating method with a traditional LSTM gating unit, and "GATA replace with Gating" means replacing GATA with a common gating mechanism.

[0136] When the visual initialization gate in this experiment was replaced with the gating mechanism and LSTM gating unit from Yin et al. (2020), the experimental results on the Test 2016 test set showed a certain degree of decline. This may be because redundant connections exist in the LSTM gating unit, leading to ambiguity in inter- and intra-class relationships between modalities, resulting in unsatisfactory final results. As for replacing the gating unit with an attention mechanism, the experimental results showed a slight decline. This may be because the attention mechanism is better suited for semantic alignment of different modalities. Although it can also integrate the semantic representation of visual content into the text to guide the generation of hidden states, this is clearly not its strength. Using a suitable gating unit can better accomplish this task.

[0137] Case Analysis

[0138] To better explore the rationality of the design of each part of the proposed method, a case study analysis was conducted on the entire multimodal fusion ablation experiment process. Detailed results can be found in [link to results]. Figure 5 The bolded and underlined text is the part we focus on for comparison. In the ablation experiments on image features, the method of this invention can better learn words like "seitenlinie" (edge), which are influenced by spatial positional relationships. The mesh features of the image show the most significant effect on its spatial semantic learning. Furthermore, in the ablation experiments on the components of the method, when the Visual Initialization Gating Unit (GATA) is removed and in plain text translation, words like "inmitten" (in the middle), which have spatial positional semantics, are not learned well. Meanwhile, our GATA helps guide visual information to assist text translation, accurately translating the word "schutthaufen" (rubble). When the GATA is removed, visual information lacks its auxiliary role in text translation. The method constructed in our method is highly efficient.

[0139] Results in the English-French translation task

[0140] To better test the robustness of our method, it was also tested on an English-French translation task. The results are shown in Table 6, and to our knowledge, the experimental results reach the current state-of-the-art values. This fully demonstrates that the method of this invention is not only reasonable and interpretable, but also that its experimental results are highly competitive.

[0141] Table 6: Experimental results on English-French translation tasks

[0142]

[0143] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A multimodal neural machine translation method based on two-level fusion of image and text features, characterized in that: The specific steps of the multimodal neural machine translation method based on two-level fusion of image and text features are as follows: Step 1: Prepare the dataset by preprocessing the public dataset Multi30K. Step 2: The processed data is represented by source sentences and visual representations; images and text are encoded using an attention mechanism; cross-modal alignment of images and text is performed; cross-modal semantic fusion of images and text is performed; and two-level image-text fusion based on gating is performed. Step 3: The last layer output of the encoder is sent to a traditional decoder for decoding; Step 2, specifically the gate-based two-level image-text fusion, includes: (1) Visual perception is used as a text gating initialization variable; For text feature vectors that incorporate visual information, they are used as initial vectors and incorporated into the encoding of text semantics. Visual initialization gating units are designed to guide the reasoning of semantic relationships within sentences and assist in the generation of hidden sentence states. ； Here It is a linearly changing function. , , , , , All are visual initialization gating parameters. , It is an activation function. Its function is to check whether the generation of the hidden state of a sentence requires the supplementation of visual information. When it is 0, it indicates that the semantic context of the sentence is complete and the text feature vector alone can complete the encoding; when it is 1, it indicates that the semantic information of the sentence is insufficient, and image features are introduced to supplement and enhance the semantic information of the sentence. Responsible for updating the text feature vector; in, , For semantic representation of entity objects with spatial location relationships, This indicates that a multi-head self-attention mechanism is used to process the feature vector output after the embedding layer. Hidden states generated through self-attention encoding; To use an embedding layer with positional embeddings to extract feature vectors from the source sentence text The feature vector obtained by embedding; The fused feature vector output is represented using a location-feedforward network (FFN) as follows: ； Finally, the decoder is implemented using the Transformer decoder standard to translate the source language sentence; (2) Update the text state using visual information related to the text; By designing a gating unit that can continuously update and supplement text information, this gating unit acquires information related to another modality from text features and visual features; thus realizing the updating of text feature vectors and semantic enhancement. ； ； in , It updates the text gating parameters. It is an activation function. This is the modality correlation matrix, used to update the text hidden state. A position-feedforward network (FFN) is used to represent the encoder output as follows: ； Similarly, the decoder is implemented using the Transformer decoder standard.

2. The multimodal neural machine translation method based on two-level fusion of image and text features according to claim 1, characterized in that: The specific steps of Step 1 are as follows: Download the Multi30K dataset from the WMT2018:mltimodel-task1 international translation competition. The training, validation, and test sets contain 29k, 1014, and 1000 text-image pairs, respectively. Additionally, the WMT17 test set (containing 1000 text-image pairs) and the fuzzy MSCOCO test set (containing 461 text-image pairs) are used to evaluate the model.

3. The multimodal neural machine translation method based on two-level fusion of image and text features according to claim 1, characterized in that: In Step 2, the processed data is divided into source sentences and visual representations, including: For text input, the source sentence text feature vector is processed using an embedding layer with positional embedding. Embedded For visual information, pre-trained ResNet-101 and Faster R-CNN are used to obtain grid and region features of the image. , After the embedding layer, feature vectors with the same dimensions as the text feature vectors are generated. and The specific embedding details are as follows: ； ； 。 4. The multimodal neural machine translation method based on two-level fusion of image and text features according to claim 1, characterized in that: Step 2, which uses an attention mechanism to encode images and text, specifically includes: using a multi-head self-attention mechanism to encode the feature vector output from the embedding layer. , and Perform self-attention encoding to generate hidden states respectively. , , : ; ; ; in, To use an embedding layer with positional embeddings to extract feature vectors from the source sentence text The feature vector obtained by embedding; and For visual information, pre-trained ResNet-101 and Faster R-CNN are used to obtain grid and region feature vectors of the image, respectively.

5. The multimodal neural machine translation method based on two-level fusion of image and text features according to claim 1, characterized in that: Step 2, specifically, includes cross-modal alignment of images and text: Design a text-guided visual gating unit to select the image target most relevant to the sentence and align the sentence to the object in the image; use the text-guided visual gating unit to consider local image features while employing an attention mechanism to focus on the image's grid features to achieve sentence alignment with the entire image; specific details are as follows: ； ； in, , These are the model parameters of the gating unit. This indicates the relevance between the text and images, with values ​​between 0 and 1. A value of 1 indicates the highest relevance, and 0 indicates no relevance. Then, the generated... The data is fed into a cross-attention mechanism to obtain the image spatial semantic representation while aligning the sentence with the entire image; where K and V are... Q is ; ； in, , , These represent the multi-head self-attention mechanism applied to the feature vector output from the embedding layer. , and Hidden states generated through self-attention encoding; To use an embedding layer with positional embeddings to extract feature vectors from the source sentence text The feature vector obtained by embedding; and For visual information, pre-trained ResNet-101 and Faster R-CNN are used to obtain grid and region feature vectors of the image, respectively. After the above processing, a semantic representation of the entity object with spatial location relationship is obtained. Next, this semantic representation is input into the next processing unit.

6. The multimodal neural machine translation method based on two-level fusion of image and text features according to claim 1, characterized in that: Step 2, specifically the cross-modal semantic fusion of images and text, includes: An attention mechanism is used to fuse image feature vectors and text feature vectors, effectively eliminating interfering information; Set to Q, Set to K and V; ； in, For semantic representation of entity objects with spatial location relationships, This indicates that a multi-head self-attention mechanism is used to process the feature vector output after the embedding layer. Hidden states generated through self-attention encoding; To use an embedding layer with positional embeddings to extract feature vectors from the source sentence text The feature vector obtained by embedding.

Images