A two-stage visual fusion multi-modal translation method and system

Abstract

The application provides a kind of two-stage visual fusion multimodal translation method and system, to solve the problem of insufficient utilization of visual information in existing multimodal translation, single fusion mechanism, model prone to ignore visual information.The method comprises: obtaining source language text and corresponding image, extracting fine-grained regional visual features and coarse-grained global visual features;In the first stage, the fine-grained regional features are spliced with the text representation and encoded through guided self-attention to realize cross-modal alignment;In the second stage, the coarse-grained global features are projected and broadcasted through gated residual modulation to inject scene-level prior information;The multimodal machine translation target and the visual conditional mask language modeling target are jointly optimized for training.The application effectively improves the translation quality through two-stage multi-granularity fusion, and the BLEU and METEOR indicators on Multi30K and other data sets are significantly better than existing methods, which can be applied to multimedia translation, cross-language retrieval and other fields.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and machine translation technology, specifically to a two-stage visual fusion multimodal translation method and system. Background Technology

[0002] Multimodal Machine Translation (MMT) aims to incorporate visual information, such as images, into the translation process, providing additional semantic cues for target text generation, thereby alleviating semantic ambiguity in plain text translation and improving translation quality. In recent years, with the development of visual representation learning and cross-modal modeling techniques, MMT has made steady progress in both model architecture design and training paradigms.

[0003] However, existing research has shown that current MMT methods often struggle to consistently outperform powerful plain text translation models. On standard benchmark datasets such as Multi30K, the performance gains from multimodal inputs are typically limited and unstable, and some systems exhibit weak sensitivity to visual input. Furthermore, while visual input can provide translation-related cues, it may also introduce information irrelevant to the translation task. In diagnostic experiments, even replacing images with irrelevant images or random noise may only slightly alter model performance, indicating that the model can easily degenerate into a text-dominated decoding strategy, potentially diminishing the contribution of visual information.

[0004] From the perspective of the timing of visual information injection, existing fusion strategies are generally divided into early fusion, mid-term fusion, and late-term fusion. Early fusion concatenates visual features with text embeddings and inputs them into the encoder, which is simple to implement but easily introduces cross-modal noise. Mid-term fusion introduces an interaction module in the middle layer of the encoder, which can reduce the risk of noise propagation but is sensitive to hyperparameters. Late-term fusion injects visual information at a later stage, which is more robust to noisy visual signals but is difficult to achieve fine-grained cross-modal alignment. Many existing methods rely on a single fusion stage or only emphasize single-granularity interaction, which makes it difficult for the model to achieve the following two goals simultaneously: (i) fine-grained fusion for disambiguation; (ii) robust global semantic complementation under visual noise conditions.

[0005] Therefore, designing a more effective visual fusion mechanism that can achieve fine-grained cross-modal alignment to eliminate semantic ambiguity, inject robust global semantic priors to suppress noise interference, and alleviate the problem that models tend to ignore visual information during training has become the key to improving the quality of multimodal machine translation. Summary of the Invention

[0006] To address the problems in existing technologies, this invention provides a two-stage visual fusion multimodal translation method and system, aiming to solve the problems of insufficient utilization of visual information, single fusion mechanism, and easy neglect of visual information by models in existing technologies, and to achieve a stable improvement in translation quality.

[0007] This invention provides a two-stage visual fusion multimodal translation method, comprising the following steps: Step 1: Obtain the source language text and its corresponding image; Step 2: Extract fine-grained regional visual features and coarse-grained global visual features from the image; Step 3: In the first stage of fusion, the fine-grained regional visual features and the text representation of the source language text are concatenated in the sequence dimension to form a multimodal input sequence, and the multimodal input sequence is encoded through a guided self-attention mechanism to obtain a multimodal encoded representation; Step 4: In the second stage of fusion, the coarse-grained global visual features are mapped to the hidden layer dimension of the model through a linear projection layer and copied and expanded along the sequence dimension to align with the sequence length of the source language text; residual modulation is performed on the multimodal coding representation using a gating mechanism to obtain the feature representation after global visual gating fusion. Step 5: Based on the feature representation after global visual gating fusion, generate the target language translation through the decoder.

[0008] Furthermore, the fine-grained regional visual features are extracted using the MDETR model, and the coarse-grained global visual features are extracted using the CLIP model.

[0009] Furthermore, the guided self-attention mechanism is based on a guiding matrix determined by the alignment results of the MDETR model, restricting cross-modal interactions in self-attention computation to occur only between semantically related text terms and image regions. Specifically, a binary guiding matrix C is constructed, where intramodal positions are fully connected, and intermodal connections are determined by the MDETR alignment results; the self-attention weights are calculated using masked softmax, and normalization is performed only at positions allowed by the guiding matrix.

[0010] Furthermore, the gating mechanism is implemented in the following ways: The copied and expanded coarse-grained global visual features are concatenated with the portion of the multimodal encoded representation corresponding to the source language text. The concatenated features are input into a multilayer perceptron, and a gating value is generated using the tanh activation function. The gated value is multiplied element-wise with the copied and expanded coarse-grained global visual features, and then the residual is added to the part of the multimodal encoded representation corresponding to the source language text to obtain the feature representation after global visual gating fusion.

[0011] This invention provides a training method for a multimodal translation model, which uses image-text sample pairs processed by the first-stage fusion and the second-stage fusion as described above, and includes the following steps: Based on the feature representation after global visual gating fusion, the multimodal machine translation loss is calculated; Based on the coarse-grained global visual features or the fine-grained regional visual features, the masked lexical units in the target language translation are predicted, and the visual conditional masking language modeling loss is calculated. The multimodal machine translation loss and the visual conditional mask language modeling loss are jointly optimized to update the model parameters.

[0012] The multimodal machine translation loss is the negative log-likelihood loss at the lexical level of the target language translation; the visual conditional masking language modeling loss is the cross-entropy loss for predicting masked lexical terms under visual feature conditions; and the joint optimization combines the two losses in a weighted summation manner at a preset ratio.

[0013] This invention provides a two-stage visual fusion multimodal translation system, comprising: The visual feature extraction module is used to extract fine-grained regional visual features and coarse-grained global visual features from the input image; The first fusion module is used to concatenate the fine-grained regional visual features with the text representation of the source language text in the sequence dimension to form a multimodal input sequence, and to encode the multimodal input sequence through a guided self-attention mechanism to output a multimodal encoded representation; The second fusion module is used to map the coarse-grained global visual features to the hidden layer dimension of the model through a linear projection layer, and copy and expand them along the sequence dimension to align them with the sequence length of the source language text; and to perform residual modulation on the multimodal encoding representation using a gating mechanism to output the feature representation after global visual gating fusion. The decoding module is used to autoregressively generate the target language translation based on the feature representation after global visual gating fusion.

[0014] Furthermore, the visual feature extraction module includes: A fine-grained feature extraction unit is used to extract the visual features of the fine-grained region using the MDETR model; The coarse-grained feature extraction unit is used to extract the coarse-grained global visual features through the CLIP model.

[0015] Furthermore, the guided self-attention mechanism in the first fusion module is based on a guiding matrix determined by the alignment results of the MDETR model, which restricts cross-modal interactions in self-attention computation to occur only between semantically related text terms and image regions.

[0016] Furthermore, the gating mechanism in the second fusion module is specifically used for: The copied and expanded coarse-grained global visual features are concatenated with the portion of the multimodal encoded representation corresponding to the source language text. The concatenated features are input into a multilayer perceptron, and a gating value is generated using the tanh activation function. The gated value is multiplied element-wise with the copied and expanded coarse-grained global visual features, and then the residual is added to the part of the multimodal encoded representation corresponding to the source language text to obtain the feature representation after global visual gating fusion.

[0017] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above, or implements the training method described above.

[0018] The beneficial effects of this invention are: 1. Two-stage, multi-granularity fusion: In the first stage, guided self-attention encoding using fine-grained regional features enhances the model's ability to align and disambiguate local details across modalities. In the second stage, gated residual modulation using coarse-grained global features injects scene-level prior information and suppresses noise. The two stages complement each other, effectively improving translation quality.

[0019] 2. Gated Residual Modulation Mechanism: A novel gated residual structure was designed, which enables the model to adaptively control the contribution of global visual information to different source language lexical units, thereby enhancing the flexibility and robustness of the fusion.

[0020] 3. Joint training objective: Visual conditional mask language modeling (VMLM) is introduced as an auxiliary task, which forces the model to pay more attention to visual information during training, effectively alleviating the problem of the model ignoring visual cues and further improving the effect of multimodal fusion.

[0021] 4. Significant performance improvement: Experimental results on the Multi30K Test2016 and MSCOCO image-bilingual parallel test sets show that the present invention achieves superior performance to existing baseline models in both English-German (En-De) and English-French (En-Fr) translation directions, with significant improvements in BLEU and METEOR metrics, demonstrating its effectiveness and advancement. Attached Figure Description

[0022] Figure 1 This is a flowchart of a two-stage visual fusion multimodal translation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall framework of the model provided in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings. Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The directional terms such as left, center, right, top, and bottom in the embodiments of the present invention are only relative concepts or referenced to the normal use state of the product, and should not be considered restrictive.

[0024] Example 1: Model Architecture; This embodiment provides a multimodal translation model based on two-stage visual fusion, such as... Figure 1 and Figure 2 As shown, the model mainly includes: an input representation layer, a first-stage fusion module (guided self-attention encoder), a second-stage fusion module (global visual gating modulation), and a decoder; its multimodal translation method includes the following steps: Step 1: Input representation construction; Before proceeding to the first stage of fusion, the input representations of text and vision are first constructed; that is, given the source sentence... ,in, This represents the t-th source sentence token. Indicates the length of the source sentence; The source sentence is mapped to a text embedding representation through word embedding layers and text position encoding: in, Represents word embedding layer, Here, B represents the text position encoding, B represents the batch size, and D represents the hidden layer dimension. Therefore... A text representation of the source sentence; For images, extraction is performed using the MDETR model. Fine-grained regional features ; Represents the original visual features of the k-th image region. The original dimension represents the region features; to ensure that visual features and text features reside in the same latent space, they are mapped to the latent space through linear projection and region location encoding: in, It is a learnable linear projection matrix. For bias terms, Representation layer normalization. Indicates the area location code. This represents the projected regional visual representation. Meanwhile, coarse-grained global visual features are extracted using the CLIP model. ,in, Represents the global visual feature dimension. Represents batch size; Step 2: First-stage fusion, guiding self-attention encoding; The visual representation of the region and the text representation are concatenated along the sequence dimension to form a unified multimodal input sequence: in, This indicates a concatenation operation along the sequence dimension; thus, It simultaneously incorporates fine-grained visual region representation and text representation, which are used as input to the multimodal encoder. Subsequently, guided self-attention is introduced into the multimodal encoder, so that cross-modal interactions only occur between semantically related word-region pairs, thereby enhancing cross-modal alignment, visual fusion, and disambiguation capabilities, while suppressing noisy connections. Using Guided Self-Attention (GSA) Encoding is performed to inject cross-modal priors into the self-attention mechanism; this method ensures that interaction is only allowed between semantically related image regions and text terms, thereby reducing pseudo-cross-modal connections and noise interference; specifically, text embeddings and fine-grained region visual features are concatenated into a unified sequence of length N, where T and V represent the text terminology index set and the visual region terminology index set, respectively, and satisfying... ; First, the cross-modal correspondences are obtained using MDETR, and a binary guiding matrix is ​​constructed: , The matrix rules are as follows: when hour, (i.e., fully connected within the modality); otherwise, Determined by the MDETR alignment result (i.e., intermodal connectivity); Given a query, key, and value projection The weights for standard self-attention assignment are: Guiding attention, on the other hand, restricts the normalization of softmax to the allowed set specified by C (i.e., masked softmax): in, , These are the query and key vectors, respectively. Represents the first in the guiding matrix C The position and the first Whether an attention connection is allowed at a given location, specifically, when... A value of 1 allows a connection to be established, while a value of 0 prevents a connection from being established. Indicates the first The key vector corresponding to each position; Indicates the dimension of the key vector; The summation index represents the traversal of all possible positions; this mechanism injects alignment priors into the attention computation, allowing only semantically related text terms to interact across modalities with image regions, thereby reducing pseudo-cross-modal connections and noise interference; The output of the first-stage fusion is a multimodal coded representation H; in, This represents the defined guided self-attention mask. This indicates an encoder that incorporates a guided self-attention mechanism; Step 3: Second-stage fusion, global visual gating modulation; First, coarse-grained global visual features are projected onto the hidden layer dimension of the model and used as a semantic summary to modulate the multimodal encoder representation obtained in the first stage: in, Let be the projection matrix. For bias terms, Represents the projected global visual vector; Then, The sequence is copied and expanded (i.e. broadcast) along the sequence dimension to all source language lexical positions to obtain a global visual representation aligned with the length of the text sequence: in, Indicates a broadcast operation; To adaptively control the contribution of visual information to different lexical units, a gating signal is constructed: in, yes The part that corresponds to the source language text. It is a two-layer, multi-layer sensor. This indicates concatenation along the feature dimension; Perform gated residual injection: in, This represents element-wise multiplication. This represents the feature representation after global visual gating fusion. Step 4: Decoding and Generation; Based on the feature representation after global visual gating fusion The target language translation is generated autoregressively using a multimodal decoder; whereby the target language translation is generated autoregressively. The conditional probability of each word is expressed as: in, Indicates model parameters, Indicates the first sentence in the target sentence Each word element, This indicates the previous target-side historical terminology. This indicates a multimodal decoder. This expression represents visual features; it shows that the target-side generation ultimately depends on the source sentence. and its encoded representation after global visual modulation .

[0025] Example 2: This embodiment provides a training method for a multimodal translation model, which adopts the fusion architecture described in Embodiment 1 and jointly optimizes two training objectives.

[0026] Step 1: Multimodal Machine Translation Loss (MMT Loss); Given source sentence The corresponding visual feature v, and the target sentence We employ teacher forcing and minimize the lexical-level negative log-likelihood loss: The implementation ignores the fill position; Step 2: Visual Conditional Mask Language Modeling Loss (VMLM Loss); Given the target sentence Sample a set of masked locations And replace the corresponding word with [MASK] to get Given visual feature v, the model is trained only to recover the masked terms. Step 3: Joint optimization; The objectives of multimodal translation (MMT) and visual conditional masking language modeling (VMLM) are jointly optimized. VMLM is a cross-modal training objective that predicts masked text terms under visual feature conditions to enhance multimodal fusion capabilities. The overall training objective is defined as a weighted sum of the two losses: in, This represents the probability of sampling a VMLM training step. The model parameters are represented by ; through joint optimization, the model is forced to learn to utilize visual information to reduce both types of loss at the same time, thereby effectively alleviating the problem of visual neglect.

[0027] Experimental results This embodiment verifies the effectiveness of the method of the present invention on the Multi30K dataset. The experimental setup is as follows: The text backbone model, mBART-large-50, is first trained on a large-scale parallel corpus of plain text to obtain a strong baseline for plain text machine translation. Visual features: Fine-grained region features are extracted using MDETR, and coarse-grained global features are extracted using CLIP; Dataset: Multi30K training set (29k image-text pairs), with the model fine-tuned using image-source language-target language triples; evaluated on Test2016 (1000 pairs) and MSCOCO (461 pairs) test sets; Evaluation metrics: BLEU (SacreBLEU, 13a tokenization, case sensitive) and METEOR; Training parameters: Adam optimizer, learning rate 1e-5, dropout 0.4, label smoothing 0.1, batch size 32 (gradient accumulation in 4 steps), early stopping policy patience=5, beam size during inference=4; The experimental results are shown in the table below. In the English-French translation task, this invention achieved 68.07 BLEU / 84.82 METEOR on Test2016 and 52.53 BLEU / 75.30 METEOR on MSCOCO, significantly outperforming baseline models such as GIIFT. In the English-German translation task, this invention achieved 46.31 BLEU / 71.98 METEOR on Test2016 and 37.21 BLEU / 62.89 METEOR on MSCOCO, also achieving the best performance.

[0028] As shown in the table, the BLEU and METEOR results on the Multi30K Test2016 and MSCOCO image-bilingual parallel test sets are presented for the two translation directions, En–Fr and En–De. Overall, DSVF achieves the best performance in all four evaluation settings, indicating that the proposed method is stable and effective in different language directions and test domains.

[0029] On the En–Fr task, the model of this invention achieved 68.07 BLEU / 84.82 METEOR on Test2016; compared with the best-performing competing system GIIFT (with image) (65.17 BLEU / 83.32 METEOR) in this setting, the method of this invention improved by +2.90 BLEU and +1.50 METEOR, respectively. On the more challenging MSCOCO test set, the model of this invention further achieved 52.53 BLEU / 75.30 METEOR, which improved by +2.77 BLEU and +1.32 METEOR compared with GIIFT (with image) (49.76 BLEU / 73.98 METEOR), respectively. These results demonstrate that the proposed two-stage visual fusion method remains effective even under conditions of larger neighborhood distribution shifts and stronger noise.

[0030] On the En–De task, the model of this invention also achieved state-of-the-art results. On Test2016, the model achieved 46.31 BLEU / 71.98 METEOR, surpassing the second-best method, Soul-Mix (44.24 BLEU / 69.93 METEOR) by +2.07 BLEU and +2.05 METEOR. On MSCOCO, the method of this invention achieved 37.21 BLEU / 62.89 METEOR, outperforming the second-best method, GIIFT (with image) (34.66 BLEU / 61.37 METEOR), by +2.55 BLEU and +1.52 METEOR, respectively. It was also observed that the absolute scores on En–De were generally lower than those on En–Fr, which is largely consistent with the higher linguistic complexity of German (e.g., richer morphological variations and more flexible word order). More importantly, the performance gains of the method of this invention remained stable across different linguistic directions, further demonstrating the robustness of the proposed framework.

[0031] Comparisons across different baseline categories revealed two key findings. First, compared to the robust plain-text baseline mBART, the method presented in this invention consistently improved BLEU and METEOR across both translation directions and test sets, indicating that visual signals were effectively utilized to support disambiguation and translation generation. Second, while image-free inference methods (e.g., CLIPTrans, VALHALLA, and ImgiT) may offer some improvements in certain situations, the two-stage framework of this invention, by explicitly introducing visual input, achieved stronger overall performance, particularly on MSCOCO. This trend suggests that discarding images during the inference stage may weaken visual fusion capabilities and cross-domain generalization.

[0032] Example 4: This embodiment provides a two-stage visual fusion multimodal translation system, including: Visual feature extraction module: includes a fine-grained feature extraction unit (MDETR) and a coarse-grained feature extraction unit (CLIP), which output regional and global visual features respectively; The first fusion module concatenates the region visual features with the text representation and processes them through a guided self-attention encoder to output a multimodal coded representation. The second fusion module projects and broadcasts the global visual features, then fuses them with gated residual modulation and multimodal coding representation to output the final feature representation. Decoding module: Based on a multimodal decoder, it generates target language translations through autoregression.

[0033] The specific implementation methods of each module are consistent with those described in Embodiments 1 and 2.

[0034] Industrial applicability The two-stage visual fusion multimodal translation method and system provided by this invention can be applied to multimedia content translation, cross-language information retrieval, intelligent customer service, and auxiliary translation tools. By effectively utilizing image visual information, translation quality can be significantly improved, especially in scenarios with semantic ambiguity. The technical solution of this invention can be implemented by computer programs and deployed on servers or terminal devices, showing promising prospects for industrial applications.

[0035] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A two-stage visual fusion multimodal translation method, characterized in that, Includes the following steps: Step 1: Obtain the source language text and its corresponding image; Step 2: Extract fine-grained regional visual features and coarse-grained global visual features from the image; Step 3: In the first stage of fusion, the fine-grained regional visual features and the text representation of the source language text are concatenated in the sequence dimension to form a multimodal input sequence, and the multimodal input sequence is encoded through a guided self-attention mechanism to obtain a multimodal encoded representation; Step 4: In the second stage of fusion, the coarse-grained global visual features are mapped to the hidden layer dimension of the model through a linear projection layer and copied and expanded along the sequence dimension to align with the sequence length of the source language text; residual modulation is performed on the multimodal coding representation using a gating mechanism to obtain the feature representation after global visual gating fusion. Step 5: Based on the feature representation after global visual gating fusion, generate the target language translation through the decoder.

2. The multimodal translation method based on two-stage visual fusion according to claim 1, characterized in that: The fine-grained regional visual features are extracted using the MDETR model, and the coarse-grained global visual features are extracted using the CLIP model.

3. The multimodal translation method based on two-stage visual fusion according to claim 1, characterized in that: The guided self-attention mechanism is based on a guidance matrix determined by the alignment results of the MDETR model, which restricts cross-modal interactions in self-attention computation to occur only between semantically related text terms and image regions.

4. The multimodal translation method based on two-stage visual fusion according to claim 1, characterized in that: The gating mechanism is implemented in the following ways: The copied and expanded coarse-grained global visual features are concatenated with the portion of the multimodal encoded representation corresponding to the source language text. The concatenated features are input into a multilayer perceptron, and a gating value is generated using the tanh activation function. The gated value is multiplied element-wise with the copied and expanded coarse-grained global visual features, and then the residual is added to the part of the multimodal encoded representation corresponding to the source language text to obtain the feature representation after global visual gating fusion.

5. A training method for a multimodal translation model, characterized in that, The method employs the image-text sample pairs processed by the first-stage fusion and the second-stage fusion in any of the multimodal translation methods described in claims 1 to 4, and includes the following steps: Based on the feature representation after global visual gating fusion, the multimodal machine translation loss is calculated; Based on the coarse-grained global visual features or the fine-grained regional visual features, the masked lexical units in the target language translation are predicted, and the visual conditional masking language modeling loss is calculated. The multimodal machine translation loss and the visual conditional mask language modeling loss are jointly optimized to update the model parameters. The multimodal machine translation loss is the negative log-likelihood loss at the lexical level of the target language translation; the visual conditional masking language modeling loss is the cross-entropy loss for predicting masked lexical terms under visual feature conditions; and the joint optimization combines the two losses in a weighted summation manner at a preset ratio.

6. A two-stage visual fusion multimodal translation system, characterized in that, include: The visual feature extraction module is used to extract fine-grained regional visual features and coarse-grained global visual features from the input image; The first fusion module is used to concatenate the fine-grained regional visual features with the text representation of the source language text in the sequence dimension to form a multimodal input sequence, and to encode the multimodal input sequence through a guided self-attention mechanism to output a multimodal encoded representation; The second fusion module is used to map the coarse-grained global visual features to the hidden layer dimension of the model through a linear projection layer, and copy and expand them along the sequence dimension to align them with the sequence length of the source language text; and to perform residual modulation on the multimodal encoding representation using a gating mechanism to output the feature representation after global visual gating fusion. The decoding module is used to autoregressively generate the target language translation based on the feature representation after global visual gating fusion.

7. The multimodal translation system with dual-stage visual fusion according to claim 6, characterized in that: The visual feature extraction module includes: A fine-grained feature extraction unit is used to extract the visual features of the fine-grained region using the MDETR model; The coarse-grained feature extraction unit is used to extract the coarse-grained global visual features through the CLIP model.

8. The multimodal translation system with dual-stage visual fusion according to claim 6, characterized in that: The guided self-attention mechanism in the first fusion module is based on a guiding matrix determined by the alignment results of the MDETR model, which restricts cross-modal interactions in self-attention computation to occur only between semantically related text terms and image regions.

9. The multimodal translation system with dual-stage visual fusion according to claim 6, characterized in that: The gating mechanism in the second fusion module is specifically used for: The copied and expanded coarse-grained global visual features are concatenated with the portion of the multimodal encoded representation corresponding to the source language text. The concatenated features are input into a multilayer perceptron, and a gating value is generated using the tanh activation function. The gated value is multiplied element-wise with the copied and expanded coarse-grained global visual features, and then the residual is added to the part of the multimodal encoded representation corresponding to the source language text to obtain the feature representation after global visual gating fusion.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the multimodal translation method as described in any one of claims 1 to 4, or the training method as described in claim 5.

Images