Multi-modal translation method and system for dual-stage visual fusion

By employing a two-stage visual fusion method, which utilizes the dual fusion of fine-grained and coarse-grained features, combined with guided self-attention and gated residual modulation, the problem of insufficient utilization of visual information in multimodal machine translation is solved, and a stable improvement in translation quality is achieved.

CN122263913APending Publication Date: 2026-06-23HENAN NORMAL UNIV
0 Cites 0 Cited by

Patent Information

Application Number
CN202610385636.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-03-27
Publication Date
2026-06-23

AI Technical Summary

Technical Problem

Existing multimodal machine translation methods struggle to consistently improve translation quality, fail to fully utilize visual information which is easily overlooked, and employ simplistic fusion mechanisms that hinder fine-grained alignment and robust global semantic supplementation.

Method used

A two-stage visual fusion method is adopted, which enhances cross-modal alignment and noise suppression by fusing fine-grained regional features and coarse-grained global features, combined with guided self-attention and gated residual modulation. Visual conditional masking language modeling is introduced as an auxiliary task.

Benefits of technology

It significantly improves translation quality, enhances the model's utilization of visual information, and improves the stability and robustness of translation, outperforming existing baseline models.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention provides a dual-stage visual fusion multi-modal translation method and system, and aims to solve the problems that visual information is insufficiently utilized, a fusion mechanism is single, and a model easily neglects the visual information in existing multi-modal translation. The method comprises the following steps: acquiring a source language text and a corresponding image, and extracting fine-grained region visual features and coarse-grained global visual features; in the first stage, after fine-grained region features and text representation are spliced, cross-modal alignment is achieved by guiding self-attention coding; in the second stage, after the coarse-grained global features are projected and broadcasted, scene-level prior information is injected through gating residual modulation; and jointly optimizing a multi-modal machine translation target and a visual condition mask language modeling target for training. According to the method, the translation quality is effectively improved through double-stage multi-granularity fusion, BLEU and METEOR indexes on data sets such as Multi30K and the like are remarkably superior to those of an existing method, and the method can be applied to the fields of multimedia translation, cross-language retrieval and the like.
Need to check novelty before this filing date? Find Prior Art