Multi-modal machine translation method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-modal machine translation method and device, electronic equipment and a storage medium. The method comprises the steps: determining a to-be-translated source language text; inputting the source language text into a translation model to obtain a target language text output by the translation model, wherein the translation model is obtained based on a sample source language text, a sample target language text and a sample image matched with the sample source language text by combining training of a reconstruction model, the translation model and the reconstruction model share a feature coding layer, the feature coding layer is used for coding a first sequence and a second sequence in a model training process, the translation model performs translation based on coding of the first sequence, the reconstruction model performs reconstruction based on coding of the second sequence, the first sequence is determined based on a sample source language text, the second sequence is determined based on the regional image of each entity in the sample source language text in the sample image and the non-entity of the sample source language text, thereby improving the interpretability of the quality improvement and reducing the translation complexity.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a multimodal machine translation method, device, electronic equipment and storage medium. Background technique [0002] Multimodal machine translation refers to the use of modal information other than text information to help text translation, such as using images to help improve the translation quality of image descriptions. The premise of this approach is that images contain more complete information than single sentences. [0003] The general multi-modal machine translation model is designed for the multi-modal environment of the text language mode and the static image mode. The data form is a picture with a sentence image description and the translation of the image description. The semantic fusion method adopted Generally, it can be divided into the following two categories: one is to input visual information into the translation system as the context of the sente...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a multi-modal machine translation method, device, electronic equipment and storage medium, which are used to solve the defect of lack of interpretability in the improvement of translation performance in the prior art, and realize the interpretability of improving translation quality

Images