Multimodal machine translation method, device, electronic device and storage medium

Abstract

The present invention provides a multimodal machine translation method, device, electronic device and storage medium. The method includes: determining a source language text to be translated; inputting the source language text into a translation model to obtain a target language output by the translation model Text; the translation model is obtained by joint reconstruction model training based on the sample source language text, the sample target language text, and the sample images matching the sample source language text; the translation model and the reconstruction model share the feature coding layer, and the feature coding during the model training process The layer is used to encode the first sequence and the second sequence, the translation model translates based on the encoding of the first sequence, the reconstruction model reconstructs based on the encoding of the second sequence, the first sequence is determined based on the sample source language text, and the second sequence is based on the sample source. The non-entity determination of each entity in the language text in the region image of the sample image and the sample source language text improves the interpretability of quality improvement and reduces the complexity of translation.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a multimodal machine translation method, device, electronic equipment and storage medium. Background technique [0002] Multimodal machine translation refers to the use of modal information other than text information to help text translation, such as using images to help improve the translation quality of image descriptions. The premise of this approach is that images contain more complete information than single sentences. [0003] The general multi-modal machine translation model is designed for the multi-modal environment of the text language mode and the static image mode. The data form is a picture with a sentence image description and the translation of the image description. The semantic fusion method adopted Generally, it can be divided into the following two categories: one is to input visual information into the translation system as the context of the sente...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a multi-modal machine translation method, device, electronic equipment and storage medium, which are used to solve the defect of lack of interpretability in the improvement of translation performance in the prior art, and realize the interpretability of improving translation quality

Images