Multi-modal machine translation method based on variational reasoning and multi-task learning

A multi-task learning and machine translation technology, applied in multi-modal machine translation based on variational reasoning and multi-task learning, in the field of machine translation, can solve problems such as not being effective, improve machine translation and reduce computational complexity Effect

Active Publication Date: 2020-12-01
EAST CHINA NORMAL UNIVERSITY
View PDF5 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0035] How to balance multiple loss functions of multi-task learning is a p

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-modal machine translation method based on variational reasoning and multi-task learning
  • Multi-modal machine translation method based on variational reasoning and multi-task learning
  • Multi-modal machine translation method based on variational reasoning and multi-task learning

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0121] The following is the implementation process of this embodiment:

[0122] 1, such as figure 1 As shown, first, the original data such as images and texts are preprocessed accordingly. Image preprocessing includes denoising, normalization, etc., and text preprocessing includes word piece segmentation, word embedding, etc., which are input to the RNN text as training data Feature Encoder and VGG-16 Image Feature Encoder.

[0123] 2. After obtaining the data, first specify the hyperparameters such as the learning rate, then initialize the parameters of the variational multimodal machine translation model, and finally learn these parameters.

[0124] 3. Get a new image or text data, and do corresponding preprocessing as the training data, as the test data.

[0125] 4. Use a zero-matrix mask for other modalities, input the mask and the test data extracted above into the model, and use the previously learned variational multimodal machine translation model to obta...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a multi-modal machine translation method based on variational reasoning and multi-task learning. Before the method has translation capability, multi-task modeling is performedon multi-modal information such as images, texts and the like by using a variational reasoning theory, and then a variational multi-modal machine translation model is obtained by training under the condition of giving a sufficient training set, so that the machine translation capability is obtained. And finally, predicting a plurality of translated texts through bundle search and maximum likelihood. The innovation point of the invention lies in that a model capable of integrating multi-modal information such as images into machine translation, namely variational multi-modal machine translation, is created and used. According to the variational model, a set of feature extraction neural network framework confusing image and text semantics is constructed, a modeling process and a self-learning updating process are derived at the same time, a detailed derivation algorithm is given, and an application method is given in an instructive mode.

Description

technical field [0001] The present invention relates to the field of computer technology and machine translation technology, in particular to a multi-modal machine translation method based on variational reasoning and multi-task learning. Background technique [0002] The background technology involves four major blocks: variational inference and variational encoder-decoder framework, information bottleneck theory, multi-task learning, and neural machine translation. [0003] 1) Variational Inference and Variational Encoder-Decoder (Variational Inference and VariationalEncoder-Decoder) [0004] Variational inference is a common approximate inference technique. Variational encoder-decoder is an important application in the field of machine translation. The variational encoder-decoder is generalized from the variational auto encoder (Variational Auto Encoder), which maps the input data x to different output data y. Different from the general encoder-decoder framework, the v...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F40/58G06N3/04G06N3/08G06N5/04
CPCG06F40/58G06N3/049G06N3/08G06N5/04G06N3/045
Inventor 孙仕亮刘啸赵静张楠
Owner EAST CHINA NORMAL UNIVERSITY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap