Cross-modal understanding and generating method and device based on multi-modal pre-training model

Abstract

The invention provides a cross-modal understanding and generating method and device based on a multi-modal pre-training model, and the method comprises the steps of determining to-be-processed multi-modal information which comprises an image, a text and an audio; and inputting the multi-modal information to the multi-modal pre-training model, learning the correlation of the multi-modal information to obtain a fusion representation of the multi-modal information, and inputting the fusion representation to the understanding and / or generating unit to execute a cross-modal understanding and generating task to obtain an understanding result and / or a generating result. According to the method and the device provided by the invention, understanding and generation are carried out by combining three modalities of an image, a text and an audio, so that full application of information is realized. Through combination of two tasks of cross-modal understanding and cross-modal generation, the multi-modal pre-training model can perform feature extraction and cross-modal association construction more comprehensively, so that the accuracy of cross-modal understanding and generation is further improved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for cross-modal understanding and generation based on a multi-modal pre-training model. Background technique [0002] Multimodal pre-training is an interdisciplinary subject that spans multiple domains and involves multiple modal information. This task aims to train a unified framework on a large scale to achieve various cross-modal understanding and generation tasks, such as image recognition, image generation, visual question answering, text generation, etc. [0003] At present, when performing multimodal pre-training, common methods and frameworks only consider a single modality or two modalities, such as image and text, or video and text, and it is very easy to ignore other information that is prevalent in the surrounding environment. Effects on cross-modal comprehension and generation. Moreover, the current multimodality usually only focuses on cross-...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a cross-modal understanding and generation method and device based on a multi-modal pre-training model to solve the existing cross-modal understanding and generation problems

Images