Semantic representation model pre-training method and device, electronic equipment and storage medium

Abstract

The invention discloses a pre-training method and device for a semantic representation model, electronic equipment and a storage medium, and relates to the field of artificial intelligence. The specific implementation scheme is as follows: collecting a plurality of single-mode training data sets with different modes and a multi-mode training data set comprising a plurality of modes at the same time; respectively adopting the training data set of each single mode to pre-train the semantic representation model of the corresponding single mode; and based on the trained semantic representation model of each single mode, adopting a multi-mode training data set to train the fused multi-mode semantic representation model, wherein the multi-mode semantic representation model comprises a pluralityof single-mode semantic representation models and a fusion model. According to the method, the multi-modal semantic representation model can be trained in stages, and the expression accuracy of the multi-modal semantic representation model can be effectively improved.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence, and in particular to a pre-training method, device, electronic equipment and storage medium based on a multimodal semantic representation model. Background technique [0002] With the widespread application of semantic representation technology in the field of Natural Language Processing (Natural Langunge Possns; NLP), large-scale data pre-training (pre-train) learning general semantic representation and fine-tuning (fine-tune) learning on downstream tasks Paradigm, refreshing the effect on multiple tasks of NLP. The semantic representation technology for multimodal tasks has also received extensive attention recently, and has achieved improved effects on typical multimodal tasks such as visual question answering and image retrieval. [0003] In the current multi-modal semantic representation technology, for pre-training methods,...

AI Technical Summary

Problems solved by technology

[0004] However, the existing above-mentioned direct use of multi-modal alignment training data to pre-train the multi-modal semantic representation model leads to poor accuracy of the trained multi-modal semantic representation model

Images