Training method and device of multi-modal model, equipment and storage medium

Abstract

Embodiments of the present disclosure relate to a multi-modal model training method, device and equipment and a storage medium. The multi-modal model training method comprises: obtaining paired image samples and text samples corresponding to a target task; performing region feature extraction and text recognition on the image samples to generate image region features and image text features respectively; inputting the text samples, the image region features and the image text features into a multi-modal pre-training model to generate output data corresponding to the target task; wherein the output data comprises output images and / or output texts; and training the multi-modal pre-training model based on sample true values corresponding to the output data and the image samples. According to the embodiments of the present disclosure, more effective information is provided for the multi-modal pre-training model to obtain a text-image alignment function, the number of training samples required for model training is reduced, and the model training efficiency is improved.

Images