Multi-modal pre-training model training method, application method and device thereof

Abstract

The invention provides a multi-modal pre-training model training method, and an application method and a device thereof. The multi-modal pre-training model training method comprises the following steps: constructing a multi-modal pre-training model of a double-tower structure; obtaining a positive sample data set comprising positive sample image-text pairs and a negative sample data set comprising negative sample image-text pairs; and training a multi-modal pre-training model according to the positive sample data set and the negative sample data set, wherein the multi-modal pre-training model comprises a cross-modal comparison learning module used for carrying out image-text similarity comparison learning on the positive sample image-text pair and the negative sample image-text pair. The multi-modal pre-training model adopts a double-tower structure and a cross-modal contrast learning algorithm, a large number of negative samples are constructed for image and text modals, the model expression ability is high, and the processing precision of image-text pairs is improved. According to the model, the overall similarity between images and texts is calculated, whether the images and texts correspond or not is judged according to the similarity, and on the basis of the image-text weak correlation hypothesis, the actual situation of semantic weak correlation between the images and texts in image-text pairs in actual application is better fit.

Description

technical field [0001] The application belongs to the field of computer application technology, and in particular relates to a training method, application method and device of a multimodal pre-training model. Background technique [0002] In recent years, pre-training models have become a hot topic in the field of Natural Language Processing (NLP, Natural Language Processing) research. Multimodal pre-training models involving multiple modal information interactions are suitable for more application scenarios. For example, multi-modal pre-training models for image-text pairs have gradually attracted widespread attention. [0003] At present, related technologies provide some multimodal pre-training models for processing image-text pairs. These multi-modal pre-training models assume that there is a strong semantic correlation between text and images in input image-text pairs. By judging whether the text includes Whether there is a semantic correspondence between the words in...

AI Technical Summary

Problems solved by technology

[0004] However, in practical applications, there may not be a strong semantic correlation between the image and the text in the picture-text pairing. The above-mentioned related technologies judge whether the text and the image correspond through the semantic correspondence between words and images, and the accuracy is very low.

Images