Cross-modal processing method and device, electronic equipment and computer storage medium

Abstract

The invention discloses a cross-modal processing method and device, electronic equipment and a computer storage medium, and relates to the technical field of natural language processing. According tothe specific implementation scheme, the method comprises the steps that 1, acquiring sample sets, wherein the sample set comprises a plurality of corpora and a plurality of images; generating multipletraining samples, wherein each training sample is a combination of at least one corpus and at least one corresponding image; using multiple training samples, training semantic models, learning the semantic model to obtain a semantic vector of corpus and image combination; adopting a trained semantic model to carry out cross-modal processing between the corpora and the images; and combining the corpora and the corresponding images for training .The semantic model learns the semantic association between the corpus and the corresponding image, the training effect of the semantic model is improved, and the problems that in the prior art, during multi-modal processing, each modal is independently trained, semantic association between different modals is isolated, and the effect of the model obtained through training is poor are solved.

Description

Technical field [0001] This application relates to the field of computer technology, in particular to the field of natural language processing technology, and specifically to a cross-modal processing method, device, electronic equipment, and computer storage medium. Background technique [0002] The world we live in is a multi-modal world, and content of different modalities such as text and vision fills our lives. With the rapid development of artificial intelligence technology, the requirements and requirements for multi-modal processing, such as visual-language multi-modal processing, are getting higher and higher. [0003] However, the current multi-modal processing method cannot capture enough semantic information during model training, and at the same time, it does not establish a semantic association between the text and visual modalities, which makes the training effect of the model poor. Summary of the invention [0004] A method, device, electronic device and computer sto...

AI Technical Summary

Problems solved by technology

[0003] However, the current multimodal processing methods cannot capture enough semantic information during model training, and at the same time, do not establish the semantic association between the two modalities of text and vision, making the training effect of the model poor

Images