Cross-modal image text retrieval method of hybrid fusion model

Abstract

The invention discloses a cross-modal image text retrieval method of a hybrid fusion model. The method comprises the following steps of: in an early fusion structure, firstly combining local visual area features and original global features of a text to obtain a unified cross-modal fusion representation, and then taking the fusion features as input, enhancing interaction between the local visual features and the language information in a subsequent embedded network; meanwhile, on the basis of a traditional late fusion structure, inputting an original image and sentence features into a vision encoder and a text encoder respectively for intra-modal feature enhancement, and enriching semantic information of respective modals; and finally, the whole network similarity is a weighted linear combination of early fusion similarity and late fusion similarity so that complementation of early fusion in a cross-modal learning layer and late fusion in an intra-modal learning layer is realized, and potential alignment between image and text modals is completed.

Description

technical field [0001] The invention belongs to the technical field of cross-modal retrieval, and more specifically relates to a cross-modal image text retrieval method of a hybrid fusion model. Background technique [0002] Cross-modal retrieval means that users retrieve semantically relevant data in all modalities by inputting query data in any modality. With the increasing amount of multi-modal data such as text, images, and videos in the mobile Internet, retrieval across different modalities has become a new trend in information retrieval. Realizing fast and accurate image-text retrieval has great application value and economic benefits. [0003] Since computer vision features from image data and language features from text data naturally have a "heterogeneous gap" in data distribution and underlying feature representation, how to measure the high-level semantic correlation between images and text is still a challenge. The solution of the current method is usually to fu...

AI Technical Summary

Problems solved by technology

[0006] At present, the mainstream cross-modal retrieval method adopts the late fusion strategy, and the image and text data are respectively embedded and coded by using a more complex network design. Such methods often have the problem of insufficient cross-modal learning, and at the same time, the calculation cost is high.

On the other hand, the existing early fusion methods are often relatively rough, and can only fuse image and text data at the global level, and their performance is relatively average.

Images