Multi-step self-attention cross-media retrieval method and system based on limited text space

Abstract

The invention discloses a multi-step self-attention cross-media retrieval method and retrieval system based on a restricted text space, and the method comprises the steps: constructing the restrictedtext space of a relatively fixed vocabulary, and converting an unrestricted text space into the restricted text space; Extracting image features and text features of the limited text space through a feature extraction network; Wherein the features comprise global features, a regional feature set and associated features; Sending the extracted features into a feature mapping network, and extractingobject-level sharing information between the image and the text through a multi-step self-attention mechanism; Collecting useful information at each moment through a similarity measurement network tomeasure the similarity between the image and the text, and calculating a triple loss function. Therefore, multi-step self-attention cross-media retrieval based on the limited text space is realized. The cross-media retrieval recall rate is greatly improved by introducing a multi-step self-attention mechanism and associated characteristics.

Description

technical field [0001] The invention relates to the technical field of computer vision and information retrieval, in particular to a multi-step self-attention cross-media retrieval method and system based on a limited text space. Background technique [0002] In recent years, with the rapid development of information technology, multimedia data on the Internet has become more and more abundant, and multimedia data of different modalities (text, image, audio, video, etc.) can be used to express similar content. In order to meet the growing needs of users for multimedia retrieval, people propose a cross-media retrieval task to find an isomorphic semantic space (public space, text space, image space) to make the similarity between the underlying heterogeneous multimedia data can be directly measured. More precisely, the core problem of this cross-media retrieval task can be subdivided into two sub-problems. [0003] The first sub-problem is how to learn effective low-level fe...

AI Technical Summary

Problems solved by technology

However, most of the existing attention-based methods only consider the shared information at the object level between images and text, and do not consider the interactive information between objects.

[0004]The second sub-problem is how to find a suitable isomorphic feature space

If the additive or product self-attention mechanism is used in the cross-media retrieval algorithm, the focus information of images and texts cannot be fixed, resulting in the uncertainty of image and text encoding, which affects the practical application value of the algorithm

Images