Image-text mutual retrieval method based on complementary semantic alignment and symmetric retrieval

Abstract

The invention belongs to the technical field of computer vision and natural language processing, and discloses an image-text mutual retrieval method based on complementary semantic alignment and symmetric retrieval, comprising: using convolution neural network to extract the depth visual features of images; Using the model of object-based convolutional neural network and scene-based convolutionalneural network to extract depth visual features to ensure that the visual features contain multiple complementary semantic information of the object and the scene; encoding the text by using short-term and long-term memory network, and extracting the corresponding semantic features. mapping visual features and text features into the same cross-modal embedding space by using two mapping matrices; Using the k-nearest neighbor method, retrieving the initial list in the cross-modal embedding space. Using the neighborhood relation of symmetrical bi-directional retrieval based on mutual nearest neighbor method, the initial retrieval list is reordered and the final retrieval level list is obtained. The invention has the advantages of high accuracy.

Description

technical field [0001] The invention belongs to the technical field of computer vision and natural language processing, and in particular relates to an image-text mutual retrieval method based on complementary semantic alignment and symmetrical retrieval. Background technique [0002] At present, the existing technology commonly used in the industry is as follows: the image-semantic description mutual retrieval task aims to retrieve a related text description sentence in the text library given a search image, or given a text description in the image library The corresponding image is retrieved from . It has important practical significance, such as helping the blind to "see" the world; in addition, this task is also regarded as a major challenge in image understanding and is a core issue in computer vision. Therefore, the image-semantic description mutual retrieval task is one of the hottest researches in the field of computer vision and natural language processing. At pre...

AI Technical Summary

Problems solved by technology

[0003] However, the problem in the existing technology is: the image features used in the existing methods only contain the target information of the image, ignoring the scene context information of the image, while the text features include both the target information and the scene context information, so with Image features and text features will have a large deviation when the embedding space is aligned

In addition, because the information in the text is highly condensed semantic information, and the image features contain richer semantic information, there will be a problem of asymmetric bidirectional retrieval results during retrieval. For example, a picture retrieved The first k sentences of these sentences, when reversely retrieving pictures for these sentences, the pictures may not appear in the first k retrieval results of the sentence, which will affect the accuracy of two-way retrieval

Images