Image text matching method and device, electronic equipment and storage medium

Abstract

The invention relates to an image text matching method and device, electronic equipment and a storage medium. The method comprises the steps of acquiring image features and text features; performing image enhancement processing on the image features to generate two image features containing noise; performing text enhancement processing on the text features to generate two text features containing noise; processing the two noise-containing image features by using an image memory unit added with noise; processing the two noisy text features by using a noisy text memory unit; and respectively constraining the image memory unit and the text memory unit. According to the embodiment of the invention, through the key memory constraint, the semantic consistency between the image with the noise and the text memory is higher than other memories generated by different images and texts, so that the matching effect is improved; and through addition of two different noises and comparative learning, a good matching effect can still be obtained when image text input is interfered, and the matching robustness is improved.

Description

technical field [0001] The present invention relates to the technical field of image-text matching, in particular to an image-text matching method, device, electronic equipment and storage medium. Background technique [0002] At present, the image-text matching model generally consists of two main modules: feature extraction and feature retrieval. Feature extraction models usually use convolutional neural networks to extract image features, word vector models and cyclic neural networks to extract text features, and map the two features into the same multimodal space; feature retrieval models usually use distance metric functions to measure the distance between the query vector and other feature vectors in the multimodal space, resulting in a retrieval ranking table based on the distance scores. However, the above matching process is easily disturbed by polysemous words or locally similar images, which reduces the matching effect of image text; and when disturbed by text or...

AI Technical Summary

Problems solved by technology

However, the above matching process is easily disturbed by polysemous words or locally similar images, which reduces the matching effect of image text; and when disturbed by text or image noise, the retrieval effect will be greatly reduced

Images