Cross-modal deep hash retrieval method based on self-supervision

Abstract

The invention relates to a cross-modal joint hash retrieval method based on self-supervision. The method comprises the following steps: step 1, processing image modal data: carrying out feature extraction on the image modal data by adopting a deep convolutional neural network, carrying out Hash learning on the image data, and setting the number of nodes of the last full connection layer of the deep convolutional neural network as the length of a Hash code; step 2, processing the text modal data; using a word bag model for modeling text data, a two-layer full-connection neural network is established for feature extraction of text modal data, wherein the input of the neural network is a word vector represented by the word bag model, and the length of data of a first full-connection layer node is the same as that of data of a second full-connection layer node and a Hash code; step 3, for the neural network of category label processing, extracting semantic features from the label data by adopting a self-supervised training mode; and step 4, minimizing the distance between the features extracted from the image and the text network and the semantic features of the label network, so thatthe Hash model of the image and the text network can more fully learn the semantic features among different modals.

Description

technical field [0001] The invention belongs to the technical field of cross-modal deep hash retrieval, in particular to a cross-modal deep hash retrieval method based on self-supervision. Background technique [0002] Artificial intelligence technology has experienced many outbreaks and cold winters since its birth, and this outbreak of artificial intelligence technology is even more menacing, because compared with the previous outbreaks, it has a distinctive feature-big data as the core. Base. Big data is not only because of its large amount of data, but more importantly, its data types are diverse, and the value density of data is low. We generate and receive various information every day, and these information will be recorded, and then analyze our daily behavior and living habits through various artificial intelligence technologies, so as to provide various convenient services for our lives. Among the massive multimedia data, some data are not independent of each othe...

AI Technical Summary

Problems solved by technology

However, these methods ignore the semantic value of the most important tag data, only use tag data to generate a similarity matrix, do not make full use of tag data to describe finer-grained tag information, and more accurately describe the category data to which they belong

At the same time, because there is a data imbalance problem in the existing data sets, that is, the amount of similar data in different modalities is much smaller than the amount of dissimilar data, and the existing methods do not deal with the data imbalance problem, which easily leads to insufficient training Even the case of overfitting

At the same time, because hash codes need to be generated to represent data of various modalities, most of the existing methods add a sigmoid function to the last layer of the neural network to compress its output between 0 and 1, and then generate discrete Binary code, but the sigmoid function itself can easily lead to the gradient disappearance of the neural network during backpropagation, and after directly compressing the output value of the neural network to between 0 and 1, it will cause damage to both image and text data. certain loss of information

Images