Image-text cross-modal feature unentanglement method based on depth mutual information constraint

Abstract

The invention discloses an image-text cross-modal feature unentanglement method based on depth mutual information constraint. The method comprises the following steps: reading text and image files ina specified data mode; secondly, respectively extracting original features of the text and the image data by utilizing ResNet and BiGRU; then, under the effect of depth mutual information constraint,mapping the original features to a mixed feature space; finally, using the generative adversarial network to reconstruct the data to different degrees. By controlling the reconstruction process, the unentanglement of the cross-modal features is realized, so that the modal common information and the modal specific information are mapped to different feature spaces respectively. According to the method, the unentanglement features can be learned on large-scale image-text data, and only the features are unentangled, so that the retrieval accuracy is improved, and the depth features have better interpretability.

Description

technical field [0001] The invention belongs to the field of graphic-text cross-modal calculation, and in particular relates to a graphic-text cross-modal feature disentanglement method constrained by depth mutual information. Background technique [0002] Due to the rapid rise and development of social networks and short video platforms in recent years, multimedia data on the Internet has exploded. People urgently hope to find appropriate and effective methods to deal with these multimodal data. Cross-modal retrieval is the most basic and representative type of cross-modal data computing methods. [0003] The task of cross-modal information retrieval is that when people give data from one modality (such as images), the retrieval algorithm can return data from another modality (such as text modality) and query data through the processing and calculation of hardware devices. Related return results. However, there is large heterogeneity in data from different modalities. T...

AI Technical Summary

Problems solved by technology

Existing image-text cross-modal retrieval methods often map these two types of information into the learned feature representation at the same time, and the modality-specific information will degrade the performance of the model during the learning process.

Images