Deep theme model-based text image multi-mode retrieval method

Abstract

The invention discloses a deep theme model-based text image multi-mode retrieval method which can be used for text and image multi-mode retrieval. The method comprises the following steps of: (1) preprocessing training data and test data; (2) initializing a hyper-parameter and a share parameter of a deep theme model; (3) training the deep theme model; (4) training a classifier by using a joint feature; and (5) carrying out test by using the test data. According to the method, deep-to-shallow relation between different mode hidden layers is mined by utilizing the deep theme model, and a joint feature which comprises multi-mode information is obtained to carry out retrieval.

Description

technical field [0001] The invention belongs to the technical field of image processing, and further relates to a text image multimodal retrieval method based on a deep topic model in the technical field of artificial intelligence. The invention can be used to mine the deep connection between two different modalities of image and text, extract joint features, and use the extracted joint features to retrieve the text-image. Background technique [0002] The multi-modal retrieval technology uses joint learning of different modal features, and excavates the connection between different modal features to obtain joint features containing multi-modal information, so that different modal data can be mutually generated. Deep Topic Model PGBN (Poisson Gamma Belief Network) is a deep topic model based on Bayesian framework. The PGBN model has a multi-layer network structure, which can extract multi-layer features of data, and is superior to traditional topic models in text processing...

AI Technical Summary

Problems solved by technology

Although this method can mine the connection between different modal features, there are still shortcomings in this method: due to the "black box" characteristics of the deep neural network, the hidden layer unit of the multi-layer restricted Boltzmann machine Limited to binary values ​​and limited expression ability, the multimodal technology based on deep neural network cannot accurately describe the deep connection between different modal features. Linear mapping, it is difficult to visualize the relationship between the hidden layer and the observation

Although this method can directly establish a probabilistic model for multi-modal input, and transform the joint feature representation problem into the hidden layer distribution inference problem of the Bayesian model, the disadvantages of this method are: limited by the traditional Topic models are all shallow models, which are limited to building shallow connections between different modalities, and cannot dig deeper connections between modalities, thus affecting retrieval performance

Images