Multi-modal data fusion method and system based on discriminant multi-modal deep confidence network

Abstract

The invention discloses a multi-modal data fusion method based on a discriminant multi-modal deep confidence network. The multi-modal data fusion method based on the discriminant multi-modal deep confidence network comprises the steps that the discriminant multi-modal deep confidence network is established; for the deep confidence network corresponding to multi-modal data, the weight of the network after the deep confidence network is optimized is obtained by means of limited Boltzmann machines; objective functions of the multi-modal Boltzmann machines are minimized by means of the alternative optimization strategy, the weights of the optimized Boltzmann machines are obtained, and a final discriminant multi-modal deep confidence network model is obtained; the multi-modal data to be fused are input into the deep confidence network model, and then a fusion result is obtained. The invention further discloses a multi-modal data fusion system based on the discriminant multi-modal deep confidence network. According to the multi-modal data fusion method and system based on the discriminant multi-modal deep confidence network, monitored label information is introduced into a traditional multi-modal deep confidence network, the relations between the data with different modals are mined in a discriminant mode, and thus the high accuracy rate can be guaranteed during a large-scale multi-modal data classifying and searching task.

Description

technical field [0001] The invention relates to the fields of pattern recognition and machine learning, in particular to a multimodal data fusion method and system based on a discriminative multimodal deep belief network. Background technique [0002] A concept or content can usually be represented by multiple modal data. For example, an image and its corresponding text annotation are two modal data representing the same content; when people are reading, lip movements and corresponding sounds are also two modal data. corresponding modal data. Extensive works have confirmed that multiple modality data can provide a more comprehensive description of concepts than single modality data, which can potentially help some common pattern recognition problems, such as classification and retrieval. The general practice of multimodal data fusion is to fuse multimodal data into the same expression, and then this common expression can be further used for subsequent classification or retr...

AI Technical Summary

Problems solved by technology

In fact, since each modality contains its modality-related characteristics, it will have a negative impact on the data fusion process, which will greatly reduce the quality of the final data fusion; another type of work is to use "deep" layers Although these models can handle the complex characteristics of multimodal data well, a large number of existing models are generative models, which are not suitable for discriminative tasks, such as classification and retrieval.

Images