Multi-modal fusion method and device based on normalized mutual information, medium and equipment

Abstract

The invention provides a multi-modal fusion method and device based on normalized mutual information, a medium and equipment. The method comprises the following steps: acquiring a plurality of modal data sets of a human body, wherein data in each modal data set is provided with a label; preprocessing the various modal data sets; respectively carrying out feature extraction on the preprocessed various modal data sets; obtaining width learning feature mapping of each modal data set through a width learning system; determining a multi-modal fusion mode of normalized mutual information; training and testing a width learning system; and carrying out modal feature fusion and final decision output according to the trained and tested multi-modal fusion mode and the discrimination architecture model. The method is high in training speed and low in resource consumption, and can quickly construct an incremental learning model; information complementation between modals can be realized, and redundant modal information can be reduced; and the method has good reliability, accuracy and robustness.

Description

technical field [0001] The present invention relates to the technical field of data processing, and more specifically, to a multimodal fusion method, device, medium and equipment based on normalized mutual information. Background technique [0002] Multi-modal fusion is the process of comprehensively utilizing two or more modal information (such as text sentence modality, facial visual expression, voice modality and physiological data modality) for target prediction (classification or regression). Other common aliases exist, such as multi-source information fusion, multi-sensor fusion. According to the level of fusion, multi-modal fusion can be divided into three types: data layer, feature layer and decision-making layer, which respectively correspond to fusion of original data, fusion of abstract features and fusion of decision results. The fusion of feature layers can occur in the early and late stages of feature extraction, and of course there are methods of mixing multi...

AI Technical Summary

Problems solved by technology

However, the problems of difficult training and reasoning reflected by the current deep learning algorithms in the face of different modal data types cannot be well solved

Overall, the deep neural network still does not have a perfect processing method for cross-modal comprehensive data.

Images