Multi-modal fusion method and device based on normalized mutual information, medium and equipment

A fusion method and multi-modal technology, applied in the field of data processing, can solve problems such as difficult training, lack of perfect processing methods for cross-modal comprehensive data, and difficult reasoning

Active Publication Date: 2020-07-28
SOUTH CHINA UNIV OF TECH
View PDF2 Cites 18 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the problems of difficult training and reasoning reflected by the current deep learning algorithms in the face of different modal d...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-modal fusion method and device based on normalized mutual information, medium and equipment
  • Multi-modal fusion method and device based on normalized mutual information, medium and equipment
  • Multi-modal fusion method and device based on normalized mutual information, medium and equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0063] In this embodiment, a multimodal fusion method based on normalized mutual information, such as figure 1 shown, including the following steps:

[0064] Step S1: Acquire multiple modal data sets of the human body, and the data in each modal data set has labels respectively; for various modal data sets, the total amount of data is the same, the label classification and division of the data are the same, and the data model state is not the same;

[0065] Step S2, preprocessing various modal data sets; performing feature extraction on the preprocessed various modal data sets, so as to obtain corresponding feature data that is beneficial to decision-making labels;

[0066] Step S3, obtain the width learning feature map of each modal data set through the width learning system; determine the multi-modal fusion method of normalized mutual information; use the feature data of various modal data sets to train and test the width learning system , to obtain the discriminant archit...

Embodiment 2

[0113] In order to realize the multimodal fusion method based on normalized mutual information described in Embodiment 1, this embodiment provides a multimodal fusion device based on normalized mutual information, including:

[0114] The multi-modal data set acquisition module is used to acquire multiple modal data sets of the collected human body, and the data in each modal data set has labels respectively; for each modal data set, the total amount of data is the same, and the data labels The classification and division are the same, but the data modes are different;

[0115] The feature extraction module is used to preprocess various modal data sets; perform feature extraction on the preprocessed various modal data sets, so as to obtain corresponding feature data that is beneficial to decision-making labels;

[0116] The width learning training and testing module is used to obtain the width learning feature map of each modal data set through the width learning system; determ...

Embodiment 3

[0119] A storage medium in this embodiment is characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the normalization-based A multimodal fusion method for mutual information.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a multi-modal fusion method and device based on normalized mutual information, a medium and equipment. The method comprises the following steps: acquiring a plurality of modal data sets of a human body, wherein data in each modal data set is provided with a label; preprocessing the various modal data sets; respectively carrying out feature extraction on the preprocessed various modal data sets; obtaining width learning feature mapping of each modal data set through a width learning system; determining a multi-modal fusion mode of normalized mutual information; training and testing a width learning system; and carrying out modal feature fusion and final decision output according to the trained and tested multi-modal fusion mode and the discrimination architecture model. The method is high in training speed and low in resource consumption, and can quickly construct an incremental learning model; information complementation between modals can be realized, and redundant modal information can be reduced; and the method has good reliability, accuracy and robustness.

Description

technical field [0001] The present invention relates to the technical field of data processing, and more specifically, to a multimodal fusion method, device, medium and equipment based on normalized mutual information. Background technique [0002] Multi-modal fusion is the process of comprehensively utilizing two or more modal information (such as text sentence modality, facial visual expression, voice modality and physiological data modality) for target prediction (classification or regression). Other common aliases exist, such as multi-source information fusion, multi-sensor fusion. According to the level of fusion, multi-modal fusion can be divided into three types: data layer, feature layer and decision-making layer, which respectively correspond to fusion of original data, fusion of abstract features and fusion of decision results. The fusion of feature layers can occur in the early and late stages of feature extraction, and of course there are methods of mixing multi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06K9/62
CPCG06F18/217G06F18/253G06F18/214
Inventor 陈变娜张通晋建秀陈俊龙
Owner SOUTH CHINA UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap