Multi-modal data fusion method based on compound collaborative structure feature recombination network

Abstract

The invention provides a multi-modal data fusion method based on a compound collaborative structure feature recombination network. The existing multi-modal data fusion technology is mainly characterized by direct feature fusion, neglects bidirectional interaction between modals, and has the problem of semantic gap between features during multi-modal fusion. In order to solve said problems, the method of the invention comprises the following steps: extracting image and text single-mode features by using a deep neural network to establish an image-text bidirectional interaction attention model based on a transform mechanism; mining feature relation between images and texts, performing multi-modal semantic association, and introducing a compound collaborative structure network to deepen communication of interaction information between modals; performing feature two-way recombination under multi-modal deep fusion, so that image and text semantic space alignment is realized. The invention better adapts to search of complementary information among different modals by a neural network, the understanding and generalization ability of a model to multi-modal semantics is enhanced, and the classification accuracy of a multi-modal feature network is further improved.

Description

technical field [0001] The invention relates to the field of feature extraction in deep learning, in particular to a multimodal data fusion method based on a complex collaborative structure feature recombination network. Background technique [0002] In the field of deep learning, due to the single training and prediction of single-modal data, there will be cases where the data utilization rate is low and the correlation between the attributes of the object of interest cannot be reflected. Therefore, it is necessary to establish the correlation and comprehensive analysis of these data from multiple dimensions, and achieve better results through multi-modal feature fusion. The fusion of feature levels means that different multimodal features are associated in the early stage, and the combined features are pulled into subsequent unified analysis to provide better information supplementation, but there is a problem of semantic asynchrony between different modal features. For m...

AI Technical Summary

Problems solved by technology

The fusion of feature levels means that different multimodal features are associated in the early stage, and the combined features are pulled into the subsequent unified analysis to provide better information supplementation, but there is a problem of semantic asynchrony between different modal features

For multi-modal feature fusion technology, the existing methods generally only perform simple splicing, dot multiplication, and addition operations on the features of each modality, without considering the information interaction between the modalities, and the semantic gap between features is not considered. Difficulties are not solved enough, and the modal generalization ability is not strong, so it is urgent to explore more effective fusion methods

Images