A multi-modal fusion method based on optimal transmission and information difference guidance

Abstract

The present application relates to the field of artificial intelligence and medical information processing technology, especially to a multi-modal fusion method based on optimal transmission and information difference guidance. The method is aimed at the joint modeling scene of at least two types of heterogeneous modal data. First, the first modal data is structured and preprocessed to extract instance-level features, and the second modal data is grouped and encoded for semantic representation. Then, the difference variational information bottleneck module is used to compress and de-redundant the main modal features, while retaining the task-related information and enhancing the difference in latent variable expression. When the second modal is missing, the optimal transmission is used to establish the global distribution matching relationship between the first and second modal, and the cross-modal reconstruction module is used to generate the reconstructed features with consistent structure. Further, cross-attention and Transformer are used for multi-modal feature fusion, and the task head is used to output the downstream task results. During the training process, the gradient update strength of the dominant modal is adaptively re-weighted according to the Fisher information contribution of each modal parameter to the target loss, so as to suppress the optimization deviation caused by the amplification of weak modal noise. The present application can simultaneously solve the problems of large modal feature redundancy, frequent modal missing and multi-modal training imbalance, and is suitable for various multi-modal fusion tasks. Among them, cancer survival prediction is a verification scene of the present application.

Images