Adaptive multi-modal data fusion prediction method and system based on dynamic graph convolution

The adaptive multimodal data fusion method based on dynamic graph convolution solves the problems of modal contribution imbalance and poor cross-domain robustness in multimodal data analysis, and achieves more accurate representation of individual emotional states and wider application adaptability.

CN122654992APending Publication Date: 2026-08-28启元实验室
4 Cites 0 Cited by

Patent Information

Application Number
CN202611148939.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-31
Publication Date
2026-08-28

AI Technical Summary

Technical Problem

Existing multimodal data analysis techniques suffer from an imbalance in modal contributions during feature fusion, particularly an over-reliance on text features, making it difficult to capture deep conflicts or synergistic relationships between modalities. Furthermore, they exhibit poor robustness across data domains, lack alignment mechanisms for personality-emotional state representations, and lack effective loss constraints, resulting in insufficient model generalization ability.

Method used

An adaptive multimodal data fusion method using dynamic graph convolution is adopted. By dynamically adjusting the coupling strength between modalities, semantic alignment is achieved by using contrastive learning and individual weighted regression constraints. Cross-modal feature fusion is performed by combining multi-head attention interaction and dynamic graph convolution. Single-modal prediction loss constraints are introduced to balance the contributions of each modality.

Benefits of technology

It achieves dynamic balance and higher-order reasoning between modalities, improves the modeling accuracy and cross-scene generalization ability of individual emotional state representation, and performs particularly well when dealing with language data with complex acoustic parameters and rich modal particles.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

A dynamic graph convolution-based adaptive multi-modal data fusion prediction method and system, the method comprising: extracting a text sentiment feature vector for representing sentiment semantics and a personality feature vector for representing individualized traits from the obtained text input, extracting an audio feature vector and a visual feature vector from the obtained audio and video input; after mapping the text sentiment feature vector and the personality feature vector to a shared space, performing semantic alignment by contrast learning constraint and personality weighted regression constraint to obtain an aligned text sentiment feature vector; taking the aligned text sentiment feature vector as a query vector, performing multi-head attention interaction on the audio feature vector and the visual feature vector respectively to obtain an audio enhanced feature vector and a visual enhanced feature vector; constructing a multi-node graph structure and realizing cross-modal feature fusion through dynamic graph convolution to obtain a fusion feature matrix; and generating a prediction result using the fusion feature matrix.
Need to check novelty before this filing date? Find Prior Art