Adaptive cross-modal attention and knowledge preservation multimodal data fusion method

Abstract

This invention relates to the field of data fusion technology for large models, specifically to an adaptive cross-modal attention and knowledge preservation multimodal data fusion method. The method includes the following steps: preprocessing and spatiotemporal / semantic alignment of multimodal data; extracting features through a modality-specific encoder and projecting them into a unified embedding space; dynamically learning modality importance weights using an adaptive cross-modal attention module and fusing features within a Transformer layer; controlling the backflow of multimodal information through a gating mechanism in a knowledge preservation layer to prevent catastrophic forgetting; and dynamically generating a cross-modal interaction graph to quantify intermodal dependencies. This invention introduces an adaptive cross-modal attention and knowledge preservation framework that dynamically fuses information from different modalities using a large model backbone, while explicitly addressing the catastrophic forgetting problem and enhancing interpretability.

Description

Technical Field

[0001] This invention relates to the field of data fusion technology for large models, and more specifically, to an adaptive cross-modal attention and knowledge preservation multimodal data fusion method. Background Technology

[0002] Multimodal data fusion involves integrating information from multiple sources or modalities to enhance decision-making, improve prediction accuracy, and provide a more comprehensive understanding of complex phenomena. This approach leverages the strengths of various data types, such as text, images, audio, and sensor data, to achieve richer and more detailed representations of underlying information.

[0003] The evolution of traditional fusion methods from early-stage, late-stage, to hybrid fusion demonstrates the ongoing challenge in balancing deep intermodal interactions (the advantage of early-stage fusion) with modularity and dimensionality management (the advantage of late-stage fusion). This suggests that single, traditional approaches are not universally applicable, highlighting the fundamental trade-offs that require advanced models to address.

[0004] Despite the progress made by traditional multimodal data fusion techniques in many fields, they still face the following significant limitations:

[0005] Dimensionality issues and lack of flexibility: Early feature fusion can lead to a high-dimensional feature space, especially when combining multiple modalities, making it difficult for the model to generalize effectively. Furthermore, once features are fused, modifying or removing specific modalities becomes difficult, requiring a re-evaluation of the entire feature extraction process, resulting in a lack of flexibility.

[0006] Intermodal information loss: Due to the independent processing of each modality, post-fusion may fail to capture the crucial relationships and interactions between different modalities, resulting in poor performance in scenarios where intermodal interactions are critical to performance.

[0007] Increased complexity: Training multiple independent models increases the overall complexity of the later fusion system, requiring more computational resources and time.

[0008] Data alignment and synchronization: A fundamental challenge faced by all traditional fusion methods is that, due to differences in data collection time and format, it is difficult to ensure that data from different modalities are correctly aligned in time and space.

[0009] Missing data handling: Incomplete or missing data from one or more modalities may disrupt the fusion process and negatively impact model performance, which requires robust imputation techniques to address.

[0010] Computational complexity: Processing and fusing large, diverse datasets presents high computational demands, requiring efficient algorithms and substantial computational resources.

[0011] In recent years, advancements in machine learning, particularly deep learning, have significantly propelled the development of multimodal data fusion methods. Large Language Models (MLMs) have brought about transformative changes in the field of natural language processing, demonstrating exceptional capabilities in language understanding and generation. The emergence of MLLMs marks a shift in the fusion paradigm from explicit fusion strategies to implicit, learned fusion within large models. This indicates that the "fusion network" is no longer a separate, manually designed component, but rather deeply embedded in the Transformer architecture through shared self-attention mechanisms and adapter modules that project different modalities into a common latent space.

[0012] While MLLMs have made significant progress in multimodal understanding, they have also introduced new and critical challenges.

[0013] Catastrophic forgetting (text-specific forgetting): During continuous training with multimodal instructions, MLLMs catastrophically forget the text-specific instructions initially learned by the LLM, leading to a sharp decline in performance in text-specific evaluation. This forgetting is related to the shift of cross-layer attention from pre-image text to post-image text. This phenomenon indicates that during multimodal learning, newly introduced modal information may destroy or overwrite the model's previously learned unimodal knowledge. This is not only a data problem but also a deep challenge involving architecture and training strategies. Therefore, a mechanism is needed in the multimodal fusion process to differentiate or adaptively adjust the learning paths of different modalities to prevent knowledge corruption.

[0014] Insufficient generalization ability and reasoning depth: Current MLLMs, even those designed for specific domains, fall far short of the generalization ability required to achieve artificial general intelligence (AGI). They often perform poorly in tasks requiring deep, multi-step reasoning, especially when dealing with abstract concepts, and struggle to maintain logical consistency in complex tasks.

[0015] Error propagation: An error originating from one modality (e.g., a misinterpretation of a diagram) can propagate throughout the reasoning process, leading to incorrect conclusions, especially in complex, multi-step problems.

[0016] Illusion: MLLMs may generate information that appears reasonable but is actually inconsistent with the input data or has no factual basis, which can damage their credibility and reliability, and is particularly problematic in fields such as scientific reasoning where accuracy is extremely important.

[0017] Interpretability issues: The complexity and scale of MLLMs pose significant challenges to understanding how different data types are combined, how their interactions affect the output, and how the internal structure transforms input into output. This "black box" nature hinders the establishment of trust in high-risk applications. The interpretability challenges of multimodal large language models are exacerbated by the fusion process itself, as understanding how the model fuses different modalities and influences its decisions becomes more difficult. This suggests a need for an approach that is inherently interpretable within the fusion mechanism, rather than relying solely on post-hoc analysis. Summary of the Invention

[0018] The purpose of this invention is to provide an adaptive cross-modal attention and knowledge preservation multimodal data fusion method to solve the problems mentioned in the background art.

[0019] To achieve the above objectives, the present invention aims to provide an adaptive cross-modal attention and knowledge preservation multimodal data fusion method, comprising the following steps:

[0020] S1. Preprocess and align multimodal data spatiotemporally / semantically;

[0021] S2. Extract features using a modality-specific encoder and project them into a unified embedding space;

[0022] S3. Dynamically learn modality importance weights using an adaptive cross-modal attention module and fuse features within the Transformer layer;

[0023] S4. Control the backflow of multimodal information through the gating mechanism of the knowledge retention layer to prevent catastrophic forgetting;

[0024] S5. Dynamically generate cross-modal interaction graphs and quantify intermodal dependencies;

[0025] S6. Training outputs task prediction results based on the fused representations.

[0026] As a further improvement to this technical solution, the specific operation steps of step S1 are as follows:

[0027] Perform word segmentation and embedding on the text;

[0028] Features are extracted from images / videos using a pre-trained visual encoder;

[0029] Extract acoustic features from audio;

[0030] Dynamic time warping (DTW) is used for alignment of time series modes, and a projection mechanism is used for semantic alignment of static modes.

[0031] As a further improvement to this technical solution, the adaptive cross-modal attention module in step S3 is implemented as follows:

[0032] Calculate the query Q, K, V matrices at the Transformer layer;

[0033] Learn modal dynamic weights based on the current context. ;

[0034] Adjusting attention scores using modality masks:

[0035]

[0036] in, It is a dynamically generated mask, by generate.

[0037] As a further improvement to this technical solution, the gating mechanism of the knowledge retention layer in step S4 includes:

[0038] Calculate the gated scalar: ;

[0039] Calculate residual update: ;

[0040] Update hidden status: .

[0041] As a further improvement to this technical solution, the method for constructing the cross-modal interaction graph in step S5 is as follows:

[0042] The nodes are token sequences for each modality. ;

[0043] The edge weights are determined by the normalized cross-modal attention scores:

[0044]

[0045]

[0046]

[0047] Among them, text tokens and image tokens The edge between The attention scores derived from them .

[0048] As a further improvement to this technical solution, the representation training in step S6 includes:

[0049] Use a task-specific loss function;

[0050] The parameters of the pre-trained model are optimized by combining the LoRA fine-tuning strategy.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] 1. This invention uses a large model as the core fusion unit, leveraging its powerful self-attention mechanism and deep network structure to surpass traditional methods and capture deeper, nonlinear, and dynamically changing complex semantics and feature associations in multimodal data, thereby forming a more comprehensive and accurate unified multimodal representation.

[0053] 2. The hierarchical dynamic modal weight allocation mechanism introduced in this invention enables the model to intelligently evaluate and selectively utilize information from different modalities. The contributions of low-quality modalities, those with noise, or those that are partially missing can be effectively suppressed, thereby significantly improving the robustness of the entire fusion system in complex real-world data environments.

[0054] 3. By adopting an incremental learning strategy based on residual adapters, this invention can maximize the retention of the rich knowledge and powerful reasoning ability of large models pre-trained on a single modality while introducing multimodal capabilities. This avoids the catastrophic forgetting problem common in traditional fine-tuning and ensures that the model can give full play to its original advantages in multimodal tasks.

[0055] 4. This invention provides a new approach to understanding model decision-making by dynamically assigning weights to modal gating networks and attention mechanisms. Researchers or users can analyze which modalities are considered more important under specific tasks or inputs, and how they interact, based on gating values ​​and attention weights, thereby improving the transparency and credibility of the model. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the steps of the adaptive cross-modal attention and knowledge preservation multimodal data fusion method of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] In one specific embodiment, such as Figure 1As shown, this invention provides an adaptive cross-modal attention and knowledge-preserving multimodal data fusion method, comprising several interconnected modules: a set of modality-specific encoders, an adaptive cross-modal attention (ACMA) module, a knowledge-preserving layer (KPL), a cross-modal interaction graph (CMIG) generator, a large model backbone (e.g., a Transformer-based decoder-only LLM), and a task-specific prediction head. This architecture aims to facilitate rich cross-modal interactions while maintaining modality-specific computation and preserving pre-trained knowledge.

[0059] Specifically, the following steps are included:

[0060] (1) Data preprocessing and modal alignment.

[0061] Diverse multimodal data (e.g., text, images, audio, video, sensor data) are first subjected to modality-specific preprocessing.

[0062] Text: Perform word segmentation and embedding (RoBERTa embedding).

[0063] Images: Resize and normalize them, and extract features using a pre-trained visual encoder (based on CLIP architecture).

[0064] Audio / Video: Feature extraction (spectral analysis, acoustic features, frame-level visual features).

[0065] Alignment and Synchronization: To address the challenges of data alignment and synchronization, this invention employs a time alignment module. For time-series data (audio, video, sensor data), Dynamic Time Warping (DTW) is used to align features to a common time reference. For static modalities (text, images), a projection mechanism is used to map them to a shared latent space, ensuring semantic alignment.

[0066] (2) Feature extraction and representation learning.

[0067] a. Calculate characteristics.

[0068] Each preprocessed modal input in They are fed into their respective single-mode encoders The specific features of these encoder generation modes were calculated. , ,in For sequence length, For feature dimensions.

[0069] b. Projection calculation.

[0070] A projection layer These modality-specific features are then mapped to a unified embedding space compatible with the large model backbone. The calculation method is as follows This creates a set of modality-specific token sequences, preparing for fusion.

[0071] (3) Adaptive cross-modal attention module (ACMA module) and knowledge retention layer (KPL).

[0072] This mechanism integrates features from different modalities within the large model backbone (usually a Transformer). Unlike simple concatenation or fixed cross-attention, the ACMA module dynamically learns the importance and interactions between different modalities at each layer of the Transformer.

[0073] a. Attention score calculation.

[0074] The input to each Transformer layer includes the concatenated modal token: ,pass Calculate Q, K, and V. For a given attention head, calculate the attention score: .

[0075] b. Modal importance weighting.

[0076] This mechanism learns a set of dynamic weights for each modality m in each layer based on the current context and the relevance of the modality to the task. The attention output for each token is a weighted sum of its values, where the weights are influenced by... The impact is calculated as follows: .in It's a dynamically generated mask that emphasizes or de-emphasizes attention to specific modality tokens based on learned importance scores. This allows the model to adaptively focus on the most relevant modalities for a given input or task. This mechanism goes beyond standard attention by introducing an explicit learning component for modality importance into the attention computation. This is technically superior to general attention, which may implicitly learn this importance but cannot directly control or explain its learning process. This approach is more robust to imbalanced data and provides a pathway to interpretability by observing these learned weights to understand the model's decision-making process.

[0077] c. Knowledge retention layer.

[0078] To mitigate catastrophic forgetting, KPL is integrated in parallel with the main attention block in each layer of the Transformer, similar to the concept of Low-Rank Residual Attention (LoRRA). However, in this invention, KPL operates as a "gating mechanism," selectively allowing or restricting the backflow of newly learned multimodal interactions into the knowledge base of the core LLM.

[0079] set up This represents the hidden state of the LLM backbone at layer l. This is the output of the ACMA module. KPL calculates a gated scalar. and a residual update :

[0080]

[0081]

[0082] Updated hidden status

[0083]

[0084] KPL's gating mechanism is a more sophisticated knowledge retention method than simple parallel learners. It implies a learned "veto" or "filter" that prevents harmful updates, directly addressing the root cause of catastrophic forgetting by controlling the information flow. By dynamically controlling the information flow, the model can achieve a better balance between learning new multimodal patterns (plasticity) and retaining old unimodal knowledge (stability).

[0085] (4) Interpretable module: Cross-modal interaction graph generator (CMIG).

[0086] To enhance interpretability, a CMIG is dynamically constructed at each layer of the Transformer. This graph represents the dependencies and influences between the learned modalities and their features. The nodes of the graph are tokens from each modality. Edges represent attention weights between tokens of different modalities. For example, text tokens. and image tokens The edge between The attention scores derived from them After normalization or thresholding.

[0087]

[0088]

[0089]

[0090] This graph provides a quantitative representation of how different modalities interact and influence each other's representations during the fusion process. It can be used for post-hoc analysis and even as a feedback mechanism during training. CMIG is a novel approach to achieving interpretability during fusion. It transforms abstract attention weights into structured, visual graphs, making "black box" models more transparent by explicitly showing intermodal relationships. By explicitly representing intermodal attention weights as a graph, the model's "reasoning" process when handling cross-modal cues becomes observable. This provides a powerful diagnostic tool for understanding model behavior, identifying potential biases, and validating decisions in high-risk applications, elevating interpretability from a post-hoc consideration to an intrinsic feature of the fusion architecture.

[0091] (5) Model training and optimization.

[0092] The CMAP model is trained end-to-end using multimodal datasets (e.g., image-text pairs, video-audio-text data).

[0093] Loss function: Use task-specific loss (e.g., cross-entropy loss for classification, mean squared error for regression).

[0094] Knowledge Preservation Auxiliary Loss: An auxiliary loss term can be introduced to explicitly encourage KPL to maintain single-modal performance. For example,

[0095] A text-specific loss can be calculated using a small subset of plain text data and then combined with the main multimodal loss.

[0096] Optimization: A standard deep learning optimizer (AdamW) is used. LoRA or QLoRA fine-tuning strategies can be applied to effectively update model parameters while utilizing pre-trained LLM weights.

[0097] Data augmentation: Modality-specific and cross-modal data augmentation techniques are employed to enhance robustness and generalization ability.

[0098] During inference, new multimodal inputs are preprocessed and encoded. The encoded features are then fed into the ACMAP system. The ACMAP module dynamically weights the modalities, and KPL ensures knowledge consistency. The large model backbone processes the fused representations to generate the desired outputs (e.g., predictions, generation, responses). CMIG can be selectively generated during inference to provide real-time interpretability.

[0099] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An adaptive cross-modal attention and knowledge preservation multimodal data fusion method, characterized in that, Includes the following steps: S1. Preprocess and align multimodal data spatiotemporally / semantically; S2. Extract features using a modality-specific encoder and project them into a unified embedding space; S3. Dynamically learn modality importance weights using an adaptive cross-modal attention module and fuse features within the Transformer layer; S4. Control the backflow of multimodal information through the gating mechanism of the knowledge retention layer to prevent catastrophic forgetting; The gating mechanism of the knowledge retention layer in step S4 includes: Calculate the gated scalar: ; Calculate residual update: ; Update hidden status: ; S5. Dynamically generate cross-modal interaction graphs and quantify intermodal dependencies; The method for constructing the cross-modal interaction graph in step S5 is as follows: The nodes are token sequences for each modality. ; The edge weights are determined by the normalized cross-modal attention scores: ； ； ； S6. Training outputs task prediction results based on the fused representations.

2. The adaptive cross-modal attention and knowledge preservation multimodal data fusion method according to claim 1, characterized in that, The specific operation steps of step S1 are as follows: Perform word segmentation and embedding on the text; Features are extracted from images / videos using a pre-trained visual encoder; Extract acoustic features from audio; Dynamic time warping (DTW) is used for alignment of time series modes, and a projection mechanism is used for semantic alignment of static modes.

3. The adaptive cross-modal attention and knowledge preservation multimodal data fusion method according to claim 1, characterized in that, The adaptive cross-modal attention module in step S3 is implemented as follows: Calculate the query Q, K, V matrices at the Transformer layer; Learn modal dynamic weights based on the current context. ; Adjusting attention scores using modality masks: ； in, It is a dynamically generated mask, by generate.

4. The adaptive cross-modal attention and knowledge preservation multimodal data fusion method according to claim 1, characterized in that, The representation training in step S6 includes: Use a task-specific loss function; Add an auxiliary loss term and use plain text data to supervise unimodal knowledge preservation; The parameters of the pre-trained model are optimized by combining the LoRA fine-tuning strategy.

Images