Adaptive cross-modal attention and knowledge preserving multi-modal data fusion method

Abstract

The invention relates to the technical field of data fusion of large models, in particular to an adaptive cross-modal attention and knowledge preserving multi-modal data fusion method. The method comprises the following steps: carrying out preprocessing and space-time / semantic alignment on multi-modal data; extracting features through a modal specific encoder, and projecting the features to a unified embedding space; dynamically learning a modal importance weight by using an adaptive cross-modal attention module, and fusing features in a Transform layer; multi-mode information backflow is controlled through a gating mechanism of the knowledge keeping layer, and catastrophic forgetting is prevented; and dynamically generating a cross-modal interaction diagram, and quantifying the dependency relationship between the modals. According to the method, a self-adaptive cross-modal attention and knowledge preserving framework is introduced, the framework dynamically fuses information from different modals by using a large model backbone, meanwhile, the problem of disastrous forgetting is clearly solved, and the interpretability is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data fusion of large models, in particular to a self-adaptive cross-modal attention and knowledge preservation multi-modal data fusion method. BACKGROUND

[0002] Multi-modal data fusion involves integrating information from multiple sources or modalities to enhance decision-making, improve prediction accuracy, and provide a more comprehensive understanding of complex phenomena. This approach leverages the strengths of various data types, such as text, images, audio, and sensor data, to achieve a richer and more nuanced representation of underlying information.

[0003] The evolution of traditional fusion methods from early, late, to hybrid fusion indicates that the field continues to face challenges in balancing deep inter-modal interactions (the strengths of early fusion) and modularity and dimension management (the strengths of late fusion). This suggests that a single traditional approach is not universally applicable, highlighting the need for advanced models to address fundamental trade-offs.

[0004] Despite the progress made by traditional multi-modal data fusion techniques in many fields, they still face the following significant limitations: Dimensionality issues and lack of flexibility: Early fusion can result in high-dimensional feature spaces, especially when combining multiple modalities, making it difficult for models to effectively generalize. Moreover, once features are fused, modifying or removing specific modalities becomes difficult, requiring re-evaluation of the entire feature extraction process, lacking flexibility.

[0005] Inter-modal information loss: Late fusion, due to independent processing of each modality, may fail to capture crucial relationships and interactions between different modalities, leading to poor performance in scenarios where inter-modal interactions are critical.

[0006] Increased complexity: Training multiple independent models increases the overall complexity of late fusion systems, requiring more computational resources and time.

[0007] Data alignment and synchronization: A fundamental challenge faced by all traditional fusion methods is ensuring that data from different modalities is correctly aligned in time and space due to differences in data collection time points and formats.

[0008] Missing data handling: Incomplete or missing data from one or more modalities can disrupt the fusion process and negatively impact model performance, requiring robust imputation techniques to address.

[0009] Computational complexity: Processing and fusing large, diverse datasets can result in high computational demands, requiring efficient algorithms and substantial computational resources.

[0010] In recent years, the advancements in machine learning, particularly deep learning, have significantly propelled the development of multi-modal data fusion methods. Large language models (LLMs) have revolutionized the field of natural language processing, exhibiting exceptional capabilities in language understanding and generation. The emergence of MLLMs marks a shift in the fusion paradigm from explicit fusion strategies to implicit, learning-based fusion within the large model. This indicates that the "fusion network" is no longer an independent, manually designed component, but is deeply embedded into the Transformer architecture through shared self-attention mechanisms and adapter modules that project different modalities into a common latent space.

[0011] Despite the significant progress made by MLLMs in multi-modal understanding, they also introduce new and critical challenges.

[0012] Catastrophic forgetting (text-specific forgetting): During the continuous training process of multi-modal instructions, MLLMs catastrophically forget the text-specific instructions mastered by the initial LLM, leading to a sharp decline in performance on text-specific evaluations. This forgetting is related to the shift in cross-layer attention from text before images to text after images. This phenomenon indicates that during multi-modal learning, newly introduced modal information can disrupt or overwrite previously learned single-modal knowledge, which is not only a data problem but also a deep challenge involving architecture and training strategies. This requires mechanisms to distinguish or adaptively adjust the learning paths of different modalities during multi-modal fusion to prevent the damage of knowledge.

[0013] Insufficient generalization ability and depth of reasoning: Current MLLMs, even those tailored for specific domain applications, are far from achieving the generalization ability required for artificial general intelligence (AGI). They often perform poorly in tasks that require deep, multi-step reasoning, especially when abstract concepts are involved, and struggle to maintain logical consistency in complex tasks.

[0014] Error propagation: Errors originating from a certain modality (e.g., incorrect interpretation of a graph) can propagate throughout the reasoning process, leading to erroneous conclusions, especially in complex, multi-step problems.

[0015] Hallucinations: MLLMs can generate information that appears reasonable but is actually inconsistent with the input data or lacks factual basis, which undermines their credibility and reliability, especially in domains such as scientific reasoning that require high accuracy.

[0016] Explainability issues: The complexity and scale of MLLMs pose significant challenges to understanding how different data types are combined, how their interactions affect the output, and how the internal structure transforms inputs into outputs. This “black box” nature hinders trust-building in high-risk applications. The explainability challenges of multimodal large language models are exacerbated by the fusion process itself, as it becomes more difficult to understand how the model fuses different modalities and influences its decisions. This suggests the need for a method that is inherently explainable in the fusion mechanism, rather than just post-hoc analysis. SUMMARY

[0017] The purpose of the present application is to provide an adaptive cross-modal attention and knowledge preserving multimodal data fusion method to solve the problems raised in the background art.

[0018] To achieve the above-mentioned purpose, the present application aims to provide an adaptive cross-modal attention and knowledge preserving multimodal data fusion method, comprising the following steps:

[0019] S1, pre-processing and spatio-temporal / semantic alignment of multimodal data;

[0020] S2, extracting features through a modality-specific encoder and projecting them into a unified embedding space;

[0021] S3, using an adaptive cross-modal attention module to dynamically learn modality importance weights and fuse features within a Transformer layer;

[0022] S4, controlling multimodal information backflow through the gating mechanism of the knowledge preserving layer to prevent catastrophic forgetting;

[0023] S5, dynamically generating a cross-modal interaction graph to quantify inter-modal dependencies;

[0024] S6, training output task prediction results based on the fused representations.

[0025] As a further improvement of the technical solution, the specific operation steps of step S1 are:

[0026] Tokenizing and embedding text;

[0027] Extracting features from images / videos through a pre-trained visual encoder;

[0028] Extracting acoustic features from audio;

[0029] Aligning time series modalities using dynamic time warping (DTW) and aligning static modalities using a projection mechanism to achieve semantic alignment.

[0030] As a further improvement of the technical solution, the implementation of the adaptive cross-modal attention module in step S3 is:

[0031] Compute query Q, K, V matrix in Transformer layer

[0032] Learn modal dynamic weight according to current context

[0033] Adjust attention score by modal mask

[0034]

[0035] Wherein, is a dynamically generated mask, generated by .

[0036] As a further improvement of the technical solution, the gating mechanism of the knowledge maintaining layer in step S4 includes:

[0037] Compute gating scalar: Compute residual update: Update hidden state: .

[0038] As a further improvement of the technical solution, the construction method of the cross-modal interaction graph in step S5 is:

[0039] The nodes are token sequences of each modal

[0040] The edge weight is determined by the normalized cross-modal attention score:

[0041]

[0042]

[0043]

[0044] Wherein, the edge between the text token and the image token is derived from the attention score between them .

[0045] As a further improvement of the technical solution, the representation training in step S6 includes:

[0046] Adopt task-specific loss function

[0047] Optimize pre-training model parameters combined with LoRA fine-tuning strategy

[0048] Compared with the prior art, the beneficial effects of the present application are: ​​​​

[0049] 1、The application can capture deeper, non-linear, dynamic changes and complex semantic and feature associations in multi-modal data by using the powerful self-attention mechanism and deep network structure of the large model as the core fusion device, thus forming a more comprehensive and accurate unified multi-modal representation.

[0050] 2、The hierarchical dynamic modal weight distribution mechanism introduced in the application enables the model to intelligently evaluate and selectively use information from different modalities. For modalities with low quality, noise or partial missing, their contribution can be effectively suppressed, thereby significantly improving the robustness of the entire fusion system in complex real-world data environments.

[0051] 3、By adopting the residual adapter-based incremental learning strategy, the application can maximize the preservation of the rich knowledge and powerful reasoning ability of the large model pre-trained on a single modality while introducing multi-modal capabilities, avoiding the common catastrophic forgetting problem in traditional fine-tuning, and ensuring that the model can fully leverage its original advantages in multi-modal tasks.

[0052] 4、The application provides a new approach to understanding model decisions by dynamically distributing weights for modal gating networks and attention mechanisms. Researchers or users can analyze which modalities are considered more important and how they interact under specific tasks or inputs based on the gating values and attention weights, thereby improving the transparency and credibility of the model. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The figure is a schematic diagram of the step flow of the adaptive cross-modal attention and knowledge preservation multi-modal data fusion method of the application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0055] In a specific embodiment, as shown in Figure 1As shown, the present application provides an adaptive cross-modal attention and knowledge preserving multi-modal data fusion method, which contains several interconnected modules: a set of modality-specific encoders, an adaptive cross-modal attention (ACMA) module, a knowledge preserving layer (KPL), a cross-modal interaction graph (CMIG) generator, a large model backbone (e.g., a Transformer-based decoder-only LLM), and a task-specific prediction head. This architecture aims to facilitate rich cross-modal interactions while preserving modality-specific computations and retaining pre-trained knowledge.

[0056] Specifically, the following steps are included:

[0057] (1) Data preprocessing and modality alignment.

[0058] Diverse multi-modal data (e.g., text, image, audio, video, sensor data) is first subjected to modality-specific preprocessing.

[0059] Text: Tokenization and embedding (RoBERTa embedding) are performed.

[0060] Image: Size adjustment, normalization, and feature extraction through a pre-trained visual encoder (CLIP-based architecture) are performed.

[0061] Audio / video: Feature extraction (spectrogram, acoustic features, frame-level visual features) is performed.

[0062] Alignment and synchronization: To address the challenges of data alignment and synchronization, the present application employs a temporal alignment module. For time-series data (audio, video, sensor), dynamic time warping (DTW) is used to align the features to a common time reference. For static modalities (text, image), a projection mechanism is employed to map them to a shared latent space, ensuring semantic alignment.

[0063] (2) Feature extraction and representation learning.

[0064] a. Feature computation.

[0065] Each preprocessed modality input is fed into its respective single-modal encoder , resulting in the computation of modality-specific feature representations generated by these encoders , where is the sequence length, is the feature dimension.

[0066] b. Projection computation.

[0067] A projection layer These modality-specific features are then mapped to a unified embedding space compatible with the backbone of the large model , computed as This creates a set of modality-specific token sequences, ready for fusion.

[0068] (3) Adaptive Cross-Modal Attention module (ACMA module) and Knowledge Preservation Layer (KPL).

[0069] This mechanism integrates features from different modalities within the backbone of the large model, which is usually a Transformer. Unlike simple concatenation or fixed cross-attention, the ACMA module dynamically learns the importance and interaction between different modalities at each layer of the Transformer.

[0070] a. Attention score computation.

[0071] The input to each Transformer layer includes the concatenated modality tokens: , from which , Q, K, and V are computed. For a given attention head, the attention score is computed as: .

[0072] b. Modality importance weighting.

[0073] This mechanism learns a set of dynamic weights for each modality m at each layer , based on the current context and the relevance of the modality to the task. The attention output for each token is a weighted sum of the values, where the weights are influenced by , computed as: . Here is a dynamically generated mask that emphasizes or weakens attention to specific modality tokens based on the learned importance scores. This allows the model to adaptively focus on the most relevant modality for a given input or task. This mechanism goes beyond standard attention by introducing an explicit learning component for modality importance in the attention computation. This is technically superior to the universal attention, which may implicitly learn this importance but cannot directly control or interpret its learning process. This approach enables more robust handling of data imbalance issues and provides a way for interpretability by observing these learned weights to understand the model's decision-making process.

[0074] c. Knowledge Preservation Layer.

[0075] To alleviate catastrophic forgetting, KPL is integrated in parallel with the main attention block at each layer of the Transformer, similar to the concept of low-rank residual attention (LoRRA). However, KPL in the present invention operates as a “gate mechanism”, selectively allowing or restricting newly learned multimodal interaction to backflow into the knowledge base of the core LLM.

[0076] Let be the hidden state of the LLM backbone at the l-th layer, be the output of the ACMA module. KPL computes a gating scalar and a residual update :

[0077]

[0078]

[0079] the updated hidden state

[0080]

[0081] The gating mechanism of KPL is a more sophisticated knowledge preservation method than a simple parallel learner. It implies a learned “veto power” or “filter” that can prevent harmful updates, directly addressing the root cause of catastrophic forgetting by controlling information flow. By dynamically controlling information flow, the model can achieve a better balance between learning new multimodal patterns (plasticity) and preserving old unimodal knowledge (stability).

[0082] (4) Explainable module: Cross-modal interaction graph generator (CMIG).

[0083] To enhance explainability, CMIG is dynamically constructed at each layer of the Transformer. This graph represents the learned dependencies and influences between different modalities and their features. The nodes of the graph are tokens from each modality . Edges represent the attention weights between different modality tokens. For example, the edge between text token and image token originates from their attention score after normalization or thresholding.

[0084]

[0085]

[0086]

[0087] The figure provides a quantitative representation of how different modalities interact and influence each other's representation during fusion. It can be used for post-hoc analysis and even as a feedback mechanism during training. CMIG is a method to achieve explainability during fusion in a novel way. It translates abstract attention weights into structured, visualizable graphs, making "black-box" models more transparent by explicitly showing inter-modal relationships. By explicitly representing inter-modal attention weights as graphs, the "reasoning" process of the model when handling cross-modal cues becomes observable. This provides a powerful diagnostic tool for understanding model behavior, identifying potential biases, and validating its decisions in high-risk applications, elevating explainability from an afterthought to an intrinsic feature of the fusion architecture.

[0088] (5) Model training and optimization.

[0089] The CMAP model is trained end-to-end on multi-modal datasets (e.g., image-text pairs, video-audio-text data).

[0090] Loss function: Use task-specific loss for the primary task (e.g., cross-entropy loss for classification, mean squared error for regression).

[0091] Knowledge preservation auxiliary loss: An auxiliary loss term can be introduced to explicitly encourage KPL to maintain single-modal performance. For example,

[0092] Text-specific loss can be computed using a small portion of pure text data and then combined with the main multi-modal loss.

[0093] Optimization: Standard deep learning optimizers (AdamW) are used. LoRA or QLoRA fine-tuning strategies can be applied to efficiently update model parameters while leveraging pre-trained LLM weights.

[0094] Data augmentation: Modality-specific and cross-modal data augmentation techniques are used to enhance robustness and generalization capabilities.

[0095] During inference, new multi-modal inputs are preprocessed and encoded. The encoded features are then fed into the ACMAP system. The ACMA module dynamically weights the modalities, and the KPL ensures knowledge consistency. The large model backbone processes the fused representation to generate the required output (e.g., predictions, generation, answers). CMIG can be selectively generated during inference to provide real-time explainability.

[0096] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. Adaptive cross-modal attention and knowledge-preserving multimodal data fusion method, characterized by: The following steps are involved: S1. Preprocessing and spatiotemporal / semantic alignment of multimodal data; S2, extract features through modality-specific encoder and project them into a unified embedding space; S3, using the adaptive cross-modal attention module to dynamically learn modality importance weights and fuse features within the Transformer layer; S4. Control the reflow of multimodal information through the gating mechanism of the knowledge retention layer to prevent catastrophic forgetting; S5, dynamically generate cross-modal interaction graphs to quantify inter-modal dependencies; S6. Output task prediction results based on the fused representation training.

2. The adaptive cross-modal attention and knowledge-preserving multimodal data fusion method according to claim 1 is characterized in that: The specific operation steps of step S1 are: Perform word segmentation and embedding on text; Extract features from images / videos using pre-trained visual encoders; Extract acoustic features from audio; Dynamic time warping (DTW) is used to align time series modalities, and a projection mechanism is used to achieve semantic alignment for static modalities.

3. The adaptive cross-modal attention and knowledge-preserving multimodal data fusion method according to claim 1 is characterized in that: The adaptive cross-modal attention module of step S3 is implemented as follows: Compute the query Q, K, V matrices at the Transformer layer; Learn modality dynamic weights based on the current context ; Adjust the attention score by the modality mask: ； in, is a dynamically generated mask consisting of generate.

4. The adaptive cross-modal attention and knowledge-preserving multimodal data fusion method according to claim 1 is characterized in that: The gating mechanism of the knowledge retention layer in step S4 includes: Compute the gated scalar: ; Compute the residual update: ; Update hidden state: .

5. The adaptive cross-modal attention and knowledge-preserving multimodal data fusion method according to claim 1 is characterized in that: The cross-modal interaction graph in step S5 is constructed as follows: Nodes are token sequences for each modality ; The edge weight is determined by the normalized cross-modal attention score: ； ； ； Among them, the text token and image tokens The edge between Derived from the attention score between them .

6. The adaptive cross-modal attention and knowledge-preserving multimodal data fusion method according to claim 1, characterized in that: The representation training in step S6 includes: Use task-specific loss functions; Add auxiliary loss terms to supervise unimodal knowledge preservation using plain text data; Combined with LoRA fine-tuning strategy to optimize pre-trained model parameters.

Images