A two-layer missingness-resistant method for incomplete multi-modal data

Abstract

The application discloses a double-layer missing resistance method for incomplete multi-modal data, comprising the following steps: constructing a fine-grained intra-modal missing resistance module, enhancing the robustness of the model to local missing data in the intra-modal by dynamically balancing the relationship between missing area suppression and global capability maintenance; designing a coarse-grained inter-modal missing resistance module, predicting and filling missing information of the whole modal by adopting cross-modal shared features, and dynamically calculating a modal fusion weight by combining a task-driven capability perception score calculator to realize adaptive feature fusion based on an attention mechanism. The application strengthens the supervision signal of high-quality data through a dynamic weight distribution mechanism, and still maintains the robustness of the model in an extreme scene with a missing rate of up to 90%, effectively solving the semantic understanding deviation problem caused by mixed-granularity missing in multi-modal data.

Description

Technical Field

[0001] This invention relates to the field of multimodal learning, and more particularly to a two-layer missing data resistance method for incomplete multimodal data. Background Technology

[0002] With the development of artificial intelligence, multimodal learning, due to its ability to integrate multiple information such as language, vision, and audio, has shown broad application prospects in tasks such as sentiment analysis and human-computer interaction. However, the incompleteness of multimodal data, including the loss of information within a single modality and the lack of cross-modal data, seriously affects the robustness and generalization ability of the model.

[0003] Existing research mainly employs generative methods, multimodal joint learning, and knowledge distillation to address the problem of missing data. For example, generative methods use generative adversarial networks or denoising autoencoders to fill in missing data, multimodal joint learning improves overall performance through information sharing between different modalities, and knowledge distillation transmits cross-modal knowledge to student networks through teacher networks, enabling student models to effectively learn shared features between modalities even when some modalities are missing, thereby improving the final feature representation ability. However, these methods still have certain limitations: (1) it is difficult to dynamically adjust during model training to adapt to interference from low-quality filler data; (2) the research focus is often on cross-modal missing data, while insufficient attention is paid to the more common intramodal missing data. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a two-layer missing data resistance method for incomplete multimodal data, aiming to simultaneously address the challenges posed by intramodal missing data and cross-modal missing data. This method consists of an intramodal missing data resistance module (IADR) and an intermodal missing data resistance module (IEDR). By introducing an adaptive attention mechanism, shared feature prediction, and capability-aware scoring mechanism, it improves robustness and generalization ability to incomplete multimodal data.

[0005] The objective of this invention is achieved through the following technical solution: a two-layer missing data resistance method for incomplete multimodal data, comprising:

[0006] Fine-grained intramodal missing data resistance module: Employs a dynamic attention compensation mechanism to handle locally missing data within a single modality;

[0007] Cross-modal shared feature prediction module: Constructs a multimodal shared semantic space based on feature consistency constraints, and uses the collaborative information of available modalities to realize the joint representation reconstruction of missing modalities;

[0008] Capability-aware scoring module: Generates contribution indicators for each modality through a task-driven pseudo-quality assessment network, and achieves non-linear mapping of modality weights by combining dynamic scaling factors;

[0009] Coarse-grained intermodal fusion module: Based on the weights of each modality, and combined with normalized weight preprocessing technology, adaptive fusion of multimodal features is achieved.

[0010] Furthermore, the fine-grained intramodal missing-resistance module specifically includes:

[0011] Window-shifting and block-based preprocessing is performed on the digitized two-dimensional matrix data of audio and video modalities to enhance the globality of the data; an attention compensation mechanism is constructed, a compensation component is added to the standard attention calculation, and a global information compensation term controlled by learnable parameters is introduced; by dynamically adjusting the coefficient balance, the data attention level and the strength of global information retention can be obtained.

[0012]

[0013] Where Q, K, V come from input x i The linear transformation adds a compensation component U to the original attention calculation and introduces a learnable parameter ∈ to dynamically adjust the degree of global capability supplementation; d k M represents the dimension of the key vector; M is the mask information.

[0014] Furthermore, the cross-modal shared feature prediction module specifically includes:

[0015] Establish a feature distribution consistency constraint mechanism based on KL divergence to force different modalities to share feature space alignment;

[0016] The residual connection is designed to connect the original features with the extracted shared features to enhance semantic expressive power. The original features are extracted from the data of each modality through the corresponding encoder, and the shared features are extracted from the original features through the shared feature extractor.

[0017] Modal co-compensation can be achieved by implementing a weighted fusion of available modal features to generate alternative representations.

[0018] Furthermore, the capability-aware scoring module specifically includes:

[0019] A single-modal isolation test network was constructed, and the pseudo-quality score of each mode was calculated using the zero-filling method;

[0020] Based on the aforementioned pseudo-quality scores, the scoring ability of the scorer to assess modal quality is perceived through mean square error optimization.

[0021] A dynamic scaling factor is introduced to adjust the influence of weights on the final cross-modal fusion, thereby enhancing the supervision signal of high-quality modalities.

[0022] Furthermore, the coarse-grained intermodal fusion module specifically includes:

[0023] Construct a segmented attention adjustment matrix and differentiate positive and negative attention scores based on weight values;

[0024] A multi-layer cross-attention mechanism is adopted to achieve deep interaction and semantic alignment of cross-modal features, ensuring that higher-quality modalities are given higher weights in the fusion stage.

[0025] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described two-layer missing data resistance method for incomplete multimodal data.

[0026] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described two-layer missing data resistance method for incomplete multimodal data.

[0027] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described two-layer missing data resistance method for incomplete multimodal data.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] 1. IADR primarily addresses the issue of fine-grained missing information within a single modality. Due to potential information loss, noise interference, or uneven sampling during data acquisition, incomplete segments may exist within a single modality. Traditional methods often employ hard masking or interpolation imputation strategies, but these methods can weaken the model's global learning ability. To solve this problem, IADR uses an adaptive attention mechanism (Intra-Attn) to dynamically adjust the attention distribution during model training, balancing the utilization of effective data with the impact of missing information. This mechanism not only reduces the interference of missing parts on model predictions but also ensures improved feature extraction capabilities without losing global information.

[0030] 2. To address the issue of missing cross-modal data, IEDR (comprising a cross-modal shared feature prediction module, a capability-aware scoring module, and a coarse-grained inter-modal fusion module) is optimized through Shared Feature Prediction (SFP) and a Capability-Aware Scoring Mechanism (CAS). In multimodal learning, different modal data may originate from heterogeneous sensors or different acquisition time points, resulting in some modal data being unavailable or having poor synchronization. Traditional methods typically rely on simple feature concatenation or missing modality completion, but these methods often fail to fully utilize information from available modalities. Therefore, SFP employs a cross-modal shared feature extraction strategy, using information from known modalities to predict shared features of missing modalities, thus maintaining reasonable modal representation even in cases of missing data. On the other hand, CAS calculates the contribution of different modalities to downstream tasks and dynamically weights them according to task requirements, ensuring the effectiveness of information fusion and avoiding the negative impact of low-quality features on the final prediction. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of the present invention;

[0033] Figure 2 This is a flowchart of the present invention for scenarios with missing modalities;

[0034] Figure 3 This is a flowchart illustrating the fusion module of the present invention;

[0035] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0037] This invention provides a two-layer missing data resistance method for incomplete multimodal data, such as... Figure 1 As shown, it includes a fine-grained intramodal missing feature resistance module, a cross-modal shared feature prediction module, a capability-aware scorer module, and a coarse-grained intermodal fusion module.

[0038] The fine-grained intramodal missing data resistance module includes an Intra-Attn module, which addresses the impact of missing data in a single modality by introducing a dynamic Intra-Attn mechanism (dynamic attention compensation mechanism). Specifically, when processing data for each modality, the Intra-Attn module is first applied. This module dynamically adjusts the attention mechanism based on the location of the missing data, focusing on available data while avoiding excessive suppression of missing data regions. By incorporating a compensation component into the computation and introducing a globally information compensation term controlled by learnable parameters, the model maintains its global capabilities while selectively updating features in locally missing regions. The specific formula is shown below:

[0039]

[0040] Where Q, K, and V come from the input x i The linear transformation adds a compensation component to the original attention calculation. A learnable parameter ∈ is introduced to dynamically adjust the degree of global capability supplementation; M is mask information, with missing parts represented by -∞ and non-missing parts represented by 0.

[0041] The cross-modal shared feature prediction module uses Shared Feature Prediction (SFP) technology to reconstruct missing modalities by leveraging collaborative information among available modalities. In practical applications, shared features among modalities are inferred from existing modal data to fill in the gaps in missing modalities. Specifically, KL divergence is used to ensure the consistency of shared features, guaranteeing that shared features extracted from existing modal data can effectively fill in missing modalities, thereby mitigating information loss caused by missing modalities.

[0042] The capability-aware scoring system is implemented by generating modality contribution metrics through a task-driven pseudo-quality evaluation network. The data for each modality is evaluated for quality according to task requirements, and the evaluation results are used as dynamic weights for further weight allocation. In practice, by calculating the pseudo-quality of each modality (e.g., evaluating the task performance of a specific modality using the cross-entropy loss function CE) and combining it with a dynamic scaling factor, the modality weights can influence the final modality fusion to varying degrees based on different task requirements, thereby enhancing the accuracy of multimodal feature fusion.

[0043]

[0044] Among them W i Let λ be the scoring weight of the capability-aware scorer for modality i, and let λ be the dynamic scaling factor as the normalization weight and its influence on the final modality fusion.

[0045] The coarse-grained intermodal fusion module includes the Inter-Attn module, which combines the features of each modality based on the weights of each modality.

[0046] Specifically, during modality fusion, the contribution of each modality to the current task is first dynamically evaluated by calculating capability-aware weights. This process is performed by the Capability-Aware Scorer (CAS), see [link to documentation]. Figure 3 It trains the network to evaluate modality quality by using pseudo-quality for each modality. Specifically, pseudo-quality is achieved by constructing a single-modality isolation test network, inputting the features of a particular modality into the decoder, retaining the features of that modality, and setting the features of other modalities to zero, thereby evaluating the impact of that modality on the final task. By calculating the cross-entropy loss (CE) and combining it with the task-specific objective, CAS generates a weight for each modality, reflecting its importance in the current task.

[0047] Next, during the normalization weight preprocessing, the weights of each modality are first standardized to ensure that the contributions of all modalities can be compared on the same scale. Then, adaptive fusion is performed using the Inter-Attn module. The Inter-Attn module dynamically adjusts the weights of each modality output by the capability-aware scorer, thereby adjusting the weights in the attention score calculation part of the attention-based fusion stage. High-quality modal features receive more attention and weight, while lower-quality modalities are appropriately suppressed. This modality-quality-based weighted fusion method ensures that even when a modality is missing, the model can still effectively utilize features from other modalities for inference, maximizing information utilization efficiency.

[0048] In this way, the coarse-grained intermodal fusion module achieves adaptive fusion of intermodal features, dynamically adjusting the contribution of each modality to the final decision based on its quality and task requirements. This optimizes information fusion during multimodal learning, ensuring high-quality feature representations are generated even when some modalities are missing. This method improves the model's robustness and prediction accuracy in environments with missing data. The method of this invention can be applied to various tasks, such as multimodal sentiment classification tasks. See [link to related documentation]. Figure 2 The method acquires video data, audio data, and text data, processes them using the method of this invention to obtain fused features, and inputs them into a decoder for emotion classification.

[0049] The modules interact with each other. A fine-grained intramodal missing data resistance module ensures that data within each modality retains maximum effective information, avoiding the negative impact of missing regions on overall performance. A cross-modal shared feature prediction module utilizes data from available modalities to supplement missing modalities through feature sharing, ensuring the model can still make effective predictions even with cross-modal missing data. A capability-aware scorer dynamically adjusts the weights of each modality in the fusion process based on its task performance, ensuring that key modal features are fully utilized. Finally, a coarse-grained intermodal fusion module precisely adjusts the contribution of each modality, enabling adaptive fusion of multimodal features, thereby improving the robustness and prediction accuracy of the entire model in missing data environments. These modules work together to form a powerful method that effectively addresses missing data issues in multimodal data and enhances the model's generalization ability.

[0050] Experimental results demonstrate that this proposed method outperforms existing methods on multiple datasets. Testing on the widely used benchmark datasets CMU-MOSI and CMU-MOSEI for multimodal sentiment analysis shows that it achieves good performance under various missing data scenarios. Furthermore, experiments on unimodal tasks (such as text classification, image classification, and audio classification) further validate the effectiveness of IADR in handling fine-grained missing data, while IEDR demonstrates excellent adaptability when facing large-scale modality missing data. Overall, this method improves the robustness and generalization ability of the model in multimodal learning tasks, providing a new solution for handling incomplete multimodal data in the future.

[0051] Figure 4 This is a schematic diagram of an electronic device structure provided in an embodiment of the present invention. Please refer to... Figure 4 The electronic device provided in this embodiment includes a memory and a processor. The memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, they implement a two-layer missing data resistance method for incomplete multimodal data according to the present invention.

[0052] It should be noted that, in addition to Figure 4 In addition to the memory and processor shown, electronic devices may include other hardware depending on their actual functions, which will not be elaborated further.

[0053] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described two-layer missing data resistance method for incomplete multimodal data.

[0054] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described two-layer missing data resistance method for incomplete multimodal data.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

Claims

1. A feature fusion method for incomplete multimodal data, characterized in that, include: Fine-grained intramodal missing data resistance module: Employs a dynamic attention compensation mechanism to handle locally missing data within a single modality; multimodal data includes video, audio, and text data, while single-modal data is one of these three types. Cross-modal shared feature prediction module: Constructs a multimodal shared semantic space based on feature consistency constraints, and uses the collaborative information of available modalities to realize the joint representation reconstruction of missing modalities; The fine-grained intramodal missing data resistance module includes the Intra-Attn module, which introduces a dynamic Intra-Attn mechanism to handle the impact of missing data in a single modality. When processing each modality's data, the Intra-Attn module is applied first. This module dynamically adjusts the attention mechanism based on the location of the missing data, focusing on available data while avoiding excessive suppression of missing data regions. By adding a compensation component to the calculation and introducing a globally information compensation term controlled by learnable parameters, the model maintains its global capabilities while selectively updating features in locally missing regions. The specific formula is shown below: Where Q, K, and V come from the input. The linear transformation adds a compensation component U to the original attention calculation and introduces learnable parameters. To dynamically adjust the level of global capability enhancement; M represents mask information; Capability-aware scoring module: Generates contribution indicators for each modality through a task-driven pseudo-quality assessment network, and achieves non-linear mapping of modality weights by combining dynamic scaling factors; Coarse-grained intermodal fusion module: Based on the weights of each modality, and combined with normalized weight preprocessing technology, adaptive fusion of multimodal features is achieved.

2. The method according to claim 1, characterized in that, The cross-modal shared feature prediction module specifically includes: Establish a feature distribution consistency constraint mechanism based on KL divergence to force different modalities to share feature space alignment; Residual connections are designed to link the original features with the extracted shared features, thereby enhancing semantic expressive power; the original features are extracted from each modality data through the corresponding encoder. Modal co-compensation can be achieved by implementing a weighted fusion of available modal features to generate alternative representations.

3. The method according to claim 1, characterized in that, The capability-aware scoring module specifically includes: A single-modal isolation test network was constructed, and the pseudo-quality score of each mode was calculated using the zero-filling method; Based on the aforementioned pseudo-quality scores, the scoring ability of the scorer to assess modal quality is perceived through mean square error optimization. A dynamic scaling factor is introduced to adjust the influence of weights on the final cross-modal fusion, thereby enhancing the supervision signal of high-quality modalities.

4. The method according to claim 1, characterized in that, The coarse-grained intermodal fusion module specifically includes: Construct a segmented attention adjustment matrix and differentiate positive and negative attention scores based on weight values; A multi-layer cross-attention mechanism is adopted to achieve deep interaction and semantic alignment of cross-modal features, ensuring that higher-quality modalities are given higher weights in the fusion stage.

5. An electronic device, comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the feature fusion method for incomplete multimodal data as described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a feature fusion method for incomplete multimodal data as described in any one of claims 1-4.

7. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements a feature fusion method for incomplete multimodal data as described in any one of claims 1-4.

Images