Context modal completion multi-modal learning method based on semantic matching

Abstract

The invention discloses a context modal completion multi-modal learning method based on semantic matching, and relates to the technical field of multi-modal learning, and the method comprises the following steps: for a multi-modal sample with modal damage, matching a semantic-related complete sample from a database based on reserved intact modal data; according to modal data of the semantic related complete sample, combining semantic information of the remaining intact modals to generate complementation representation of the damaged modals; inputting the complemented modal data and the remaining intact modal data into a multi-modal fusion model to generate a prediction result; and the performance stability under the modal damage condition is improved by optimizing model parameters through the combination of the task-related loss and the complementation loss. According to the method, through an innovative framework of semantic matching and context modal completion, the modal damage problem in multi-modal learning is effectively solved, and the performance and stability of the multi-modal fusion model under the modal damage condition are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal learning, and more particularly to a contextual modality completion multimodal learning method based on semantic matching. Background Art

[0002] Multimodal learning is a key area of ​​artificial intelligence, capable of integrating diverse data types such as images, text, and audio. It is widely used in fields such as medical diagnosis, intelligent driving, and content moderation. However, in real-world applications, modal corruption is a common problem. Device failure, privacy restrictions, or data corruption can render some modalities unusable, severely impacting model performance.

[0003] Traditional multimodal systems are usually designed under the assumption that all modalities can be obtained normally. When faced with modal corruption, the system performance often degrades significantly. To address this problem, researchers have proposed two main solutions: (1) modality robust representation learning and (2) cross-modal content completion. The former attempts to construct a unified feature representation that is insensitive to modal corruption, but often uses a simple zero-filling method to deal with the corrupted modality, failing to effectively utilize the remaining modal information; the latter attempts to directly complete the corrupted modal content from the available modalities, but due to the large essential differences between the modalities, the completion quality is limited. (3) Using multimodal transformers to combat corrupted modal learning, through the powerful adaptability of transformers and by introducing some hints to improve their performance. For example, AMTR uses multi-task training and strategy fusion to deal with modal corruption; MAPs and MSPs introduce hint adjustment techniques to improve the performance of pre-trained transformers under modal corruption conditions. However, the above three methods usually simply replace the damaged mode with zero value, or reconstruct with limited quality, ignoring the important role of the context provided by similar samples in the database in high-quality and effective modality completion, and are difficult to adapt to complex and changing practical application scenarios.

[0004] In summary, existing technologies face two key challenges when dealing with modal corruption: first, they fail to effectively utilize similar samples in the database for modal completion; second, they lack consideration of sample-specific semantics, resulting in unstable performance in complex application environments.

[0005] Therefore, it is necessary to propose a contextual modality completion multimodal learning method based on semantic matching to solve the above problems. Summary of the Invention

[0006] The purpose of the present invention is to address two key challenges faced by existing technologies when dealing with modal corruption: one is the failure to effectively utilize similar samples in the database for modal completion; the other is the lack of consideration of sample-specific semantics, which leads to unstable performance in complex application environments.

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] A multimodal learning method for contextual modality completion based on semantic matching, comprising the following steps:

[0009] S1. For multimodal samples with modality damage, match semantically relevant complete samples from the database based on the retained intact modality data;

[0010] S2. generating a complete representation of the damaged modality based on the modal data of the semantically relevant complete sample and the semantic information of the remaining intact modalities;

[0011] S3. Input the completed modal data and the remaining intact modal data into the multimodal fusion model to generate prediction results;

[0012] S4. Optimize model parameters through a combination of task-related loss and completion loss to improve performance stability under modal damage conditions.

[0013] Furthermore, in S1, the step of matching semantically relevant complete samples includes:

[0014] Select the retained mode as the query mode according to the mode damage situation;

[0015] Map the query modality to the semantic space and calculate the semantic similarity with the complete samples in the database;

[0016] The most relevant Top-N complete samples are selected based on semantic similarity.

[0017] Furthermore, in S2, the step of generating a damaged modality completion representation includes:

[0018] The semantic contribution weight of each relevant complete sample is evaluated through the semantic association network;

[0019] The modal features of the relevant complete samples are aggregated based on the semantic contribution weights to generate a completed damaged modal representation.

[0020] Furthermore, in S3, the parameters of the multimodal fusion model remain frozen during the training process, and only the parameters related to modality completion are updated.

[0021] Furthermore, the task-related loss is the cross entropy loss of the classification task, and the completion loss is the mean square error loss.

[0022] Furthermore, the modal data includes at least two of text, image, and audio.

[0023] Furthermore, the semantic matching and completion process is applicable to scenarios of single-modality damage, multi-modality partial damage, or complete damage.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. This invention addresses the problem of incomplete information caused by modal corruption by integrating a semantic matching mechanism into the modal corruption learning task and leveraging the concept of content recovery from information theory to reconstruct the damaged modal information from matched semantically relevant samples. A dynamic completion mechanism based on contextual weight allocation uses a semantic association network to adaptively weight matching samples, ensuring that the completed content is semantically consistent with the target sample. This mechanism effectively completes the damaged modal, providing more complete model input and more robust decision-making basis.

[0026] 2. This invention effectively solves the problem of modal corruption in multimodal learning through an innovative framework of semantic matching and contextual modal completion, significantly improves the performance and stability of multimodal fusion models in the case of modal corruption, and provides an effective solution for multimodal systems in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the framework of the contextual modality completion multimodal learning method based on semantic matching of the present invention.

[0028] Figure 2 This is an example display of the semantic matching part of the present invention.

[0029] Figure 3 Schematic diagram of the performance comparison of each model on three data sets at a modality missing rate of 70%. DETAILED DESCRIPTION

[0030] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] See also Figures 1 to 3A multimodal learning method for contextual modality completion based on semantic matching. This method integrates the semantic matching mechanism into the modality damage learning task and, with the help of the content recovery concept in information theory, reconstructs the damaged modal information from the matched semantically related samples, solving the problem of incomplete information caused by modality damage. A dynamic completion mechanism based on contextual weight allocation implements adaptive weight calculation of matching samples through a semantic association network to ensure that the completed content is semantically consistent with the target sample. Through this mechanism, the damaged modality is effectively completed, the model input is more complete, and the decision-making basis is more sufficient.

[0032] The contextual modal completion system based on semantic matching provided by the present invention includes:

[0033] The semantically guided matching module is responsible for flexibly adjusting the matching strategy based on modal corruption, and performing efficient intra-modal semantic matching using the retained modal. This module can find the most semantically relevant complete samples for different modal corruption scenarios, providing high-quality reference material for subsequent modality completion.

[0034] The context-weighted completion module completes the damaged modality based on the matched relevant samples and the remaining intact modality. This module uses a semantic association network to evaluate the semantic relevance of each matching sample to the target sample, achieving intra-modal context completion and effectively addressing the modality discrepancy problem in cross-modal completion.

[0035] In the contextual modal completion system based on semantic matching, the semantic-guided matching module includes:

[0036] The encoder submodule maps data from different modalities into a unified semantic space to facilitate the calculation of semantic similarity between modalities;

[0037] The adaptive matching strategy submodule dynamically adjusts the matching strategy according to the modal damage status of the current sample, selects the appropriate modality as the query, and matches the most relevant sample from the database.

[0038] In the contextual modality completion system based on semantic matching, the contextual weight completion module includes:

[0039] The semantic association network submodule evaluates the semantic association between the matching sample and the target sample, generates a dynamic weight distribution based on the remaining intact modalities, and determines the contribution of each matching sample in the completion process;

[0040] The intra-modal context completion submodule aggregates the features of matching samples based on the weights calculated by the semantic association network to generate a representation of the damaged modality.

[0041] Based on the above technical ideas, the present invention specifically includes the following steps:

[0042] S1. For a given multimodal sample (possibly with modal corruption), a semantically guided matcher is used to perform a matching process. Based on the modal corruption of the sample, the matching strategy is flexibly adjusted, using the intact modality as the query to match the most relevant modality-complete sample from the database.

[0043] Specifically, for samples with intact modalities, joint matching is performed using both text and image modalities. For samples with partially damaged modalities, matching is performed using the intact modality. The matched samples are then used to complete the damaged modality.

[0044] S2. Based on the relevant samples matched in S1 and the remaining intact modalities, the context-weighted completer performs completion of the corrupted modality. First, the matched data of the same type as the corrupted modality is considered a "semantic reference." The contribution of each reference sample is then evaluated using a semantic association network. This network generates a dynamic weight distribution based on the semantic interactions between the remaining intact modality and these reference samples, determining the relative importance of each reference sample in the completion process. Based on these weights, the features of the reference samples are aggregated to generate a representation of the corrupted modality, achieving intra-modal context completion.

[0045] S3: Input the damaged mode completed in S2 together with the original intact mode into the multimodal fusion model to perform the final prediction task.

[0046] During the model prediction process, the parameters of the multimodal fusion model remain frozen, and only the parameters of the context weight completer are updated. This design significantly reduces the amount of training parameters and improves training efficiency.

[0047] S4. Use a combination of task loss function (such as cross entropy loss for classification tasks) and completion loss (mean squared error loss) to optimize the model, and update the parameters through backpropagation so that the model can better cope with the challenges of modal damage.

[0048] The specific implementation details of the above contextual modal completion learning method based on semantic matching are as follows:

[0049] In this paper, the task is a multimodal classification involving two modalities: a text modality T and an image modality V. In practical applications, these modalities may be corrupted, meaning that the data for one modality may be partially or completely damaged, resulting in degraded multimodal system performance. By restoring the damaged modal information through semantic matching and context completion techniques, the model can maintain high classification performance even under modal corruption conditions.

[0050] In S1, to achieve efficient semantic matching and modality representation, a pre-trained semantic encoding model is used as an encoder to map the data into a unified semantic space. For samples with modality damage, semantic matching is performed using the remaining intact modalities. For example, for a sample X with image modality damage, its retained text modality T is used for encoding to obtain the text query vector where h t is the dimension of the text modality. The query vector is then used to search for semantically related samples in the database:

[0051]

[0052] in Indicates the matched Top-N complete modal instances, Is a database storing (image, text) pairs. Similarly, for samples with corrupted text modality, the retained image modality V is used for encoding and matching:

[0053]

[0054] The semantically guided matcher automatically selects the appropriate matching strategy according to the modal damage condition, ensuring that the most relevant complete sample can be found as a repair reference under any modal damage condition.

[0055] In step S2, in order to address the modality difference challenge faced by traditional cross-modal completion methods, a context weight completer is proposed, which effectively solves this problem through intra-modal context completion.

[0056] For the dynamic integration of intra-modal reference samples, a semantic relevance network (SRN) is designed to evaluate the contribution of each reference sample. Specifically, for a target sample X with image modality damage, its remaining text modality T is used to match the image modality data (i.e., intra-modal reference sample) Perform semantic interaction to generate a weight that controls the contribution of each reference sample in completing the damaged image modality.

[0057] The semantic association network first applies the embedding layer from the multimodal fusion model to convert the text and matched image data into embedded representations to obtain text embeddings and the matched image embedding where p and q are the number of text tokens and image patches, respectively, and z is the embedding dimension. Subsequently, an aggregation operation is applied to these embeddings to reduce the sequence length, resulting in and

[0058] The semantic association network uses the remaining modality to interact semantically with the reference samples in each modality and generates a weight that determines the contribution of each reference sample in completing the damaged modality:

[0059]

[0060] Where Q1 and Q2 are learnable transformation matrices and z is the embedding dimension. Finally, a softmax operation is applied to generate normalized weight scores:

[0061]

[0062] where α j is the weight score assigned to the jth reference sample, reflecting its relative importance in completing the damaged image modality for sample X. With the weight scores α = [α1, α2, ..., α N ], the contextual weighted completer first applies a set of transformation networks (TNs) to the embedding of each reference sample, and then generates the corrupted image modality by aggregating the reference sample-specific outputs by weight scores:

[0063]

[0064] in represents the completed embedding of the damaged image modality of the target sample X. Completing the text modality is similar. This intra-modality contextual completion strategy provides semantically consistent restoration of damaged content, providing a more complete multimodal input for the pre-trained multimodal fusion model.

[0065] For steps S3 and S4, the prediction process of the model is as follows: for a damaged sample X in a text modality, first match it with its most relevant Top-N modality complete samples through a semantically guided matcher. Recover its text embedding through contextual weight completer And embed the completed text and the original intact image into F v Input into the multimodal fusion model together.

[0066] S5. The repaired modality and the remaining intact modality are input into the fusion and classification network to generate the final prediction result. Specifically, for samples with damaged text modality, the repaired text modality is represented as and the retained image modality representation F v Jointly fed into the fusion classification network:

[0067]

[0068] Where Fusion(·,·) is a multimodal fusion network used to integrate information from different modalities; Ω represents the classifier, W and b are the weight matrix and bias vector of the classification network, respectively. is the predicted category probability distribution. For image modality corruption, use During the training process, the parameters of the multimodal fusion model remain frozen, and only the parameters of the context weight completer are updated. The framework uses a combination of task-related losses (such as cross entropy loss for classification tasks) and completion losses (mean squared error loss) for optimization:

[0069]

[0070] Where λ is a hyperparameter that balances the two losses, is the task-related loss, is the completion loss, which is used to ensure that the generated modality representation is close to the true representation.

[0071] 1. Experimental Setup

[0072] 1.1 Dataset

[0073] To validate the effectiveness of our method on a variety of datasets, we conducted a comprehensive evaluation using three widely used multimodal classification benchmark datasets: MM-IMDb, HateMemes, and Food101. MM-IMDb is a multi-label movie classification dataset consisting of movie posters (images) and plot summaries (text); HateMemes is a hate speech detection dataset consisting of images and text; and Food101 is a food classification dataset consisting of food images and descriptions. We simulated varying degrees of modality loss: text loss, image loss, and partial loss of both modalities to evaluate our method's performance under various modality loss scenarios.

[0074] 1.2 Evaluation indicators

[0075] In order to comprehensively evaluate the performance of the present invention, different evaluation indicators were used according to the characteristics of the dataset. For the MM-IMDb dataset, F1-Macro (F1-M) and F1-Samples (F1-S) were used as the main evaluation indicators. These two indicators can reflect the performance of multi-label classification tasks; for the HateMemes dataset, the area under the ROC curve (AUROC) was used as the evaluation indicator; for the Food101 dataset, the classification accuracy (ACC) was used as the evaluation indicator. The larger the four evaluation indicators, the better. These evaluation indicators can comprehensively reflect the performance of the model under different task types and data characteristics.

[0076] 1.3 Baseline Model

[0077] To evaluate the performance of our proposed method (CALM), we compare it with several state-of-the-art baseline models, which can be divided into three groups:

[0078] Modality robust learning methods: including IF-MMIN, ShaSpec, DrFuse, CorrKD, and MoMKE. These methods mainly address the modality missing problem by learning shared representations between modalities.

[0079] Cross-modal completion methods: including SMIL, TFR-Net, and AcMAE. These methods attempt to solve the problem of missing modalities by generating missing modalities from available modalities.

[0080] Multimodal Transformer-based methods include MAPs and MSPs. These methods mainly improve the performance of multimodal Transformers in the absence of modalities through techniques such as hint tuning.

[0081] 2. Comparison of main results

[0082] The performance of CALM is compared with other baseline models under the same experimental settings. Figure 3 The performance comparison of different models under 70% modality missing rate is shown, covering various modality missing scenarios. In all experimental settings, CALM shows excellent performance and significantly outperforms the existing state-of-the-art methods in most indicators. Figure 3 As shown in Figure 2, the performance comparison of each model on three datasets under a modality missing rate of 70%. The black bold results represent suboptimal results, and the red bold results represent optimal results.

[0083] Experimental results demonstrate that the proposed CALM significantly outperforms existing techniques across all datasets in various modal corruption scenarios. For example, on the MM-IMDb dataset, compared to the suboptimal method, the proposed method achieves improvements of 8.79% and 6.78% in the F1-Macro and F1-Samples metrics, respectively, in text corruption scenarios. On the HateMemes dataset, the AUROC metric improves by 11.73% in text corruption scenarios. On the Food101 dataset, classification accuracy improves by 5.77% in text corruption scenarios.

[0084] Ablation experiments confirmed the significant contributions of the semantically guided matching module and the contextual weight completion module to system performance; removing either component resulted in a significant performance degradation. Parameter sensitivity analysis showed that the system performs best when the number of matching samples is between 3 and 7. This finding provides important parameter configuration guidance for practical applications.

[0085] In summary, this paper effectively solves the problem of modal corruption in multimodal learning through an innovative framework of semantic matching and contextual modal completion, significantly improves the performance and stability of the multimodal fusion model under modal corruption, and provides an effective solution for multimodal systems in practical applications.

[0086] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. The scope of patent protection of the present invention shall be based on the claims. Any equivalent structural changes made using the contents of the description of the present invention shall also be included in the scope of protection of the present invention.

Claims

1. A contextual modality completion multimodal learning method based on semantic matching, characterized by: The following steps are involved: S1. For multimodal samples with modality damage, match semantically relevant complete samples from the database based on the retained intact modality data; S2. generating a complete representation of the damaged modality based on the modal data of the semantically relevant complete sample and the semantic information of the remaining intact modalities; S3. Input the completed modal data and the remaining intact modal data into the multimodal fusion model to generate prediction results; S4. Optimize model parameters through a combination of task-related loss and completion loss to improve performance stability under modal damage conditions.

2. The contextual modality completion multimodal learning method based on semantic matching according to claim 1 is characterized by: In S1, the step of matching semantically related complete samples includes: Select the retained mode as the query mode according to the mode damage situation; Map the query modality to the semantic space and calculate the semantic similarity with the complete samples in the database; The most relevant Top-N complete samples are selected based on semantic similarity.

3. The contextual modality completion multimodal learning method based on semantic matching according to claim 1, characterized in that: In S2, the step of generating a damaged modality completion representation includes: The semantic contribution weight of each relevant complete sample is evaluated through the semantic association network; The modal features of the relevant complete samples are aggregated based on the semantic contribution weights to generate a completed damaged modal representation.

4. The contextual modality completion multimodal learning method based on semantic matching according to claim 1, characterized in that: In S3, the parameters of the multimodal fusion model remain frozen during the training process, and only the parameters related to modality completion are updated.

5. The contextual modality completion multimodal learning method based on semantic matching according to claim 1, characterized in that: The task-related loss is the cross entropy loss of the classification task, and the completion loss is the mean square error loss.

6. The contextual modality completion multimodal learning method based on semantic matching according to claim 1, characterized in that: The modality data includes at least two of text, image, and audio.

7. The contextual modality completion multimodal learning method based on semantic matching according to claim 1, characterized in that: The semantic matching and completion process is applicable to scenarios where a single modality is damaged, or multiple modalities are partially damaged or completely damaged.

Images