Improved multi-mode learning method and device, equipment and medium

Abstract

The invention discloses an improved multi-mode learning method, device, equipment and medium, and the method comprises the steps: collecting a plurality of modes, and constructing a training set; extracting modal features of corresponding modals through various modal specific encoders, and fusing the modal features of different modals by adopting a bidirectional cross attention mechanism to obtain bidirectional fusion features; training a shared classifier based on the bidirectional fusion features, and introducing a regularization item into training loss; parameters of the shared classifier are frozen, and a specific Low-Rank adaptation (LoRA) branch is configured for each mode of the shared classifier; in the training process, each modal specific encoder and the LoRA branch are updated alternately; after each iteration update, identifying an extremely unbalanced sample based on a modal contribution score, and performing secondary optimization on a corresponding modal specific encoder and a LoRA branch by using weak modal features of the extremely unbalanced sample; and finally, the reasoning part obtains a prediction result through decision-level weighted fusion.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an improved multimodal learning method, apparatus, device, and medium. Background Technology

[0002] Multimodal learning, by fusing information from different perceptual modalities (such as audio, images, text, and video), has demonstrated powerful modeling capabilities in various machine learning tasks, achieving significant progress in areas such as speech recognition, emotion recognition, video understanding, and human-computer interaction. However, a growing body of research has found that multimodal models often face performance bottlenecks in practical applications, and in some scenarios, their performance is even inferior to models using only a single modality. The root cause of this phenomenon is the modality imbalance problem: due to significant differences between different modalities in terms of information quality, data distribution, learning architecture, and task relevance, certain strong modalities dominate the optimization process during training, while the contributions of weak modalities are gradually marginalized, thus affecting overall performance.

[0003] To address the modality imbalance problem, several representative works have emerged in recent years: OGM (On-the-fly Gradient Modulation), AGM (Adaptive Gradient Modulation), and PMR (Prototypical Modality Rebalance) dynamically modulate modal gradients through differential indices; while UMT (Uni-Modal Teacher), GBlending (Gradient-Blending), and MMPareto focus on the inconsistency between single-modal and multimodal objectives, enhancing weak modality representations by introducing single-modal supervision terms into the loss function. However, these methods all focus on optimizing the model update mechanism, neglecting the differences between modalities. To address this, MBSD, Calibrating Multimodal Learning (CML), and LFM use KL divergence between modality prediction distributions, modality confidence, and contrastive learning, respectively, to constrain the feature representations of each modality in the model, thereby achieving a balance during the update process. Furthermore, SMSL (Sample-level Modality Valuation) and Wei et al., in response to the inherent imbalance in data distribution, attempted to quantify modal contributions at the sample level and screen samples of weak modalities, thereby improving the model's ability to discriminate weak modalities.

[0004] However, the framework used in the above methods still relies on the joint optimization paradigm, and its synchronous gradient update mechanism struggles to avoid inter-modal competition, which severely restricts model convergence stability and modal synergistic efficiency. To address this limitation, some researchers have proposed strategies for alternating modal optimization. For example, the Multimodal Learning with Alternating Unimodal Adaptation (MLA) method alternates between different modalities to update the encoder, effectively reducing encoder-level interference by providing each modality with independent optimization opportunities. However, this method does not address classifier-level bias. Specifically, due to differences in data complexity and representation learning difficulty between modalities, dominant modalities typically exhibit faster convergence speeds and learning efficiency, while weaker modalities require longer training times to achieve effective representations. In the early stages of alternating training, the rapidly converging dominant modality dominates the classifier parameter update process, causing the classifier to quickly adapt and become biased towards that modality's feature space. As training progresses, even though weaker modalities are still actively learning and generating large gradients, the classifier has developed a structural preference for the dominant modality, making it difficult to effectively integrate the representation information of the weaker modality, thus leading to the persistent modality imbalance problem.

[0005] Although existing theoretical studies have pointed out that fixing the structural parameters of the classifier in classification tasks helps to stabilize the discrimination space and prevent boundary drift, the relevant conclusions are mostly based on the background of single-modal learning and have not been effectively utilized in the framework of multimodal alternating optimization. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides an improved multimodal learning method, a corresponding device, an electronic device, and a computer-readable storage medium.

[0007] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0008] An improved multimodal learning method includes the following steps:

[0009] Step 1: Collect multiple modalities through the data acquisition module and construct a training set based on the collected modalities; extract modal features of corresponding modalities through modal-specific encoders, and fuse the modal features of different modalities using a bidirectional cross-attention mechanism to obtain bidirectional fused features; train a shared classifier based on the bidirectional fused features, and introduce a regularization term into the training loss, wherein the regularization term is constructed based on the difference in contribution of various modal features to the bidirectional fused features to constrain the imbalance of modal contributions;

[0010] Step 2: Freeze the parameters of the shared classifier and equip each modality with its own Low-Rank Adaptation (LoRA) branch. Then, during training, alternately update the modality-specific encoder and LoRA branch. After each iteration update, identify extremely imbalanced samples based on the modality contribution score, and use the weak modality features of the extremely imbalanced samples to perform secondary optimization of the loss of the corresponding modality-specific encoder and LoRA branch.

[0011] Step 3: The final prediction result is obtained by performing decision-level fusion on the output results of different modalities.

[0012] Preferably, in step 1, the multimodal information includes first modal information and second modal information.

[0013] Preferably, in step 1, the bidirectional cross-attention mechanism includes the following steps:

[0014] Using the first modality feature as the Query and the second modality feature as the Key and Value, after multi-head attention calculation, residual connection, layer normalization and feedforward network processing, the first direction fusion feature is obtained by pooling;

[0015] Using the second modality feature as the Query and the first modality feature as the Key and Value, after multi-head attention calculation, residual connection, layer normalization and feedforward network processing, the second direction fusion feature is obtained by pooling.

[0016] The first-direction fusion feature is superimposed with the second-direction fusion feature to obtain the bidirectional fusion feature.

[0017] Preferably, in step 1, the construction of the regularization term includes the following steps:

[0018] Calculate the mutual information between the first modality feature and the bidirectional fused feature, and between the second modality feature and the bidirectional fused feature;

[0019] The obtained mutual information is normalized to obtain the modal contribution vector;

[0020] The regularization term is obtained by taking the mean of the absolute values ​​of the differences between the first and second mode contribution scores in the modal contribution vector of the sample.

[0021] Preferably, in step 2, during the alternating update, the loss is calculated only based on the current modality information and the output of the shared classifier, the classifier parameters are frozen throughout, and the modality-specific encoder and LoRA branch for the corresponding modality are optimized; while the modality-specific encoder and LoRA branch for other modalities remain fixed.

[0022] Preferably, in step 2, the extremely imbalanced sample is a sample whose modality contribution score is lower than a preset imbalance threshold, and the secondary optimization loss is calculated based on the weak modality data of the sample and the output of the shared classifier.

[0023] An improved multimodal learning device includes:

[0024] Data acquisition module: used to acquire various modalities and send the acquired modalities to the data processing module;

[0025] Data processing module: Used to process various modalities and generate the final prediction results.

[0026] An electronic device includes a central processing unit and a memory, the central processing unit being configured to invoke and run a computer program stored in the memory to perform the steps of the improved multimodal learning method.

[0027] A computer-readable storage medium stores, in the form of computer-readable instructions, a computer program implemented according to the improved multimodal learning method, which, when invoked by a computer, executes the steps included in the corresponding method.

[0028] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0029] 1. The improved multimodal learning method of the present invention performs secondary updates through an alternating learning mechanism, a pre-training and fixed strategy of a shared classifier, and a modality-specific encoder guided by sample-level contribution. This enables the model to process information from different modalities without bias, allowing each modality to play its role to the fullest extent and thus improving model performance.

[0030] 2. The improved multimodal learning method of this invention, based on the modality alternation update mechanism to eliminate the gradient conflict problem between modes, adopts a classifier pre-training and fixing strategy, introduces a modality-specific LoRA branch, retains the modality-specific features while learning cross-modal information, and stabilizes the discrimination boundary, thereby improving the classification accuracy of the model. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the improved multimodal learning method of the present invention.

[0032] Figure 2 A pre-training framework for shared classifiers.

[0033] Figure 3 This is a flowchart illustrating the modality alternation update and sample-level retraining mechanism. Detailed Implementation

[0034] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0035] Current multimodal deep learning methods mainly suffer from modality imbalance, specifically, modality competition during joint optimization, semantic drift and inaccurate discrimination boundaries in shared classifiers during alternating training, and information imbalance inherent in the samples themselves. To address these issues, this invention employs an alternating learning mechanism, pre-training of the shared classifier, and secondary updates of modality-specific encoders guided by fixed strategies and sample-level contribution. This approach enables the model to process information from different modalities without bias, allowing each modality to maximize its contribution and thus improving model performance.

[0036] See Figures 1-3 The improved multimodal learning method of the present invention includes the following steps:

[0037] Step 1: Collect multiple modalities through the data acquisition module and construct a training set based on the collected modalities; extract modal features of corresponding modalities through modal-specific encoders, and fuse the modal features of different modalities using a bidirectional cross-attention mechanism to obtain bidirectional fused features; train a shared classifier based on the bidirectional fused features, and introduce a regularization term into the training loss, wherein the regularization term is constructed based on the difference in contribution of various modal features to the bidirectional fused features to constrain the imbalance of modal contributions;

[0038] Step 2: Freeze the parameters of the shared classifier and introduce a LoRA branch specific to each modality into the shared classifier. During formal training, alternately update the modality-specific encoder and LoRA branch. After each iteration update, identify extremely imbalanced samples based on the modality contribution score, and use the weak modality features of the extremely imbalanced samples to perform secondary optimization on the corresponding modality-specific encoder and LoRA branch.

[0039] Step 3: The final prediction result is obtained by performing decision-level fusion of the results from different modalities.

[0040] The following uses audio and visual modalities as examples to illustrate the principle of the improved multimodal learning method of this invention:

[0041] This invention proposes a novel optimization framework to address the modality imbalance problem in multimodal learning. Its core idea lies in decomposing the standard connector optimization process into three stages, wherein…

[0042] In the first stage, a cross-attention mechanism is introduced to fuse audio modal features and visual modal features. A shared classifier is trained by combining audio modal features and visual modal features to establish collaborative representations between modalities. During the training of the shared classifier, a regularization term for modal contribution balancing is introduced to alleviate the modal dominance phenomenon in the early training, while making full use of the complementary information of modalities.

[0043] In the second stage, the shared classifier is frozen, and a LoRA branch is introduced based on the shared classifier. A modality alternation update strategy is used to optimize the corresponding modality-specific encoder and LoRA branch. Based on the optimization in the first stage, extremely imbalanced samples are identified according to the contribution of different modalities in the samples. For these extremely imbalanced samples, only their weak modality information is used to perform a secondary update on the corresponding modality-specific encoder and LoRA branch, thereby effectively enhancing the discrimination ability of weak modalities and alleviating the modality imbalance phenomenon.

[0044] In the third stage (i.e., the reasoning stage), this invention employs decision-level fusion, which involves adding and fusing the outputs of each modality to obtain the final prediction.

[0045] Figure 2 The overall framework of the pre-training phase is presented; since it is desired that the classifier inherently incorporates cross-modal semantics, rather than solely relying on preset parameters, a data-driven pre-training strategy is adopted, specifically:

[0046] Training dataset D = {(x i y i )} i＝1，2，...，N Each sample The samples contain audio modalities and visual modality And it comes with its real label y i Modality-specific encoders Enc1 and Enc2 are used independently from audio modal data. and visual modality Extract features from them, namely frequency features. and video features

[0047]

[0048] To comprehensively model the interactions between modalities, a bi-directional cross-attention mechanism is employed for feature fusion. This mechanism consists of cross-attention calculations in two directions—from visual to audio and from audio to visual—defined as follows:

[0049] (1) Fusion from visual to audio

[0050] In this direction, audio features Considered a query, while video features These serve as the Key and Value. Multi-head attention (MHA) output. for:

[0051]

[0052] To stabilize and enhance the feature fusion representation, residual connections were subsequently added, followed by layer normalization (LayerNorm).

[0053]

[0054] Subsequently, a feedforward network (FFN) is applied to introduce nonlinearity and model higher-level interactions, followed by another layer normalization (LayerNorm):

[0055]

[0056] Finally, average pooling (AvgPool) is used to aggregate the sequences into a compact fused representation, oriented from visual to audio:

[0057]

[0058] (2) Fusion from audio to visual

[0059] In this direction, visual features Treated as a query, while audio features These serve as the Key and Value. The calculation process is as follows:

[0060]

[0061] Subsequently, by adding the fused features from the audio-to-visual and visual-to-audio directions, a bidirectional fused feature representation f is obtained. i ;

[0062]

[0063] Next, the final prediction is generated using a shared classifier cls(·):

[0064]

[0065] To mitigate model bias towards specific modalities, a modality contribution-oriented regularization mechanism is introduced. This mechanism quantifies the relative contribution of each modality to the bidirectional fused features based on the estimated mutual information (MI), thereby estimating the features of each modality. With fusion feature f iThe statistical dependence between them, the calculation process is as follows: Figure 2 As shown, the corresponding formula is as follows:

[0066]

[0067] In the formula: Let represent the expectation on the dataset, and N represent the total number of samples; a larger mutual information value indicates that modality j has a more significant impact on the fused representation.

[0068] Based on mutual information, the modal contribution vector in sample i is defined as:

[0069]

[0070] The normalization operation described above is used to ensure that the values ​​of the contribution vectors are appropriately scaled for subsequent comparisons and regularization.

[0071] The cross-entropy loss is combined with a regularization term that penalizes large differences in modal contributions; this regularization term promotes balanced feature extraction by encouraging the model to maintain fairness across modalities; the mathematical formula is expressed as follows:

[0072]

[0073] The total loss is obtained by adding the classification loss to the regularization term:

[0074] L total =L cls +0.001*L regularization

[0075] like Figure 3 As shown, after the pre-training phase, only the parameters of the shared classifier are retained and subsequently frozen. The shared classifier adds a LoRA branch specific to each modality. Training then enters the modality alternation phase. In each iteration, only the modality-specific encoder and LoRA branch corresponding to the current modality are updated, while the modality-specific encoders and LoRA branches for other modalities remain fixed. This enhances the discriminative power of the current modality features while mitigating gradient interference and feature suppression across modalities; essentially, the model employs a single-modality alternation training strategy, and the parameters of the shared classifier remain unchanged throughout the training process. In each iteration, only the modality-specific encoder and LoRA branch corresponding to the current modality are updated. After updating the modality-specific encoder and LoRA branch using all samples in the current batch, mutual information is estimated to quantify the modality contribution of each sample, identifying samples with extremely imbalanced modality contributions, which are then used for secondary optimization of the modality-specific encoder and LoRA branch.

[0076] Specifically, in each iteration, given a batch of input samples The model uses its corresponding modality-specific encoder to sequentially process the data of each individual modality in the entire batch, thereby obtaining their respective prediction outputs. and related cross-entropy loss

[0077]

[0078] In the formula: the function cls(·) represents a fixed shared classifier across all modalities; B is the batch size; y i This represents the true label of the i-th sample; This refers to the LoRA (Local Algorithm for Mode j) encoder. j (·) refers to the unique branch of each modality on the shared classifier, where j∈{1,2} corresponds to the two audio modalities and the visual modality.

[0079] Subsequently, the cross-entropy loss for a specific mode is used via backpropagation. To optimize Enc j and LoRA j The parameter of (·).

[0080] Furthermore, to alleviate the shortcomings of weak mode optimization, this invention also proposes a sample-level retraining mechanism; that is, after completing the initial full update of all modality-specific encoders and LoRA branches, a sample-level retraining mechanism is proposed for each sample. Calculate modal contribution score As mentioned above With C i The definition of , is used to reflect the degree of contribution of mode j in predicting the target label.

[0081] Based on modal contribution score It can be used to identify a subset S of an extremely imbalanced sample. j Defined as satisfying The samples, where α is a predefined imbalance threshold; these samples are passed through Enc j The fixed shared classifier cls(·) is reprocessed to obtain new predictions; the cross-entropy loss is re-evaluated as:

[0082]

[0083] In the formula: L represents the retrained subset S j The number of samples in y, and y j This is the true label of the i-th sample. This auxiliary loss is then used to adjust the encoder Enc. j A second-stage gradient update is performed to improve its representation capability on modally imbalanced instances.

[0084] During the inference phase, given feature vectors extracted from two different modalities... and Shared classifiers independently produce corresponding prediction results. and The prediction results are simply added together to obtain the final prediction output:

[0085]

[0086] This fusion mechanism improves the robustness and flexibility of the model by processing each modality independently and integrating them uniformly at the prediction layer, making it particularly suitable for modal heterogeneous or non-equilibrium fields.

[0087] Example 2

[0088] The improved multimodal learning device of the present invention includes:

[0089] Data acquisition module: used to acquire various modalities and send the acquired modalities to the data processing module;

[0090] Data processing module: Used to process various modalities and generate the final prediction results.

[0091] Example 3

[0092] The electronic device of the present invention includes a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the improved multimodal learning method.

[0093] Example 4

[0094] The present invention provides a computer-readable storage medium that stores, in the form of computer-readable instructions, a computer program implemented according to the improved multimodal learning method, which, when invoked by a computer, executes the steps included in the corresponding method.

[0095] The above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above content. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An improved multimodal learning method, characterized in that, Includes the following steps: Step 1: Collect multiple modalities through the data acquisition module and construct a training set based on the collected modalities; extract modal features of corresponding modalities through modal-specific encoders, and fuse the modal features of different modalities using a bidirectional cross-attention mechanism to obtain bidirectional fused features; train a shared classifier based on the bidirectional fused features, and introduce a regularization term into the training loss, wherein the regularization term is constructed based on the difference in contribution of various modal features to the bidirectional fused features to constrain the imbalance of modal contributions; Step 2: Freeze the parameters of the shared classifier and equip each modality with its own LoRA branch, and alternately update the modality-specific encoder and LoRA branch; after each iteration update, identify extremely imbalanced samples based on the modality contribution score, and use the weak modality features of the extremely imbalanced samples to perform secondary optimization on the corresponding modality-specific encoder and LoRA branch. Step 3: Decision-level fusion is performed on the results obtained from different modal data through the corresponding modality-specific encoder, LoRA branch and shared classifier to obtain the final prediction result.

2. The improved multimodal learning method according to claim 1, characterized in that, In step 1, the multimodal information includes first modal information and second modal information.

3. The improved multimodal learning method according to claim 1, characterized in that, In step 1, the bidirectional cross-attention mechanism includes the following steps: Using the first modality feature as the Query and the second modality feature as the Key and Value, after multi-head attention calculation, residual connection, layer normalization and feedforward network processing, the first direction fusion feature is obtained by pooling; Using the second modality feature as the Query and the first modality feature as the Key and Value, after multi-head attention calculation, residual connection, layer normalization and feedforward network processing, the second direction fusion feature is obtained by pooling. The first-direction fusion feature is superimposed with the second-direction fusion feature to obtain the bidirectional fusion feature.

4. The improved multimodal learning method according to claim 3, characterized in that, In step 1, the construction of the regularization term includes the following steps: Calculate the mutual information between the first modality feature and the bidirectional fused feature, and between the second modality feature and the bidirectional fused feature; The obtained mutual information is normalized to obtain the modal contribution vector; The regularization term is obtained by taking the mean of the absolute values ​​of the differences between the first and second mode contribution scores in the modal contribution vector of the sample.

5. The improved multimodal learning method according to claim 4, characterized in that, In step 2, during the alternating update, the parameters of the shared classifier are frozen throughout, and the loss is calculated only based on the current modality information and the output of the shared classifier to optimize the modality-specific encoder and LoRA branch for the corresponding modality; while the modality-specific encoder and LoRA branch for other modalities remain fixed.

6. The improved multimodal learning method according to claim 5, characterized in that, In step 2, the extremely imbalanced sample is the sample whose modality contribution score is lower than the preset imbalance threshold, and the secondary optimization loss is calculated based on the weak modality data of the sample and the output of the shared classifier.

7. The improved multimodal learning method according to claim 1, characterized in that, In step 3, the decision-level fusion is: adding and fusing the output results of each modality to obtain the final prediction result.

8. An improved multimodal learning device, characterized in that, include: Data acquisition module: used to acquire various modalities and send the acquired modalities to the data processing module; Data processing module: Used to process various modalities and generate the final prediction results.

9. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the improved multimodal learning method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implementing the improved multimodal learning method according to any one of claims 1-7, which, when called by a computer, executes the steps included in the corresponding method.

Images