Optimization method based on multi-modal social optimal model

Abstract

The invention discloses an optimization method based on a multi-modal social optimal model, belongs to the field of multi-modal learning, and is particularly suitable for efficient fusion of video data. The invention provides an optimization method based on a multi-mode social optimal model, which adjusts the gradient size and direction of a single mode in multi-mode learning, defines multi-mode utility and multi-mode jeay, and is used for respectively measuring the contribution of each mode to the performance of the model and reducing the deviation of the model to a specific mode. Through the method, efficient fusion among multi-modal data is realized. The model provided by the invention can effectively relieve the modal imbalance problem in multi-modal learning, has a wide application prospect, and can be popularized to various multi-modal fusion scenes.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal fusion in multimodal learning, and specifically relates to an optimization method based on a multimodal social optimal model, thereby improving the prediction performance of the multimodal model. Background Art

[0002] Multimodal learning has made significant progress in integrating multiple information sources (such as video, text, and sensors), providing a more comprehensive perspective for research in diverse scenarios. However, multimodal learning often faces the problem of modality imbalance: during learning, the model tends to rely on the modality with richer semantic information, causing the update direction of the model parameters to gradually shift towards that modality. This phenomenon can result in the modality with weaker semantic information not being fully learned and utilized, thereby limiting the model's ability to comprehensively express multimodal data and ultimately affecting its predictive performance.

[0003] In order to solve the problem of modal imbalance in multimodal learning, researchers have proposed to assist the learning of modalities with weaker semantic information by adjusting the training process of unimodal encoders, for example, by adjusting the size and direction of unimodal parameter gradient updates. However, this type of method mainly focuses on the optimization process of the unimodal mode, and fails to fully consider the overall mitigation effect of the multimodal optimization objective on the modal imbalance phenomenon, and therefore has certain limitations. Recent studies have shown that the introduction of multimodal optimization objectives can alleviate the modal imbalance problem to a certain extent, but simply combining unimodal optimization objectives with multimodal optimization objectives may limit the overall performance and expression ability of the model due to the conflict between the two. In order to coordinate this conflict, some studies have used the Pareto optimization strategy to find a trade-off solution in multi-objective optimization. By adjusting the gradient size and direction of each optimization objective, collaborative learning between different modalities is achieved, thereby alleviating the modal imbalance problem. Based on this, the present invention regards the constraint relationship between multimodal optimization objectives as a multi-objective optimization problem. By finding an optimal solution in the mutually influencing constraints, the comprehensive utilization efficiency of each modality in multimodal learning is further improved, the modal imbalance phenomenon is alleviated, and the model performance is improved.

[0004] At the same time, previous studies on the phenomenon of unbalanced multimodality often overemphasized the competition between training objectives and ignored their potential collaborative relationship, making it difficult to ensure the overall optimal performance of the system. In multimodal learning, the modalities collaborate with each other to jointly improve the performance of the system, and each modality makes a unique contribution to the overall performance. This collaboration is similar to the concept of social welfare, which reflects the overall effectiveness of the system, comes from the combined contributions of all modalities, and makes full use of their semantic richness. When the modalities collaborate well, they can compensate for each other's semantic deficiencies, thereby improving the overall performance. Therefore, the present invention proposes to directly optimize social welfare to enhance the performance of the multimodal system and promote the balance and collaborative contribution of all modalities. At the same time, it is crucial to design a reasonable optimization method. The method should effectively reduce the conflict between different optimization objectives, avoid the model from being biased towards a specific modality during the collaboration process, and ensure that while maximizing the overall benefits, each modality is properly optimized and adjusted.

[0005] In summary, building a unified multimodal learning framework that effectively integrates multimodal information to improve model performance and robustness while addressing modal imbalance is of great research significance. This will not only advance the field of multimodal learning but also provide more efficient and intelligent solutions in application scenarios such as intelligent human-computer interaction, intelligent customer service, online education, and healthcare, ultimately optimizing the user experience and improving the overall effectiveness of the system. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing multimodal learning methods in the problem of modal imbalance, construct a unified framework, capture the collaborative relationship between multimodal data, dynamically adjust the optimization strategy, take into account the characteristics of semantically rich modalities and weaker modalities, thereby achieving comprehensive analysis of multimodal data and improving the robustness and performance of the model in complex scenarios.

[0007] The technical solutions of the present invention are as follows: An optimization method based on a multimodal social optimal model comprises the following steps: Step 1: Select a training dataset; Step 2: Feed the original video into the model and process it: extract relevant frames from the video modality and process the audio modality into corresponding input files; Step 2: Use the pre-trained ResNet-18 model to extract frame-level features of the input video data; Step 3: Use the pre-trained ResNet-18 model to extract audio features of the input audio data; Step 4: Input the video and audio features extracted in steps 2 and 3 into the corresponding unimodal classifiers to obtain the probability scores of the video modality and audio modality; Step 5: Perform multimodal fusion on the video and audio features extracted in steps 2 and 3; Step 6: Input the fused multimodal features obtained in step 5 into the multimodal classifier to obtain the probability score of the multimodal features; Step 7: Use the cross entropy loss function to calculate the loss function of the unimodal and multimodal probability scores obtained in steps 4 and 6 respectively; Step 8: Add the three loss functions obtained in step 7 and perform backpropagation to update the parameters in the model using the stochastic gradient descent method; Step 9: Extract the gradients of the video, audio, and multimodal features calculated during backpropagation. Step 10: Calculate the multimodal utility between video and multimodal features, and between audio and multimodal features, respectively, to measure the contribution of each single modality to the final multimodal performance; Step 11: Calculate the multimodal envy between video and multimodal features, and between audio and multimodal features, respectively, to assess the potential unfairness caused by uneven resource allocation in a single modality. Step 12: Combining the multimodal utility calculated in Step 10 and the multimodal envy calculated in Step 11, construct a social welfare optimization objective function and use a linear solver to obtain the two optimal weight combinations. The optimization goal is to maximize multimodal social welfare while minimizing multimodal envy, thereby effectively alleviating modal imbalance.

[0008] Step 13: The updated video feature gradient and multimodal feature gradient obtained in step 9, and the updated audio feature gradient and multimodal feature gradient are weighted and summed according to the weight combination obtained in step 12, respectively, to obtain updated unimodal feature gradients, and the parameters in the modal feature extractor are updated accordingly; Step 14: Perform temperature scaling on the single-modal gradient obtained in step 13 to adjust the balance of modal learning; Step 15: The model updates parameters in a loop to alleviate the unbalanced modal problem during iteration; Step 16: When the model performs downstream tasks, the video and audio data are input into the model, and the final prediction result is obtained through the multimodal classifier. The model outputs the specific category to which the input video belongs; As a preferred technical solution, steps 2 and 3 include in more detail: when extracting audio and visual features, you can use a pre-trained 2D or 3D deep convolutional neural network model alone, or you can use multiple pre-trained models and concatenate the extracted features as the final features. and ; As a preferred technical solution, step 4 includes in more detail: obtaining the probability scores of the audio and video modalities through a unimodal classifier composed of fully connected layers: and ; As a preferred technical solution, step 5 includes in more detail: the unimodal features obtained in steps 2 and 3 are and Through the splicing multimodal fusion method, multimodal features are obtained and multimodal feature probability scores ; As a preferred technical solution, step 10 includes in more detail: multimodal utility is used to quantify the contribution of different modalities to the overall model prediction, which can be expressed by the following formula: in represents the utility of each mode, ( ) represents the probability value obtained by each modal feature after passing through the linear layer.

[0009] As a preferred technical solution, step 11 includes in more detail: multimodal jealousy is used to prevent the multimodal model from preferring a specific modality for learning, which can be expressed by the following formula: in and They represent the utility of single-modal features and the utility of fused modal features, respectively.

[0010] As a preferred technical solution, step 12 includes in more detail: a defined multimodal social welfare optimization objective function, the goal of which is to find a set of appropriate weight distributions for multimodal features and video features, and for multimodal features and audio features, respectively, so that under this distribution, the multimodal and unimodal features work together to maximize the final performance of the multimodal model, while reducing the multimodal model's preference for a specific modality. The defined multimodal social welfare objective function can be expressed as follows: in represents the optimal weight distribution, represents the weight of each mode, Represents the adjustable balance coefficient.

[0011] As a preferred technical solution, step 13 includes in more detail: using the optimal weight combination obtained in step 12 to adjust the updated gradients after single-modality and multi-modal back-propagation, thereby modifying the parameters in the model to avoid imbalance between the modes, which can be expressed by the following formula: in , and represents the gradient of the model parameter update, , and represents the model parameters, and They represent the weight distribution obtained by social optimization.

[0012] Therefore, by adjusting the gradient size and direction of parameter updates for different modes, the model can alleviate the phenomenon of modal imbalance; As a preferred technical solution, step 14 includes in more detail: adjusting the temperature value so that the gradient adjusted in step 13 is within a suitable range, which is conducive to updating the model parameters and can be expressed by the following formula: in Represents the temperature coefficient.

[0013] As a preferred technical solution, step 16 includes in more detail: using the model to perform multimodal downstream tasks, specifically: inputting the features of different single modalities into the model, and after multimodal fusion in step 5, inputting the obtained multimodal features into the multimodal classifier, and after activation by the softmax function, obtaining the final probability values of each category, and selecting the label with the highest probability as the final predicted category.

[0014] The beneficial effects of the present invention are as follows: This paper deeply studies the imbalance problem of heterogeneous modalities in multimodal learning, regards the gradient modulation process in unbalanced multimodal learning as an application of multi-objective optimization, and realizes the trade-off between different optimization objectives through game theory; This paper innovatively introduces social welfare optimization as a global constraint, emphasizing the collaboration between different optimization objectives, aiming to improve the overall performance of multimodal models in imbalanced phenomena and effectively resolve gradient conflicts between modalities. Starting from a single-modality perspective, this paper proposes to introduce multimodal envy as a single-modality constraint in the model learning process to further reduce gradient conflicts during the optimization process and weaken the model's preference for a specific modality, so as to meet the needs of multi-objective optimization. The present invention has been experimentally verified to outperform previous methods for solving imbalanced multimodality on two widely used multimodal classification task datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments, wherein: Figure 1 Flowchart for the implementation of the present invention; Figure 2 Schematic diagram of the framework proposed by the present invention for solving the problem of modal imbalance in multimodal learning; Figure 3 This is a comparison chart of the effects of the present invention on the multimodal classification task on the CREMA-D dataset. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. That is, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments.

[0017] ResNet-18 model: ResNet-18 is a deep convolutional neural network with 18 layers. It uses skip connections to address the degradation problem in deep network training, allowing information to be passed directly within the network, thereby improving training efficiency and model performance. ResNet-18 performs well in image classification tasks and is often used for feature extraction.

[0018] Unimodal classifiers: Unimodal classifiers are classification models designed for a single data modality, such as video or audio. They process their input data separately, extracting features and performing classification. In your pipeline, video and audio data are processed separately by separate unimodal classifiers to obtain their own probability scores.

[0019] Multimodal classifiers: Multimodal classifiers are models that process multiple data modalities, such as video and audio. They fuse features from different modalities, comprehensively considering information from each modality to improve classification performance. In your workflow, after the video and audio features are fused, they are input into the multimodal classifier for the final classification prediction.

[0020] Cross-entropy loss function: The cross-entropy loss function is used to measure the difference between the probability distribution predicted by the model and the true label distribution. In classification tasks, the cross-entropy loss function is often used to train the model to minimize the difference between the predicted results and the actual labels.

[0021] Multimodal utility: Multimodal utility measures the contribution of each modality to the final multimodal classification performance. By calculating the gradient between video and audio features and multimodal features, the importance of each modality in multimodal classification can be evaluated.

[0022] Multimodal Envy: Multimodal Envy assesses the potential unfairness of a single modality due to uneven resource allocation. It measures the extent to which insufficient resource allocation to a modality can affect overall performance during multimodal fusion.

[0023] Social Welfare Optimization Objective Function: This objective function combines multimodal utility and multimodal envy to maximize the overall performance of multimodal classification while minimizing unfairness caused by uneven resource allocation. Solving this objective function with a linear solver yields the optimal combination of modal weights.

[0024] like Figure 1 As shown, the implementation of the present invention can be divided into two stages: learning the model and using the model to solve multimodal downstream tasks.

[0025] Example 1 Step 1: Select a training dataset; Step 2: Feed the original video into the model and process it: extract relevant frames from the video modality and process the audio modality into corresponding input files; Step 2: Use the pre-trained ResNet-18 model to extract frame-level features of the input video data; Step 3: Use the pre-trained ResNet-18 model to extract audio features of the input audio data; Step 4: Input the video and audio features extracted in steps 2 and 3 into the corresponding unimodal classifiers to obtain the probability scores of the video modality and audio modality; Step 5: Perform multimodal fusion on the video and audio features extracted in steps 2 and 3; Step 6: Input the fused multimodal features obtained in step 5 into the multimodal classifier to obtain the probability score of the multimodal features; Step 7: Use the cross entropy loss function to calculate the loss function of the unimodal and multimodal probability scores obtained in steps 4 and 6 respectively; Step 8: Add the three loss functions obtained in step 7 and perform backpropagation to update the parameters in the model using the stochastic gradient descent method; Step 9: Extract the gradients of the video, audio, and multimodal features calculated during backpropagation. Step 10: Calculate the multimodal utility between video and multimodal features, and between audio and multimodal features, respectively, to measure the contribution of each single modality to the final multimodal performance; Step 11: Calculate the multimodal envy between video and multimodal features, and between audio and multimodal features, respectively, to assess the potential unfairness caused by uneven resource allocation in a single modality. Step 12: Combining the multimodal utility calculated in Step 10 and the multimodal envy calculated in Step 11, construct a social welfare optimization objective function and use a linear solver to obtain the two optimal weight combinations. The optimization goal is to maximize multimodal social welfare while minimizing multimodal envy, thereby effectively alleviating modal imbalance.

[0026] Step 13: The updated video feature gradient and multimodal feature gradient obtained in step 9, and the updated audio feature gradient and multimodal feature gradient are weighted and summed according to the weight combination obtained in step 12, respectively, to obtain updated unimodal feature gradients, and the parameters in the modal feature extractor are updated accordingly; Step 14: Perform temperature scaling on the single-modal gradient obtained in step 13 to adjust the balance of modal learning; Step 15: The model updates parameters in a loop to alleviate the unbalanced modal problem during iteration; Step 16: When the model performs downstream tasks, the video and audio data are input into the model, and the final prediction result is obtained through the multimodal classifier. The model outputs the specific category to which the input video belongs; like Figure 2 As shown, the core concept of this invention is to introduce social optimization constraints. By calculating the utility functions of unimodal and multimodal features, multimodal social welfare and multimodal envy are defined, and this is used to construct a socialized optimal objective function. By solving the objective function, the optimal weight combination is obtained, which is used to weightedly sum the update gradients of the unimodal and multimodal features, ultimately generating adjusted gradients to update the model parameters. The defined multimodal utility quantifies the contribution of unimodal and multimodal features to the final model performance; multimodal social welfare is used to measure the overall model performance; and multimodal envy reflects the model's preference for a specific modality during the learning process. The multimodal social welfare objective function, composed of multimodal utility and multimodal envy, reflects the model's balance between pursuing synergy between multimodal features to improve overall performance while reducing dependence on specific modalities during learning. Finally, by optimizing the optimal weight combination obtained by optimizing the social welfare objective function, the model's update gradients are dynamically adjusted, effectively alleviating the modal imbalance problem in multimodal learning and improving the model's robustness and overall performance.

[0027] like Figure 3As shown, the present invention proposes a method for solving the problem of modal imbalance in multimodal learning. After given multimodal data, it can not only improve the accuracy of the model using single modality prediction, but also improve the accuracy of the model using multimodal features to predict, thereby alleviating the problem of modal imbalance.

[0028] Example 2 As a preferred technical solution, steps 2 and 3 include in more detail: when extracting audio and visual features, you can use a pre-trained 2D or 3D deep convolutional neural network model alone, or you can use multiple pre-trained models and concatenate the extracted features as the final features. and ; As a preferred technical solution, step 4 includes in more detail: obtaining the probability scores of the audio and video modalities through a unimodal classifier composed of fully connected layers: and ; As a preferred technical solution, step 5 includes in more detail: the unimodal features obtained in steps 2 and 3 are and Through the splicing multimodal fusion method, multimodal features are obtained and multimodal feature probability scores ; As a preferred technical solution, step 10 includes in more detail: multimodal utility is used to quantify the contribution of different modalities to the overall model prediction, which can be expressed by the following formula: in represents the utility of each mode, ( ) represents the probability value obtained by each modal feature after passing through the linear layer.

[0029] As a preferred technical solution, step 11 includes in more detail: multimodal jealousy is used to prevent the multimodal model from preferring a specific modality for learning, which can be expressed by the following formula: in and They represent the utility of single-modal features and the utility of fused modal features, respectively.

[0030] As a preferred technical solution, step 12 includes in more detail: a defined multimodal social welfare optimization objective function, the goal of which is to find a set of appropriate weight distributions for multimodal features and video features, and for multimodal features and audio features, respectively, so that under this distribution, the multimodal and unimodal features work together to maximize the final performance of the multimodal model, while reducing the multimodal model's preference for a specific modality. The defined multimodal social welfare objective function can be expressed as follows: in represents the optimal weight distribution, represents the weight of each mode, Represents the adjustable balance coefficient.

[0031] As a preferred technical solution, step 13 includes in more detail: using the optimal weight combination obtained in step 12 to adjust the updated gradients after single-modality and multi-modal back-propagation, thereby modifying the parameters in the model to avoid imbalance between the modes, which can be expressed by the following formula: in , and represents the gradient of the model parameter update, , and represents the model parameters, and They represent the weight distribution obtained by social optimization.

[0032] Therefore, by adjusting the gradient size and direction of parameter updates for different modes, the model can alleviate the phenomenon of modal imbalance; As a preferred technical solution, step 14 includes in more detail: adjusting the temperature value so that the gradient adjusted in step 13 is within a suitable range, which is conducive to updating the model parameters and can be expressed by the following formula: in Represents the temperature coefficient.

[0033] As a preferred technical solution, step 16 includes in more detail: using the model to perform multimodal downstream tasks, specifically: inputting the features of different single modalities into the model, and after multimodal fusion in step 5, inputting the obtained multimodal features into the multimodal classifier, and after activation by the softmax function, obtaining the final probability values of each category, and selecting the label with the highest probability as the final predicted category.

[0034] Optionally, to assess the model's effectiveness in addressing modal imbalance and its performance in multimodal downstream classification tasks, three evaluation metrics are used: video modality prediction accuracy, audio modality prediction accuracy, and multimodal prediction accuracy. Unimodal prediction accuracy reflects the model's ability to learn from the semantic information of each modality when handling imbalanced multimodal problems, while multimodal prediction accuracy provides an intuitive overview of the model's overall performance in multimodal downstream tasks.

[0035] Three scenario cases are given below to more intuitively demonstrate the effects and advantages of the present invention: The first scenario case: Model training and testing were performed on the CREMA-D and AVE datasets, and the results were compared with previous methods. The CREMA-D dataset contains 7,442 2-3 second video clips of 91 actors, each of which spoke a few short sentences. In addition, there are a total of 7,442 clips, which are randomly divided into 6,698 samples as training sets and 744 samples as test sets. In addition, the AVE dataset is designed for audiovisual event localization, containing 28 event categories and a total of 4,143 videos, each of which is 10 seconds long. Each video is equipped with audio and video tracks, as well as secondary annotations for event localization. The comparative effects of the present invention on multimodal classification tasks are shown in Table 1 Table 1 Comparison of multimodal sentiment analysis results on CREMA-D and AVE datasets The second scenario example evaluated the performance of the present invention under different optimizer settings for a multimodal classification task on the CREMA-D and AVE datasets. The present invention was tested with three widely used optimizers: the Adam optimizer, the AdaGrad optimizer, and the SGD optimizer. In this scenario, the test results effectively demonstrated the robustness and generalization capabilities of the present invention under different optimizer configurations. The specific performance on the multimodal classification task is shown in Table 2.

[0036] Table 2 Comparison of the effects of using different optimizers on the CREMA and AVE datasets The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An optimization method based on a multimodal social optimal model, characterized in that: The following steps are involved: Step 1: Get the original video; Step 2: Feed the original video into the model and process it: extract relevant frames from the video modality and process the audio modality into corresponding input files; Step 2: Use the pre-trained ResNet-18 model to extract frame-level features of the input video data; Step 3: Use the pre-trained ResNet-18 model to extract audio features of the input audio data; Step 4: Input the video and audio features extracted in steps 2 and 3 into the corresponding unimodal classifiers to obtain the probability scores of the video modality and audio modality; Step 5: Perform multimodal fusion on the video and audio features extracted in steps 2 and 3; Step 6: Input the fused multimodal features obtained in step 5 into the multimodal classifier to obtain the probability score of the multimodal features; Step 7: Use the cross entropy loss function to calculate the loss function of the unimodal and multimodal probability scores obtained in steps 4 and 6 respectively; Step 8: Add the three loss functions obtained in step 7 and perform backpropagation to update the parameters in the model using the stochastic gradient descent method; Step 9: Extract the gradients of the video, audio, and multimodal features calculated during backpropagation. Step 10: Calculate the multimodal utility between video and multimodal features, and between audio and multimodal features, respectively, to measure the contribution of each single modality to the final multimodal performance; Step 11: Calculate the multimodal envy between video and multimodal features, and between audio and multimodal features, respectively, to assess the potential unfairness caused by uneven resource allocation in a single modality. Step 12: Combine the multimodal utility calculated in step 10 and the multimodal envy calculated in step 11 to construct a social welfare optimization objective function. Use a linear solver to solve the function to obtain two optimal weight combinations. The optimization goal is to maximize multimodal social welfare while minimizing multimodal envy. Step 13: The updated video feature gradient and multimodal feature gradient obtained in step 9, and the updated audio feature gradient and multimodal feature gradient are weighted and summed according to the weight combination obtained in step 12, respectively, to obtain updated unimodal feature gradients, and the parameters in the modal feature extractor are updated accordingly; Step 14: Perform temperature scaling on the single-mode gradient obtained in step 13 to adjust the balance of modal learning; Step 15: The model updates parameters in a loop to alleviate the unbalanced modal problem during iteration; Step 16: When the model performs downstream tasks, the video and audio data are input into the model, and the final prediction result is obtained through the multimodal classifier. The model outputs the specific category to which the input video belongs.

2. A multimodal learning method for solving unbalanced modes according to claim 1, characterized in that: Steps 2 and 3 include in more detail: When extracting audio and visual features, you can use a pre-trained 2D or 3D deep convolutional neural network model alone, or you can use multiple pre-trained models and concatenate the extracted features as the final features to obtain audio and video: and .

3. The multimodal learning method for solving unbalanced modes according to claim 1, characterized in that: Step 4 in more detail involves obtaining the probability scores for the audio and video modalities through a unimodal classifier consisting of fully connected layers: and .

4. The multimodal learning method for solving unbalanced modes according to claim 1, characterized in that: Step 5 includes more details: the unimodal features obtained in steps 2 and 3 are and Through the splicing multimodal fusion method, multimodal features are obtained , and the multimodal feature probability scores .

5. The multimodal learning method for solving unbalanced modes according to claim 1, characterized in that: Step 10 includes more details: Multimodal utility is used to quantify the contribution of different modalities to the overall model prediction, which can be expressed by the following formula: in represents the utility of each mode, ( ) represents the probability value obtained by each modal feature after passing through the linear layer.

6. The multimodal learning method for solving unbalanced modes according to claim 1, characterized in that: Step 11 includes more details: Multimodal jealousy is used to prevent the multimodal model from preferring to learn a specific modality, which can be expressed by the following formula: in and They represent the utility of single-modal features and the utility of fused modal features, respectively.

7. The multimodal learning method for solving unbalanced modes according to claim 1, characterized in that: Step 12 includes in more detail: the objective function of multimodal social welfare optimization is defined, the goal of which is to find a set of appropriate weight distributions for multimodal features and video features, and multimodal features and audio features, respectively, so that under this distribution, the multimodal and unimodal features work together to maximize the final performance of the multimodal model, while reducing the multimodal model's preference for a specific modality. The defined multimodal social welfare objective function can be expressed by the following formula: in represents the optimal weight distribution, represents the weight of each mode, Represents the adjustable balance coefficient.

8. The multimodal learning method for solving unbalanced modes according to claim 1, characterized in that: Step 13 includes, in more detail, using the optimal weight combination obtained in step 12 to adjust the updated gradients after single-modal and multi-modal back-propagation, thereby modifying the parameters in the model to avoid imbalance between the modalities, which can be expressed by the following formula: in and represents the gradient of the model parameter update, , and represents the model parameters, and Respectively represent the weight distribution obtained by social optimization; Therefore, by adjusting the gradient size and direction of parameter updates for different modes, the model can alleviate the phenomenon of modal imbalance.

9. The multimodal learning method for solving unbalanced modes according to claim 1, characterized in that: Step 14 in more detail includes: By adjusting the temperature value, the gradient adjusted in step 13 is kept within a suitable range, which is conducive to the update of the model parameters and can be expressed by the following formula: in Represents the temperature coefficient.

10. The multimodal learning method for solving unbalanced modes according to claim 1, characterized in that: Step 16 includes in more detail: using the model to perform multimodal downstream tasks, specifically: inputting features of different single modalities into the model, after multimodal fusion in step 5, inputting the obtained multimodal features into the multimodal classifier, activating it with the softmax function, obtaining the final probability values of each category, and selecting the label with the highest probability as the final predicted category.

Images