Multi-modal multi-label emotion recognition method and system based on modal contribution evaluation

Abstract

The invention discloses a multi-modal multi-label emotion recognition method and system based on modal contribution assessment, and the method comprises the steps: obtaining original feature sequences of a text mode, a visual mode and an audio mode, mapping the original feature sequences through an encoder, and obtaining high-dimensional feature embedding of the three modes; calculating a marginal contribution degree of each mode to emotion prediction at a sample level by utilizing emotion prediction values embedded by high-dimensional features of the three modes; according to the marginal contribution degree of each modal, fusing the features of each modal to form a fused feature, and processing the fused feature through a multi-modal encoder to obtain a multi-modal integrated feature; and through a trainable label embedding sequence, learning a dependency relationship between emotion labels, embedding and inputting the multi-modal integrated features and the labels into an emotion decoder to obtain a multi-modal emotion representation of mixed emotion, and inputting the multi-modal emotion representation into a multi-label classifier to obtain a predicted emotion label. According to the invention, the efficiency of multi-modal feature fusion and the accuracy of emotion recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal emotion computing, and particularly to a multimodal multi-label emotion recognition method and system based on modal contribution evaluation, aiming to improve the accuracy and robustness of emotion recognition through dynamic modal fusion and label-dependent decoding. Background Art

[0002] Emotion recognition is a key task in emotion computing systems, aiming to perceive and understand human emotions through various modal (such as text, audio, and visual) data. Considering that multiple emotions may occur simultaneously in a single discourse expression, multimodal multi-label emotion recognition has received increasing attention, and its goal is to identify one or more emotion categories using multiple modal information to achieve more accurate and comprehensive emotion understanding.

[0003] Specifically, multimodal multi-label emotion recognition involves two key issues, namely multimodal learning and multi-label learning.

[0004] Multimodal learning mainly focuses on how to design efficient fusion methods to integrate heterogeneous information from different modalities, including early fusion, late fusion, and hybrid fusion techniques, each of which has its own advantages and disadvantages. Early fusion methods directly combine features together, late fusion methods process each modality separately and then integrate them, while hybrid fusion methods combine the advantages of both to make full use of the advantages of different modal data. In addition, the introduction of the attention mechanism enhances the model's ability to focus on key features in emotion expressions, which is crucial for integrating complex information from different modalities. However, most methods fail to fully consider the contribution differences of different modalities to the emotion recognition results in a single sample, resulting in sub-optimal cooperation between modalities. In different situations, the information of each modality has different contribution and importance to emotion expression. Therefore, how to evaluate the contribution of each modality to potential emotions and improve sample-level multimodal collaboration is an urgent problem to be solved.

[0005] Due to the complexity and subjectivity of emotion expression, a single discourse is usually annotated with multiple emotion labels. Therefore, multi-label learning in multimodal multi-label emotion recognition regards this task as a multi-label classification problem, aiming to identify and classify complex emotion states. Early methods transformed multi-label classification into multiple binary classification problems but ignored the correlation between labels. In recent years, researchers have begun to pay attention to the correlation and dependence between labels, introduced label embedding to learn the embedding representation of emotion labels, made full use of the semantic information of labels to explore cross-modal key emotion information, and avoided the interference of redundant information. Most existing methods adopt static modal fusion or independent label processing, resulting in insufficient adaptability of the model to complex emotion scenarios, and ignoring the respective expression intensities of multiple target labels and the semantic relationship between labels. How to obtain representative label embeddings is a major challenge. Summary of the Invention

[0006] Aiming at the deficiencies in the existing technology, the present invention provides a multi-modal multi-label sentiment recognition method and system based on modal contribution evaluation. By introducing a modal contribution evaluation mechanism and a sentiment label dependence decoding mechanism, it can dynamically evaluate the contributions of different modalities to sentiment recognition, effectively capture the dependence relationships between sentiment labels, and improve the efficiency of multi-modal feature fusion and the accuracy of sentiment recognition.

[0007] The present invention achieves the above technical objectives through the following technical means.

[0008] Multi-modal multi-label sentiment recognition method based on modal contribution evaluation:

[0009] Obtain the original feature sequences of three modalities: text, vision, and audio. Use an encoder to map the original feature sequences into a high-dimensional embedding space to obtain the high-dimensional feature embeddings of text, vision, and audio.

[0010] Calculate the marginal contribution degree of each modality to sentiment prediction at the sample level using the sentiment prediction values of the high-dimensional feature embeddings of text, vision, and audio.

[0011] According to the marginal contribution degrees of each modality, fuse the features of each modality to form a multi-modal fusion feature, and then process the multi-modal fusion feature through a multi-modal encoder to obtain a multi-modal integrated feature.

[0012] Learn the dependence relationships between sentiment labels through a trainable label embedding sequence, and input the multi-modal integrated feature and label embedding into a sentiment decoder to obtain a multi-modal sentiment representation of mixed sentiment.

[0013] Input the multi-modal sentiment representation into a multi-label classifier to obtain the predicted sentiment labels.

[0014] Further, the specific method for calculating the marginal contribution degree of each modality to sentiment prediction at the sample level is as follows:

[0015] Combine the text embedding, vision embedding, and audio embedding into the feature representations of different modal subsets, calculate the marginal contribution value of each modality when added to the sentiment prediction result, and obtain the modal contribution degree by normalizing the marginal contribution value of each modality.

[0016] Furthermore, the marginal contribution value is expressed as:

[0017]

[0018] where φ m is the marginal contribution value of modality m, l is the number of labels, λ is the normalization factor; y Sis the preliminary sentiment prediction value of subset S, and is the modality set, and y S = Sigmoid(H S W S + b S ), where W S is the shared weight, and b S is the shared bias;

[0019] The modality contribution degree is calculated from the ratio of the marginal contribution values.

[0020] Furthermore, weights are assigned to the features of each modality according to the modality contribution degree:

[0021] I m = α·ω m h m

[0022] where I m represents the weighted feature of modality m, α is a scaling factor, and the weight ω m = Φ m , and h m represents the high-dimensional feature embedding from modality m.

[0023] Furthermore, the multi-modal integrated feature is obtained by encoding the multi-modal fusion feature with the multi-modal encoder E mul . The multi-modal fusion feature is obtained by concatenating the weighted modality features. The multi-modal integrated feature is:

[0024] I o = E mul ([I t , I v , I a )

[0025] where I t represents the weighted feature of the text modality t, I v represents the weighted feature of the visual modality v, and I a represents the weighted feature of the audio modality a.

[0026] Furthermore, the process of obtaining the multi-modal integrated feature is:

[0027] Use the pre-trained language model to extract the semantic embedding sequence of the sentiment label, update the semantic embedding sequence through the sentiment correlation relationship, and use it as the initial value of the label embedding sequence; concatenate the multi-modal integrated feature with the updated semantic embedding sequence as the input of the sentiment decoder, and generate the multi-modal sentiment representation of the mixed sentiment through the decoding process.

[0028] Furthermore, the multi-modal emotion representation is as follows:

[0029]

[0030] where L is the updated semantic embedding sequence, PE is the positional embedding providing sequence order information, LN represents layer normalization, W D is the weight for row feature dimension transformation, b D is the bias for row feature dimension transformation, represents an intermediate quantity.

[0031] Furthermore, inputting the multi-modal emotion representation into the multi-label classifier to obtain the predicted emotion labels is specifically as follows:

[0032] y pre = Sigmoid(OW P + b P )

[0033] where W P is the weight matrix of the classifier, b P is the bias matrix of the classifier;

[0034] The prediction of the emotion labels also requires a joint loss to train and optimize the parameters:

[0035]

[0036] where β and γ are hyperparameters for balancing the loss contributions, represents the preliminary classification loss, represents the label-dependency decoding loss, represents the multi-label classification loss.

[0037] Furthermore, the label-dependency decoding loss is calculated based on the emotion correlation vector and y pre The emotion correlation vector is obtained by calculating the cosine similarity between the multi-modal emotion representation and the label embedding sequence.

[0038] A multi-modal multi-label emotion recognition system based on modal contribution evaluation includes:

[0039] A single-modal feature extraction module that encodes the original feature sequence of each modality into a high-dimensional feature embedding;

[0040] A modal contribution degree evaluation module that calculates the marginal contribution degree of each modality using the emotion prediction values of the high-dimensional feature embeddings of each modality;

[0041] A multi-modal fusion module that inputs the feature embeddings of each modality and the marginal modal contribution degrees to obtain a multi-modal integrated feature;

[0042] The label-dependent decoding module initializes the label embedding sequence, inputs the multi-modal integrated features, and generates a multi-modal sentiment representation with the label embedding sequence;

[0043] The multi-label sentiment classification module obtains the final multi-label sentiment classification result from the multi-modal sentiment representation.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) By introducing a fusion method based on modal contribution, the present invention can dynamically evaluate the contributions of different modalities (such as text, audio, and vision) to sentiment recognition at the sample level and perform weighted fusion according to the contribution values. This adaptive fusion method enables the model to more accurately capture the modal features that are most informative for sentiment recognition, thereby significantly improving the accuracy of multi-modal multi-label sentiment recognition. In addition, by quantifying the contribution of each modality to sentiment recognition, the present invention can provide a more intuitive explanation for the sentiment prediction results. The model can not only output the sentiment recognition results but also clearly indicate which modalities play a key role in sentiment judgment in a specific sample, thus providing richer explanatory information for sentiment analysis. This interpretability is particularly important for sentiment analysis systems in practical applications and helps users understand and trust the decision-making process of the model.

[0046] (2) The present invention designs a label-dependent decoding mechanism that can effectively capture the correlations and dependencies between different sentiment labels and generate more representative label embeddings. This mechanism enables the model to better understand the semantic associations between emotions when dealing with mixed emotions and complex emotion expressions, avoiding information loss or misjudgment caused by independent label processing, thereby significantly enhancing the model's recognition ability and robustness for complex emotion scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0048] Figure 1 It is a schematic diagram of the overall framework of the multi-modal multi-label sentiment recognition method based on modal contribution evaluation of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following will further describe the present invention in detail with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0050] The multimodal multi-label emotion recognition system based on modal contribution evaluation provided in this embodiment is as follows: Figure 1 As shown in the figure, through its unique modular architecture, it achieves accurate multi-label recognition in complex emotional scenes. The system first uses the feature extraction module to process the raw data of text, visual, and audio modalities, converting this data into high-dimensional feature representations, laying the foundation for subsequent emotion recognition. Next, the modal contribution evaluation module conducts in-depth analysis of the feature representations of each modality, dynamically quantifies its contribution to emotion prediction, and identifies the most informative modal features in specific emotional scenes. Subsequently, the multimodal fusion module cleverly integrates the features and contribution information of each modality to generate a unified multimodal representation. To capture the semantic associations between emotion labels, the system introduces a label-dependent decoder. This module initializes label embeddings using a pretrained language model and emotion correlation matrix, and generates a multimodal emotion representation through a decoding process. Finally, the multi-label emotion classifier receives this representation and, through an efficient classification algorithm, simultaneously identifies multiple emotion labels, meeting the analysis requirements of complex emotional scenes. The entire system is designed with modularity at its core, with clear division of labor and synergy between modules, ensuring efficient and accurate emotion recognition. By dynamically evaluating modal contributions and capturing label dependencies, the system can provide robust recognition results in complex emotional scenarios. The specific process is as follows:

[0051] S1, unimodal feature extraction: obtain the original features of the three modalities of text, vision and audio, map the original feature sequence of each modality to a high-dimensional embedding space, and obtain high-dimensional feature representations of text, vision and audio.

[0052] In this example, we first obtain the CMU-MOSEI multimodal multi-label sentiment benchmark dataset from the Internet, and record the original feature sequences of the three modalities of text (t), vision (v) and audio (a) as x t 、x v 、x a , and the set of three modes is recorded as

[0053] For each modality Using an independent single-mode encoder E m , the original feature sequence x m Mapped to the high-dimensional embedding space, we get the high-dimensional feature embedding h m The unimodal encoder adopts a Transformer-based encoder architecture, where the attention layer is replaced by a state-space model to efficiently process long sequence data and encode the long-distance dependencies of each modality to extract global features. The specific process is as follows:

[0054]

[0055] Among them, n m represents the sequence length, d represents the embedding dimension, and E m represents the encoder of modality m.

[0056] According to the above process, the text embedding h t , visual embedding h v and audio embedding h a are finally obtained.

[0057] S2, Modality Contribution Evaluation: Calculate the marginal contribution of each modality to sentiment prediction at the sample level using the sentiment prediction values represented by the high-dimensional embedding features of each modality.

[0058] In this embodiment, in order to quantify the contribution value of the modality, first combine the text embedding h t , visual embedding h v and audio embedding h a obtained in S1 to form various integrated feature embeddings. For any subset S belonging to the modality set , is the combination of the feature embeddings of subset S, where if the text modality t ∈ S, then in the combination of the feature embeddings of subset S otherwise, is a zero vector; the same applies to the visual modality v and the audio modality a. The corresponding combined feature H S is expressed as:

[0059]

[0060] where, W H and b H are the weights and biases for feature dimension conversion.

[0061] Then, introduce the Shapley value in game theory to calculate the marginal contribution value of the modality. For each modality, calculate the multi-modal prediction benefit brought when it is added, and then evenly distribute this benefit over all possible addition orders. Specifically, the calculation formula for the marginal contribution value of modality m is:

[0062]

[0063] where, l represents the number of labels.

[0064] To represent all possible combination situations of subset S, set the normalization factor For subset S, y S = Sigmoid(H S W S + b S) is the preliminary sentiment prediction value obtained by the combination feature input classifier of subset S, where W S , b S are the shared weight and bias respectively.

[0065] For example, the process of calculating the marginal contribution value of the text modality is as follows:

[0066]

[0067] Finally, the modality contribution degree Φ m is calculated from the ratio of the marginal contributions .

[0068] S3, multimodal feature fusion: Using the marginal contribution degrees of each modality, fuse the features of each modality and input them into the multimodal encoder for cross-modal feature alignment to obtain the multimodal integrated feature.

[0069] In this embodiment, first, according to the marginal contribution degrees of each modality in S2, assign weights to the features of each modality (the weights are proportional to the contributions of the modalities), splice the weighted modality features to form the multimodal fusion feature. Then, input the multimodal fusion feature into the multimodal encoder E mul for feature encoding to obtain the final multimodal integrated feature I o . The multimodal encoder adopts the same architecture as the unimodal encoder and will not be elaborated here. The specific process of obtaining the multimodal integrated feature is as follows:

[0070] I m = α·ω m h m

[0071] I o = E mul ([I t , I v , I a )

[0072] where α is a scaling factor, h m represents the high-dimensional feature embedding from modality m, the weight ω m = Φ m , I m represents the weighted feature of modality m, I t represents the weighted feature of the text modality t, I v represents the weighted feature of the visual modality v, I a represents the weighted feature of the audio modality a.

[0073] S4, Label-dependent decoding: Learn the dependencies between sentiment labels through a trainable label embedding sequence; input the multi-modal integrated features and label embeddings into the sentiment decoder to obtain a multi-modal sentiment representation of mixed sentiment.

[0074] In this embodiment, the label embedding sequence is a set of trainable parameters, and the sentiment decoder is a decoder architecture using a state space model. The steps of specifically using the sentiment decoder for label-dependent decoding are as follows:

[0075] (1) Initialize the label embedding sequence: Use a pre-trained language model to extract the semantic embedding sequence of sentiment labels, and update the semantic embedding sequence through sentiment association relationships, and use the updated semantic embedding sequence as the initial value of the label embedding sequence.

[0076] Specifically, first pre-define the label descriptions of each sentiment label. For example, the label description corresponding to the sentiment "happy" is "happiness". Input the label description corresponding to the sentiment label into the pre-trained language model BERT to obtain the word embedding of the label description as the semantic embedding of the sentiment label. Define the semantic embedding corresponding to the i-th sentiment label as (d represents the embedding dimension), and form the semantic embedding sequence of all sentiment labels (l represents the number of labels).

[0077] Secondly, update the semantic embedding sequence using sentiment association relationships. In this embodiment, the Plutchik emotion wheel is used to generate an emotion association matrix R, which samples from a Gaussian distribution to model the relationship between the i-th sentiment label and the j-th sentiment label:

[0078]

[0079] where σ is the standard deviation, is the normalization factor to ensure that the sum of probabilities is 1. Δ i,j quantifies the emotional distance between sentiment labels.

[0080] Finally, the label embedding sequence is Use the updated semantic embedding sequence of sentiment labels L = R·L 0 as the initial value of the label embedding sequence.

[0081] (2) Generate a multi-modal sentiment representation: Input the multi-modal integrated features and the updated semantic embedding sequence of sentiment labels into the sentiment decoder to obtain a multi-modal sentiment representation.

[0082] Specifically, the multi-modal integrated representation in S3 is concatenated with the label embedding sequence obtained in (1) to obtain the input of the sentiment decoder. The decoder adopts a decoder architecture based on Transformer, where the attention layer is replaced by the state space model SSM. Through the sentiment decoder, a multi-modal sentiment representation O of mixed sentiment is generated:

[0083]

[0084] Among them, PE is the position embedding that provides sequence order information, LN represents layer normalization, W D , b D are the weights and biases for feature dimension transformation, represents an intermediate quantity, which is the multi-modal feature output by the state space model SSM.

[0085] (3) Generate the sentiment correlation vector: To ensure that the generated sentiment representation pays more attention to the target sentiment label, calculate the cosine similarity between the multi-modal sentiment representation O and the label embedding sequence L to obtain the sentiment correlation vector;

[0086] Specifically, the process of calculating the sentiment correlation vector c is as follows:

[0087]

[0088] Among them, O i represents the i-th vector of the sentiment representation, and L i represents the embedding of the i-th label.

[0089] S5, multi-label sentiment classification: Input the multi-modal sentiment representation into the multi-label classifier to obtain the predicted sentiment labels.

[0090] In this embodiment, a linear classifier is defined as the multi-label classifier. Input the multi-modal sentiment representation into the classifier to obtain the predicted value of the sentiment label:

[0091] y pre = Sigmoid(OW P + b P )

[0092] Among them, W P , b P are the weight matrix and bias matrix of the classifier.

[0093] For a sample, given the true label y and the sentiment prediction In this embodiment, binary cross-entropy loss is used to calculate the classification loss:

[0094]

[0095] Among them, y i represents the i-th true label of the sample, and

[0096] In this embodiment, the preliminary classification loss is expressed as and the final multi-label classification loss is expressed as where pre is the preliminary prediction result in S2, and y

[0097] Then, this embodiment uses the Kullback-Leibler (KL) divergence to measure the difference between the relevant vector and the label logical value. According to the sentiment correlation vector c in S4 and the final multi-label prediction result y pre calculate the label dependence decoding loss

[0098] By combining the preliminary classification loss the label dependence decoding loss and the multi-label classification loss the final loss is obtained:

[0099]

[0100] where β and γ are hyperparameters.

[0101] During training, this embodiment is optimized by the final loss, the optimizer used is Adam, the learning rate is 5e-5, the number of training epochs is 20, and the batch size is 64. Finally, after training for 20 epochs on the training set, the model parameters corresponding to the epoch with the best sentiment recognition performance on the validation set are selected as the optimal parameters.

[0102] In this embodiment, the method is experimentally verified on the CMU-MOSEI dataset with aligned and unaligned settings, and the most advanced open-source methods in recent years (HHMPN, TAILORl, and AMP) are reproduced. The performance comparison results of each method are shown in Table 1.

[0103] Table 1 Performance comparison results with advanced methods in recent years

[0104]

[0105] Among them, Acc is the accuracy, MiF1 is the micro F1-score, MaF1 is the macro F1-score, and Params is the number of model parameters. The higher the values of Acc, MiF1, and MaF1, the better the accuracy and robustness of the model, while the lower the Params value, the more lightweight the model.

[0106] As can be seen from Table 1, the method of the present invention has better performance. In the alignment setting, compared with other multimodal multi-label methods, the method of the present invention shows excellent performance on all evaluation metrics, demonstrating that multimodal information can be utilized more effectively. At the same time, in the unaligned setting, the method of the present invention not only performs better than advanced methods in recent years in terms of precision-related metrics, but also has a significant advantage in terms of model complexity, requiring fewer parameters and a more lightweight model.

[0107] To analyze the impact of different modules on the model performance, ablation experiments were conducted on the CMU-MOSEI dataset under the alignment setting. The experimental results are shown in Table 2.

[0108] Table 2 Results of Ablation Experiments

[0109]

[0110] As shown in Table 2: Obviously, deleting the modality contribution evaluation module and the multimodal fusion module will both lead to a significant decline in the performance of all three metrics, which shows the important role of the modality contribution evaluation module in accurately evaluating the contribution of each modality, improving the model's ability to make informed decisions, and the multimodal fusion module is also crucial for effectively integrating information from various modalities, thus promoting a more comprehensive understanding of the data. Deleting the label dependence decoding module will lead to a significant decline in the model's performance, indicating that decoding the fused multimodal features is crucial for accurate emotion recognition. In addition, the effectiveness of the multi-label classification loss and the label dependence decoding loss can be seen. The results show that constraining the prediction of labels and the decoding of emotion representations can significantly improve the performance of the model. The method of the present invention enables the model to better capture the nuances of emotional states and improve the accuracy of its predictions.

[0111] The embodiments described in the present invention are only preferred examples and do not limit the form of the present invention. Without departing from the core essence of the present invention, all obvious improvements or variations made by those skilled in the art shall be included in the protection scope of the present invention.

Claims

1. A multi-modal multi-label sentiment recognition method based on modal contribution evaluation, characterized in that: Obtain the original feature sequences of three modalities: text, vision, and audio. Use an encoder to map the original feature sequences into a high-dimensional embedding space to obtain the high-dimensional feature embeddings of text, vision, and audio; Calculate the marginal contribution degree of each modality to sentiment prediction at the sample level using the sentiment prediction values of the high-dimensional feature embeddings of text, vision, and audio; According to the marginal contribution degrees of each modality, fuse the features of each modality to form multi-modal fusion features, and then process the multi-modal fusion features through a multi-modal encoder to obtain multi-modal integrated features; Learn the dependency relationship between sentiment labels through a trainable label embedding sequence, and input the multi-modal integrated features and label embeddings into a sentiment decoder to obtain a multi-modal sentiment representation of mixed sentiment; Input the multi-modal sentiment representation into a multi-label classifier to obtain the predicted sentiment labels.

2. The multimodal multi-label sentiment recognition method based on modal contribution evaluation according to claim 1, wherein The calculation of the marginal contribution degree of each modality to sentiment prediction at the sample level is specifically as follows: Combine the text embedding, vision embedding, and audio embedding into the feature representations of different modality subsets, calculate the marginal contribution value of each modality when added to the sentiment prediction result, and obtain the modality contribution degree by normalizing the marginal contribution value of each modality.

3. The multimodal multi-label sentiment recognition method based on modal contribution evaluation according to claim 2, characterized in that The marginal contribution value is expressed as: Among them, φ m is the marginal contribution value of modality m, l is the number of labels, and λ is the normalization factor; y S is the preliminary sentiment prediction value of subset S, and S ∈ M, where M is the modality set, and y S = Sigmoid(H S W S + b S ), W S is the shared weight, and b S is the shared bias; The modality contribution degree is calculated from the ratio of the marginal contribution values.

4. The multi-modal multi-label sentiment recognition method based on modal contribution evaluation according to claim 3, characterized in that Allocate weights to the features of each modality according to the modality contribution degree: I m = α·ω m h m Among them, I m represents the weighted feature of modality m, α is a scaling factor, and the weight ω m = Φ m , h m represents the high-dimensional feature embedding from modality m.

5. The multimodal multi-label sentiment recognition method based on modal contribution evaluation according to claim 4, characterized in that The multi-modal integrated feature is encoded by the multi-modal encoder E mul and is obtained by encoding the multi-modal fusion feature, which is obtained by concatenating the weighted modal features. The multi-modal integrated feature is as follows: I o = E mul ([I t , I v , I a ) Among them, I t represents the weighted feature of the text modality t, and I v represents the weighted feature of the visual modality v, and I a represents the weighted feature of the audio modality a.

6. The multi-modal multi-label sentiment recognition method based on modal contribution evaluation according to claim 5, characterized in that The process of obtaining the multi-modal integrated features is as follows: Use a pre-trained language model to extract the semantic embedding sequence of sentiment labels, update the semantic embedding sequence through sentiment association relationships as the initial value of the label embedding sequence; splice the multi-modal integrated features with the updated semantic embedding sequence as the input of the sentiment decoder, and generate a multi-modal sentiment representation of mixed sentiment through the decoding process.

7. The multimodal multi-label sentiment recognition method based on modal contribution evaluation according to claim 6, characterized in that, The multi-modal sentiment representation is: Among them, L is the updated semantic embedding sequence, PE is the positional embedding that provides sequence order information, LN represents layer normalization, W D is the weight for row feature dimension transformation, b D is the bias for row feature dimension transformation, represents an intermediate quantity.

8. The multi-modal multi-label sentiment recognition method based on modal contribution evaluation according to claim 7, wherein The input of the multi-modal sentiment representation into the multi-label classifier to obtain the predicted sentiment labels is specifically as follows: y pre = Sigmoid(OW P + b P ) Among them, W P is the weight matrix of the classifier, and b P is the bias matrix of the classifier; The prediction of the sentiment labels also requires a joint loss to train and optimize the parameters: where β and γ are hyperparameters that balance the loss contributions, represents the preliminary classification loss, represents the label-dependency decoding loss, represents the multi-label classification loss.

9. The multimodal multi-label sentiment recognition method based on modal contribution evaluation according to claim 8, characterized in that The label-dependent decoding loss is calculated based on the sentiment correlation vector and y pre and is obtained by calculating the cosine similarity between the multi-modal sentiment representation and the label embedding sequence.

10. A system for implementing the multi-modal multi-label sentiment recognition method based on modal contribution evaluation according to any one of claims 1-9, characterized in that, Including: A single-modal feature extraction module that encodes the original feature sequence of each modality into a high-dimensional feature embedding; A modality contribution degree evaluation module that calculates the marginal contribution degree of each modality using the sentiment prediction values of the high-dimensional feature embeddings of each modality; A multi-modal fusion module that inputs the feature embeddings of each modality and the marginal modality contribution degrees to obtain multi-modal integrated features; A label dependency decoding module that initializes the label embedding sequence, inputs the multi-modal integrated features, and generates a multi-modal sentiment representation with the label embedding sequence; A multi-label sentiment classification module that obtains the final multi-label sentiment classification result from the multi-modal sentiment representation.

Images