Text-prior multi-modal sentiment analysis method and system

Abstract

The application provides a text-priority multimodal sentiment analysis method and system, which respectively constructs private expert networks of text, video and audio, is used for capturing unique sentiment expression features of each mode, simultaneously introduces a shared expert network to model general sentiment semantics across modes, so that collaborative modeling of multimodal information is realized, then a text mode is used as a leading mode to start a sentiment analysis process, a preliminary judgment is given based on text information, according to a confidence degree, video and audio information are gradually introduced according to needs to perform incremental supplement for sentiment classification prediction, a need-decoding strategy consistent with human cognition is realized, and a final sentiment category is obtained. Through simulating a human cognitive process of integrating multi-source information according to needs, the application reduces reasoning time and resource consumption, makes the model decision process more natural and interpretable, and is closer to the characteristics of different modal information being unbalanced and non-equivalent in actual application.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to a text-first multimodal sentiment analysis method and system. Background Technology

[0002] Multimodal sentiment analysis, aiming to identify sentiment tendencies from multiple modalities of data (e.g., text, video, and audio), has become a hot research area in affective computing in recent years. Compared to traditional sentiment analysis, which relies solely on single text signals, multimodal sentiment analysis can extract richer and more comprehensive emotional cues from multiple sources such as facial expressions, body language, and tone of voice, thus improving the accuracy and robustness of sentiment recognition. In scenarios such as short video comments, live-streaming e-commerce, and online classroom interactions, users often express their sentiment tendencies through language, but may also convey their true emotions through non-verbal signals such as tone of voice, facial expressions, and even pauses. For example, in a user review video, the text "This product is okay" may seem neutral, but combined with a dismissive tone and a disappointed facial expression, it can be inferred that the sentiment is negative. Multimodal sentiment analysis can integrate emotional information from different modalities, thereby improving the accuracy of sentiment recognition and providing more comprehensive support for upper-level applications such as recommendation systems, human-computer interaction, and public opinion monitoring. With the rapid development of video social networking and online interaction scenarios, multimodal sentiment analysis has become a research hotspot in both academia and industry.

[0003] However, due to significant differences in emotional expression and information delivery methods among different modalities, one of the core challenges of multimodal sentiment analysis is the effective fusion of emotional information from text, video, and audio to leverage modal complementarity. Existing methods typically utilize mechanisms such as cross-modal attention, bidirectional interaction, and dynamic fusion to achieve fine-grained modeling of semantic relationships between different modalities and full utilization of complementary information. On the other hand, multimodal data often exhibits significant imbalances in real-world scenarios: different modalities are often unequal in terms of information quantity, quality, and usability. Text typically contains rich emotional cues and plays a dominant role in prediction, while the emotional information from modalities such as video and audio is often relatively limited, resulting in a weaker impact on decision-making. Therefore, early multimodal sentiment analysis models were prone to problems such as excessive dominance of strong modalities (e.g., text modality suppressing other modalities) or neglect of weak modal contributions, leading to unsatisfactory fusion results. To alleviate modal imbalance, existing methods mainly employ adaptive adjustment of modal weights, weak modality enhancement training, or distillation correction to maintain stable sentiment recognition capabilities even when contributions from different modalities are uneven.

[0004] In recent years, the application of large-scale models in sentiment analysis has received widespread attention. Leveraging massive pre-trained data and powerful semantic representation capabilities, large-scale models can capture subtle semantic and emotional cues in text, exhibiting greater depth of understanding and generalization ability compared to traditional lightweight models. In unimodal text sentiment analysis, large-scale models can not only achieve more accurate sentiment classification but also handle complex implicit sentiment expressions and aspect-level sentiment tendencies. In multimodal sentiment analysis, combining large-scale text models with visual and audio encoders enables deep fusion of cross-modal information, enhancing the accuracy and robustness of sentiment recognition. Furthermore, large-scale models provide interpretability support for sentiment analysis, enabling models not only to predict sentiment tendencies but also to identify the sources and triggers of sentiment, providing more reliable sentiment analysis capabilities for higher-level applications such as public opinion monitoring, human-computer interaction, and customer service Q&A.

[0005] Existing multimodal sentiment analysis methods generally assume that video, audio, and text modalities are equally important, thus often forcibly integrating information from all modalities during inference. However, in many scenarios, sentiment can be accurately predicted using only text. In these cases, introducing additional video and audio not only increases computational costs but may also introduce weak or low-quality modalities into the model, negatively impacting the final prediction results. From the perspective of human cognition and decision-making processes, sentiment judgment often follows an "integration on demand" strategy. That is, people typically prioritize the most salient or informative modality for judgment, only referring to other modalities for supplementation when information from a single modality is insufficient or ambiguous. Existing multimodal sentiment analysis methods still lack the ability to model the necessity and dynamic contribution of modalities to achieve more realistic, efficient, and robust sentiment recognition. Summary of the Invention

[0006] In view of the above, the main objective of this invention is to propose a text-first multimodal sentiment analysis method and system to solve the aforementioned technical problems.

[0007] This invention proposes a text-first multimodal sentiment analysis method, which includes the following steps:

[0008] Step 1: Given an instance, extract the preliminary semantic representation of the instance's text, video, and audio.

[0009] Step 2: Construct private expert networks for text, video, and audio modalities respectively, and generate private sentiment feature vectors for the corresponding modalities in the preliminary semantic representation of text, video, and audio through the private expert networks;

[0010] A shared expert network is used to generate general sentiment feature vectors for corresponding modalities in the preliminary semantic representations of text, video, and audio.

[0011] Step 3: Fuse the private sentiment feature vector with the text modality part of the general sentiment feature vector to generate the final sentiment feature vector of a single text.

[0012] Step 4: Initiate the sentiment analysis process with text modality as the primary driver, use the final sentiment feature vector of a single text to perform sentiment classification prediction, obtain the sentiment category, and calculate the confidence score of the sentiment category;

[0013] Step 5: Based on the confidence level of the emotion category, incrementally supplement the video and audio with private emotion feature vectors and general emotion feature vectors as needed, and then perform emotion classification prediction to achieve an on-demand decoding strategy consistent with human cognition and obtain the final emotion category.

[0014] This invention also proposes a text-first multimodal sentiment analysis system, wherein the system applies the text-first multimodal sentiment analysis method described above, and the system includes:

[0015] A shared-private expert-based coding module is used for:

[0016] Given an instance, extract preliminary semantic representations of the instance's text, video, and audio.

[0017] Private expert networks are constructed for text, video, and audio modalities respectively. Private sentiment feature vectors for the corresponding modalities are generated through the private expert networks in the preliminary semantic representation of text, video, and audio.

[0018] A shared expert network is used to generate general sentiment feature vectors for corresponding modalities in the preliminary semantic representations of text, video, and audio.

[0019] The text-first incremental decoding module is used for:

[0020] The private sentiment feature vector is fused with the text modality part of the general sentiment feature vector to generate the final sentiment feature vector of a single text.

[0021] The sentiment analysis process is initiated primarily based on the text modality. The final sentiment feature vector of a single text is used to predict the sentiment category, and the confidence level of the sentiment category is calculated.

[0022] Based on the confidence level of the sentiment category, private sentiment feature vectors corresponding to videos and audio, as well as general sentiment feature vectors, are incrementally introduced as needed for supplementation. Then, sentiment classification prediction is performed to achieve an on-demand decoding strategy consistent with human cognition and obtain the final sentiment category.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] 1. This invention employs a text-first incremental decoding mechanism. The model initially makes predictions based solely on text information, only introducing other modalities such as video and audio as needed when the confidence level is insufficient. This avoids a large amount of unnecessary cross-modal feature extraction and fusion computation, significantly reducing inference time and resource consumption, making it particularly suitable for online application scenarios with high real-time requirements.

[0025] 2. This invention employs a shared-private expert coding structure, which not only mines unique emotional expression features of each modality but also models cross-modal universal emotional semantics, improving the discriminativeness and complementarity of multimodal representations. Combined with knowledge distillation technology, the text-driven student model can learn the fusion decision-making knowledge of the full-modal teacher model, thus maintaining high recognition accuracy even when using only partial modal reasoning. This is particularly suitable for real-world scenarios with high noise and unstable quality of weak modal information.

[0026] 3. This invention simulates the human cognitive process of "integrating" multi-source information on demand, initiates analysis with text as the main channel, and dynamically decides whether to supplement other modalities based on confidence level, making the model decision-making process more natural and interpretable, and closer to the characteristics of uneven and unequal information of different modalities in actual applications.

[0027] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0028] Figure 1 This is a flowchart of the text-first multimodal sentiment analysis method proposed in this invention;

[0029] Figure 2 This is a schematic diagram of the structure of the text-first multimodal sentiment analysis system proposed in this invention. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0031] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0032] Please see Figure 1This embodiment provides a text-first multimodal sentiment analysis method, which includes the following steps:

[0033] Step 1: Given an instance, extract the preliminary semantic representation of the instance's text, video, and audio.

[0034] In step 1, preliminary semantic representations of the text, video, and audio of the instance are extracted, and the following relationships exist:

[0035] ;

[0036] in, , and These are text encoders, video encoders, and audio encoders, respectively. , and The semantic matrix representations are for the input text, video, and audio, respectively. They are text, video, and audio, respectively. , x For a given instance, in implementing the invention, the text encoder can use pre-trained language models such as Bert and Roberta, the video encoder can use pre-trained video models such as VideoMAE and TimeSformer, and the audio encoder can use pre-trained speech models such as wav2vec and WavLM. Since emotional signals mainly originate from the face region, face detection models such as MTCNN can be used first to locate the face region from the input video, and then models such as MA-Net can be used to extract the deep semantic features of the face region.

[0037] Step 2: Construct private expert networks for text, video, and audio modalities respectively, and generate private sentiment feature vectors for the corresponding modalities in the preliminary semantic representation of text, video, and audio through the private expert networks;

[0038] A shared expert network is used to generate general sentiment feature vectors for corresponding modalities in the preliminary semantic representations of text, video, and audio.

[0039] In step 2, private sentiment feature vectors for the corresponding modalities in the preliminary semantic representations of text, video, and audio are generated through a private expert network, and the following relationship exists:

[0040] ;

[0041] in, , and These are private expert networks corresponding to text, video, and audio, respectively, each consisting of multiple stacked Transformer layers; , and These are private sentiment feature vectors for text, video, and audio, respectively.

[0042] In step 2, a shared expert network is used to generate general sentiment feature vectors for the corresponding modalities in the preliminary semantic representations of text, video, and audio, which correspond to the following relationship:

[0043] ;

[0044] in, To share the expert network, it consists of multiple stacked Transformer layers; , and These are general sentiment feature vectors for text, video, and audio, respectively.

[0045] Step 3: Fuse the private sentiment feature vector with the text modality part of the general sentiment feature vector to generate the final sentiment feature vector of a single text.

[0046] In step 3, the private sentiment feature vector and the text modality portion of the general sentiment feature vector are fused to generate the final sentiment feature vector of a single text, with the following relationship:

[0047] ;

[0048] in, This is the first LSTM unit used for fusing text information. This refers to the valid text information after input gate filtering. Let be the learnable initial state vector. This is a vector concatenation operation. This is the final sentiment feature vector for a single text.

[0049] Step 4: Initiate the sentiment analysis process with text modality as the primary driver, use the final sentiment feature vector of a single text to perform sentiment classification prediction, obtain the sentiment category, and calculate the confidence score of the sentiment category;

[0050] In step 4, the sentiment analysis process is initiated primarily based on the text modality. Sentiment classification prediction is performed using the final sentiment feature vector of a single text to obtain the sentiment category, and the confidence level of the sentiment category is calculated. The following relationship exists:

[0051] ;

[0052] in, For classification networks that utilize only textual information, multi-layer feedforward neural networks with softmax output layers are typically used. The entropy is a function used to calculate the probability distribution. To predict the distribution of sentiment categories, The confidence level of the prediction result is given. This invention uses entropy to evaluate the confidence level of the model's prediction results; the smaller the entropy, the higher the confidence level, and the larger the entropy, the lower the confidence level.

[0053] Step 5: Based on the confidence level of the emotion category, incrementally supplement the video and audio with private emotion feature vectors and general emotion feature vectors as needed, and then perform emotion classification prediction to achieve an on-demand decoding strategy consistent with human cognition and obtain the final emotion category.

[0054] In step 5, based on the confidence level of the emotion category, private emotion feature vectors and general emotion feature vectors corresponding to the video and audio are incrementally introduced as needed for supplementation, and then emotion classification prediction is performed to achieve an on-demand decoding strategy consistent with human cognition, thus obtaining the final emotion category. The specific steps include the following:

[0055] The sentiment category predicted based on the text is designated as the first sentiment category;

[0056] Given a confidence threshold, the confidence threshold is compared with the confidence of the first sentiment category; if the confidence of the first sentiment category is less than or equal to the confidence threshold, the first sentiment category is determined to be highly credible, and there is no need to introduce information from other modalities, and the first sentiment category is taken as the final sentiment category.

[0057] If the confidence score of the first sentiment category is greater than the confidence threshold, then the confidence score of the first sentiment category is low, and more sentiment information needs to be extracted from the input video for more accurate prediction. The general sentiment feature vector and the private sentiment feature vector of the video are used to calculate the final sentiment feature vector that fuses the text and video information through the second LSTM unit, and then sentiment classification prediction is performed to obtain the second sentiment category. The confidence score of the second sentiment category is then calculated, and the following relationship exists:

[0058] ;

[0059] in, This is a second LSTM unit used to fuse text and video information. The final vector of sentiment features, which integrates text and video information. The valid video information after input gate filtering. To utilize classification networks that combine text and video information, This is a general sentiment feature vector for videos. This is the private sentiment feature vector of the video. As the second emotion category, The confidence level for the second sentiment category;

[0060] Then, the confidence level of the second sentiment category is compared with the confidence threshold again. If the confidence level of the second sentiment category is less than or equal to the confidence threshold, the second sentiment category is determined to be highly credible, and there is no need to introduce information from other modalities. The second sentiment category is then used as the final sentiment category.

[0061] If the confidence score of the second sentiment category is greater than the confidence threshold, the second sentiment category is considered to have low confidence, and more sentiment information needs to be extracted from the input audio for more accurate prediction. The general sentiment feature vector and the private sentiment feature vector of the audio are used to calculate the final sentiment feature vector of the three modalities through the third LSTM unit, and then sentiment classification prediction is performed to obtain the third sentiment category, which is taken as the final sentiment category. The following relationship exists:

[0062] ;

[0063] in, The final sentiment feature vector is obtained by fusing information from the three modalities. This is the third LSTM unit used to fuse information from three modes. This refers to the valid audio information after being filtered by the input gate.

[0064] Given an input instance The final prediction result of the text-first incremental decoding module This can be uniformly represented as follows:

[0065]

[0066] in, A confidence threshold specified by the user. During the implementation of the invention, a smaller... A higher value generally leads to better sentiment classification performance, while a higher value... This value can improve inference efficiency. It should be noted that this only applies when selecting the fused video modality ( Only then can the general sentiment feature vector of the video be calculated. and private sentiment feature vector Correspondingly, only when audio modalities are introduced ( Only then can the general emotional feature vector of the audio be calculated. and private sentiment feature vector Experimental results in common scenarios show that approximately 50% of input instances can obtain reliable sentiment predictions using only text, thus saving significant inference time.

[0067] As a preferred embodiment of the present invention, the training process of the private expert network and the shared expert network specifically includes the following steps:

[0068] Three classification networks are constructed to assist training. Each network takes the general and private sentiment features of its respective modality as input and calculates the prediction results for the sentiment category. The corresponding process follows the following relationship:

[0069] ;

[0070] in, , and These are classification networks used to assist training, corresponding to text, video, and audio, respectively. , and The sentiment category prediction results are for text only, video only, and audio only, respectively.

[0071] Based on the predicted sentiment category and the true sentiment category, the first multi-task learning cost is defined, and the corresponding process has the following relationship:

[0072] ;

[0073] in, The training set is manually labeled. For input examples The true sentiment category, CE is a function that calculates the cross-entropy of two discrete probability distributions. The cost of first multi-task learning;

[0074] Based on the general and private sentiment features of each modality, an orthogonal regularization term is defined, and the corresponding process has the following relationship:

[0075] ;

[0076] in, This is the transpose of the vector. The dot product of vectors, This is an orthogonal regularization term;

[0077] Based on the first multi-task learning cost and orthogonal regularization term Define the first total cost of the shared-private expert joint training phase, and the corresponding process has the following relationship:

[0078] ;

[0079] in, For the weights corresponding to the orthogonal regularization term, As the first total cost;

[0080] By minimizing the first total cost Until convergence, the parameters in the private expert network and the shared expert network are optimized to achieve joint training of the shared-private experts.

[0081] In this embodiment, a multi-task learning method is used to achieve collaborative training between shared experts and private experts, which not only preserves the differentiated characteristics between each modality, but also promotes their effective alignment in the shared semantic space, thereby improving the effect and robustness of multimodal fusion.

[0082] In a preferred embodiment of the present invention, steps 1 to 5 above are implemented using a text-first multimodal sentiment analysis model. The training method of the text-first multimodal sentiment analysis model includes the following steps:

[0083] Given a training set, which includes several given instances and their corresponding true classes;

[0084] Using the training set as input, execute steps 1 to 5 above. During the execution, disable the confidence calculation and comparison of sentiment categories, force full-path forward propagation, and obtain the first sentiment category, the second sentiment category, and the third sentiment category.

[0085] Based on the given true categories and the first, second, and third sentiment categories, the second multi-task learning cost is defined, and the corresponding process has the following relationship:

[0086] ;

[0087] in, The training set is manually labeled. For input examples The true sentiment category, CE is a function that calculates the cross-entropy of two discrete probability distributions. The cost of the second multitasking learning;

[0088] Taking the third emotion category as the teacher and the first and second emotion categories as the students, the cost of knowledge distillation is defined, and the corresponding process has the following relationship:

[0089] ;

[0090] in, The cost of knowledge distillation To calculate the KL divergence of two probability distributions;

[0091] Using the information output by each LSTM unit, mutual information constraint terms are defined, and the corresponding process has the following relationship:

[0092] ;

[0093] in, This is a function for calculating the cosine similarity between two vectors. This is a multi-layer feedforward neural network; cosine similarity is mainly used to capture redundancy in the linear direction, but its effect on suppressing nonlinear dependencies is limited. Therefore, this invention first performs a nonlinear transformation on the feature vectors, and then minimizes the transformed cosine similarity, thereby approximately achieving mutual information constraints and reducing redundant information between incremental modal information and existing modal information.

[0094] According to the second multi-task learning cost The cost of knowledge distillation Mutual information constraints Define the second total cost during the global training phase. The corresponding process has the following relationship:

[0095] ;

[0096] in, and These are the weights of the knowledge distillation cost and the mutual information constraint term, respectively. This is the second total cost;

[0097] By minimizing the second total cost The process continues until convergence, optimizing the parameters in the text-first multimodal sentiment analysis model to train the model.

[0098] In this embodiment, a knowledge distillation mechanism is introduced, which uses a full-modal teacher model to perform knowledge transfer and global compensation on a partial-modal student model, enabling the lightweight text-dominated model to also acquire multimodal semantic capabilities, thereby maintaining high sentiment analysis performance while maintaining efficient inference speed.

[0099] Please refer to Figure 2 This embodiment also provides a text-first multimodal sentiment analysis system, wherein the system applies the text-first multimodal sentiment analysis method described above, and the system includes:

[0100] A shared-private expert-based coding module is used for:

[0101] Given an instance, extract preliminary semantic representations of the instance's text, video, and audio.

[0102] Private expert networks are constructed for text, video, and audio modalities respectively. Private sentiment feature vectors for the corresponding modalities are generated through the private expert networks in the preliminary semantic representation of text, video, and audio.

[0103] A shared expert network is used to generate general sentiment feature vectors for corresponding modalities in the preliminary semantic representations of text, video, and audio.

[0104] The text-first incremental decoding module is used for:

[0105] The private sentiment feature vector is fused with the text modality part of the general sentiment feature vector to generate the final sentiment feature vector of a single text.

[0106] The sentiment analysis process is initiated primarily based on the text modality. The final sentiment feature vector of a single text is used to predict the sentiment category, and the confidence level of the sentiment category is calculated.

[0107] Based on the confidence level of the sentiment category, private sentiment feature vectors corresponding to videos and audio, as well as general sentiment feature vectors, are incrementally introduced as needed for supplementation. Then, sentiment classification prediction is performed to achieve an on-demand decoding strategy consistent with human cognition and obtain the final sentiment category.

[0108] In the above scheme, a shared-private expert coding module is used to construct private expert networks for text, video and audio respectively to capture the unique emotional expression features of each modality. At the same time, a shared expert network is introduced to model the general emotional semantics across modalities, thereby realizing the collaborative modeling of multimodal information.

[0109] The text-first incremental decoding module initiates the sentiment analysis process primarily based on the text modality. It first makes an initial judgment based on the text information, and then, according to the confidence level, incrementally introduces video and audio information as needed to supplement the analysis, thereby achieving an on-demand decoding strategy consistent with human cognition.

[0110] In a preferred embodiment of the present invention, private expert networks for text, video, and audio are constructed based on a shared-private expert encoding module to capture the unique emotional expression features of each modality. Simultaneously, a shared expert network is introduced to model cross-modal general emotional semantics, thereby achieving collaborative modeling of multimodal information. Specifically, given an input instance... , We will first learn the preliminary semantic representations of text, video, and audio, respectively:

[0111]

[0112] in, , and These are text encoders, video encoders, and audio encoders, respectively. , and The semantic matrix representations are for the input text, video, and audio, respectively. They are text, video, and audio, respectively. Let x be a given instance. In implementation, the text encoder can use pre-trained language models such as Bert and Roberta, the video encoder can use pre-trained video models such as VideoMAE and TimeSformer, and the audio encoder can use pre-trained speech models such as wav2vec and WavLM. Since emotional signals mainly originate from the face region, face detection models such as MTCNN can be used first to locate the face region from the input video, and then models such as MA-Net can be used to extract the deep semantic features of the face region.

[0113] To better learn the unique sentiment features of each modality and the general sentiment features across modalities, a shared expert network and three private expert networks are introduced. The private expert networks, based on the initial semantic representations of text, video, and audio, further learn private sentiment features, as shown below:

[0114]

[0115] in, , and These are private expert networks corresponding to text, video, and audio, respectively, each consisting of multiple stacked Transformer layers; , and These are private sentiment feature vectors for text, video, and audio, respectively.

[0116] The shared expert network learns corresponding general sentiment features based on the initial semantic representations of each modality, as shown below:

[0117]

[0118] in, To share the expert network, it consists of multiple stacked Transformer layers; , and These are general sentiment feature vectors for text, video, and audio, respectively.

[0119] In a preferred embodiment of the present invention, the text-first incremental decoding module initiates the sentiment analysis process primarily based on the text modality. It first makes an initial judgment based on textual information, and then, according to the confidence level, incrementally introduces video and audio information as needed to supplement the analysis, achieving an on-demand decoding strategy consistent with human cognition. Long Short-Term Memory (LSTM) networks are an improved recurrent neural network that controls the flow of information through forget gates, input gates, and output gates, enabling the model to "remember" important information and "forget" unnecessary information. The text-first incremental decoding module is built on an LSTM network, gradually fusing information from the text, video, and audio modalities through LSTM cells with different parameters.

[0120] Specifically, given a general sentiment feature vector of the input text and private sentiment feature vector The final sentiment feature vector of a single text is calculated using the LSTM unit corresponding to the text modality. As shown below:

[0121]

[0122] in, This is the first LSTM unit used for fusing text information. This refers to the valid text information after input gate filtering. Let be the learnable initial state vector. This is a vector concatenation operation. This results in the final sentiment feature vector for a single text. Subsequently, based on the vector... Predicting the distribution of sentiment categories And calculate the confidence level of the prediction results. As shown below:

[0123]

[0124] in, For classification networks that utilize only textual information, multi-layer feedforward neural networks with softmax output layers are typically used. The entropy is a function used to calculate the probability distribution. To predict the distribution of sentiment categories, The confidence level of the prediction result is given. This invention uses entropy to evaluate the confidence level of the model's prediction results; the smaller the entropy, the higher the confidence level, and the larger the entropy, the lower the confidence level.

[0125] Given a confidence threshold And compare the confidence threshold with the confidence of the first sentiment category; when This indicates that the sentiment category predicted solely based on text has high reliability, and there is no need to introduce information from other modalities. At this point, [the following is a continuation of the previous sentence, which is incomplete and requires further context]. As the final prediction result. Conversely, This indicates low reliability, requiring the extraction of more sentiment information from the input video for more accurate predictions. A general sentiment feature vector is needed given the input video. and private sentiment feature vector The final sentiment feature vector, which fuses text and video information, is computed using another LSTM unit. Then predict the distribution of sentiment categories. And calculate its confidence level. As shown below:

[0126]

[0127] in, This is a second LSTM unit used to fuse text and video information. The final vector of sentiment features, which integrates text and video information. The valid video information after input gate filtering. To utilize classification networks that combine text and video information, This is a general sentiment feature vector for videos. This is the private sentiment feature vector of the video. As the second emotion category, For the confidence level of the second sentiment category. Similarly, when This indicates that the sentiment category prediction based on text and video fusion is highly reliable, and there is no need to introduce information from the audio modality. At this point, [the following text is incomplete and requires further context: "to put..."] As the final prediction result. Conversely, This indicates low reliability, requiring the extraction of more emotional information from the input audio for more accurate predictions. A general emotional feature vector is needed given the input audio. and private sentiment feature vector The final sentiment feature vector, which fuses information from the three modalities, is calculated using another LSTM unit. Then predict the distribution of sentiment categories. As shown below:

[0128]

[0129] in, The final sentiment feature vector is obtained by fusing information from the three modalities. This is the third LSTM unit used to fuse information from three modes. This refers to the valid audio information after being filtered by the input gate.

[0130] Given an input instance The final prediction result of the text-first incremental decoding module This can be uniformly represented as follows:

[0131]

[0132] in, A threshold specified by the user. In the implementation of the invention, a smaller... A higher value generally leads to better sentiment classification performance, while a higher value... This value can improve inference efficiency. It should be noted that this only applies when selecting the fused video modality ( Only then can the general sentiment feature vector of the video be calculated. and private sentiment feature vector Correspondingly, only when audio modalities are introduced ( Only then can the general emotional feature vector of the audio be calculated. and private sentiment feature vector Experimental results in common scenarios show that approximately 50% of input instances can obtain reliable sentiment predictions using only text, thus saving significant inference time.

[0133] To ensure better results, this invention also introduces a shared-private expert joint training component and a global training component enhanced by knowledge distillation.

[0134] In the shared-private expert joint training phase: a multi-task learning method is used to achieve collaborative training between shared experts and private experts, which not only preserves the differentiated characteristics between each modality, but also promotes their effective alignment in the shared semantic space, thereby improving the effect and robustness of multimodal fusion.

[0135] In the global training phase enhanced by knowledge distillation: a knowledge distillation mechanism is introduced, which uses a full-modal teacher model to perform knowledge transfer and global compensation on a partial-modal student model, enabling the lightweight text-dominated model to also acquire multimodal semantic capabilities, thereby maintaining high sentiment analysis performance while maintaining efficient inference speed.

[0136] In a preferred embodiment of the present invention, only the parameters in the encoding module based on the shared-private experts are optimized during the shared-private expert joint training phase. A multi-task learning method is used to achieve collaborative training between shared and private experts, preserving the differentiated characteristics between each modality while promoting their effective alignment within the shared semantic space. Furthermore, to avoid information redundancy between general sentiment features and modality-specific private sentiment features, the present invention introduces an orthogonal regularization term into the cost function.

[0137] Specifically, three classification networks are first constructed to assist training. These networks take the general and private sentiment features of each modality as input and calculate the prediction results for the sentiment category, as shown below:

[0138]

[0139] in, , and These are classification networks used to assist training, corresponding to text, video, and audio, respectively. , and The sentiment category prediction results are presented for text-only, video-only, and audio-only scenarios, respectively. Then, based on the prediction results and the true category of the input instance, the multi-learning cost for the first task is defined. as follows:

[0140]

[0141] in, The training set is manually labeled. For input examples The true sentiment category, CE is a function that calculates the cross-entropy of two discrete probability distributions. This represents the cost of the first multi-task learning. Finally, we define the orthogonal regularization term. as follows:

[0142]

[0143] in, This is the transpose of the vector. The dot product of vectors, This is an orthogonal regularization term.

[0144] Given the cost of multi-task learning and orthogonal regularization term Define the first total cost of the shared-private expert joint training phase. as follows:

[0145]

[0146] in, For the weights corresponding to the orthogonal regularization term, This represents the initial total cost. During this joint training phase, the cost is continuously minimized. Until convergence, the optimization of parameters in the coding module based on shared-private experts is achieved.

[0147] In a preferred embodiment of this invention, a knowledge distillation mechanism is introduced during the global training phase of knowledge distillation enhancement. This effectively transfers knowledge from the full-modal teacher model to the partial-modal student model, enabling the lightweight text-dominated model to acquire multimodal semantic capabilities. This maintains high sentiment analysis performance while preserving efficient inference speed. The text-first incremental decoding module initiates sentiment analysis primarily with text and gradually introduces video and audio information as needed, achieving incremental supplementation. To ensure that the gradually introduced modal information provides beneficial supplementation to existing information rather than redundancy, this invention also adds a mutual information constraint term during this training phase.

[0148] Specifically, firstly, based on the prediction results of each step of the text-first incremental decoding module and the true category of the input instance, the second multi-task learning cost for this stage is defined. As shown below:

[0149]

[0150] in, The training set is manually labeled. For input examples The true sentiment category, CE is a function that calculates the cross-entropy of two discrete probability distributions. The cost for the second multi-task learning is then defined as follows: The full-modal prediction results (third sentiment category) are used as the teacher, and the partial-modal prediction results (first and second sentiment categories) are used as the students.

[0151]

[0152] in, The cost of knowledge distillation To calculate the KL divergence between the two probability distributions, and finally, given the information output by each LSTM unit in the text-first incremental decoding module, a mutual information constraint term is defined. as follows:

[0153]

[0154] in, This is a function for calculating the cosine similarity between two vectors. This is a multi-layer feedforward neural network. Cosine similarity is mainly used to capture redundancy in the linear direction, but its effect on suppressing nonlinear dependencies is limited. Therefore, this invention first performs a nonlinear transformation on the feature vectors, and then minimizes the transformed cosine similarity, thereby approximately achieving mutual information constraints and reducing redundant information between incremental modal information and existing modal information.

[0155] According to the second multi-task learning cost The cost of knowledge distillation Mutual information constraints Define the total cost of the global training phase. As follows:

[0156]

[0157] in, and These represent the weights of the knowledge distillation cost and the mutual information constraint term, respectively.

[0158] It should be noted that during the global training phase of knowledge distillation enhancement, in order to fully utilize all modal information, the text-first incremental decoding module processes the text, video, and audio modalities sequentially for any input instance, without relying on the confidence score of each prediction step to determine whether to terminate prematurely. This is achieved by continuously minimizing the second total cost. The process continues until convergence, optimizing all parameters and training a text-first multimodal sentiment analysis model.

[0159] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A text-first multimodal sentiment analysis method, characterized in that, The method includes the following steps: Step 1: Given an instance, extract the preliminary semantic representation of the instance's text, video, and audio. Step 2: Construct private expert networks for text, video, and audio modalities respectively, and generate private sentiment feature vectors for the corresponding modalities in the preliminary semantic representation of text, video, and audio through the private expert networks; A shared expert network is used to generate general sentiment feature vectors for corresponding modalities in the preliminary semantic representations of text, video, and audio. Step 3: Fuse the private sentiment feature vector with the text modality part of the general sentiment feature vector to generate the final sentiment feature vector of a single text. Step 4: Initiate the sentiment analysis process with text modality as the primary driver, use the final sentiment feature vector of a single text to perform sentiment classification prediction, obtain the sentiment category, and calculate the confidence score of the sentiment category; Step 5: Based on the confidence level of the sentiment category, incrementally supplement the video and audio with private sentiment feature vectors and general sentiment feature vectors as needed, and then perform sentiment classification prediction to achieve an on-demand decoding strategy consistent with human cognition, and obtain the final sentiment category. This includes the following steps: The sentiment category predicted based on the text is designated as the first sentiment category; Given a confidence threshold, compare the confidence threshold with the confidence of the first sentiment category; If the confidence level of the first sentiment category is less than or equal to the confidence threshold, the first sentiment category is deemed to have high credibility, and there is no need to introduce information from other modalities. The first sentiment category is then used as the final sentiment category. If the confidence score of the first sentiment category is greater than the confidence threshold, then the confidence score of the first sentiment category is low, and more sentiment information needs to be extracted from the input video for more accurate prediction. The general sentiment feature vector and the private sentiment feature vector of the video are used to calculate the final sentiment feature vector that fuses the text and video information through the second LSTM unit, and then sentiment classification prediction is performed to obtain the second sentiment category. The confidence score of the second sentiment category is then calculated, and the following relationship exists: ； in, The final sentiment feature vector for a single text. The entropy is a function used to calculate the probability distribution. This is a second LSTM unit used to fuse text and video information. The final vector of sentiment features, which integrates text and video information. The valid video information after input gate filtering. To utilize classification networks that combine text and video information, This is a general sentiment feature vector for videos. This is the private sentiment feature vector of the video. As the second emotion category, The confidence level for the second sentiment category; Then, the confidence level of the second sentiment category is compared with the confidence threshold again. If the confidence level of the second sentiment category is less than or equal to the confidence threshold, the second sentiment category is determined to be highly credible, and there is no need to introduce information from other modalities. The second sentiment category is then used as the final sentiment category. If the confidence score of the second sentiment category is greater than the confidence threshold, the second sentiment category is considered to have low confidence, and more sentiment information needs to be extracted from the input audio for more accurate prediction. The general sentiment feature vector and the private sentiment feature vector of the audio are used to calculate the final sentiment feature vector of the three modalities through the third LSTM unit, and then sentiment classification prediction is performed to obtain the third sentiment category, which is taken as the final sentiment category. The following relationship exists: ； in, The final sentiment feature vector is obtained by fusing information from the three modalities. This is the third LSTM unit used to fuse information from three modes. This refers to the valid audio information after input gate filtering. This is a general emotional feature vector for audio. This is a private emotional feature vector for the audio. The training process for private and shared expert networks specifically includes the following steps: Three classification networks are constructed to assist training. Each network takes the general and private sentiment features of its respective modality as input and calculates the prediction results for the sentiment category. The corresponding process follows the following relationship: ； in, , and These are classification networks used to assist training, corresponding to text, video, and audio, respectively. , and The results show the sentiment category predictions for text only, video only, and audio only, respectively. This is a private sentiment feature vector of the text. This is a general sentiment feature vector for the text. This is the private sentiment feature vector of the video. This is a general sentiment feature vector for videos; Based on the predicted sentiment category and the true sentiment category, the first multi-task learning cost is defined, and the corresponding process has the following relationship: ； in, The training set is manually labeled. For input examples The true sentiment category, CE is a function that calculates the cross-entropy of two discrete probability distributions. The cost of first multi-task learning; Based on the general and private sentiment features of each modality, an orthogonal regularization term is defined, and the corresponding process has the following relationship: ； in, This is the transpose of the vector. The dot product of vectors, This is an orthogonal regularization term; Based on the first multi-task learning cost and orthogonal regularization term Define the first total cost of the shared-private expert joint training phase, and the corresponding process has the following relationship: ； in, For the weights corresponding to the orthogonal regularization term, As the first total cost; By minimizing the first total cost Until convergence, the parameters in the private expert network and the shared expert network are optimized to achieve joint training of the shared-private experts.

2. The text-first multimodal sentiment analysis method according to claim 1, characterized in that, In step 1, preliminary semantic representations of the text, video, and audio of the instance are extracted, and the following relationships exist: ； in, , and These are text encoders, video encoders, and audio encoders, respectively. , and The semantic matrix representations are for the input text, video, and audio, respectively. They are text, video, and audio, respectively. , where x represents a given instance.

3. The text-first multimodal sentiment analysis method according to claim 2, characterized in that, In step 2, private sentiment feature vectors for the corresponding modalities in the preliminary semantic representations of text, video, and audio are generated through a private expert network, and the following relationship exists: ； in, , and These are private expert networks corresponding to text, video, and audio, respectively, each consisting of multiple stacked Transformer layers.

4. The text-first multimodal sentiment analysis method according to claim 3, characterized in that, In step 2, a shared expert network is used to generate general sentiment feature vectors for the corresponding modalities in the preliminary semantic representations of text, video, and audio, which correspond to the following relationship: ； in, To share the expert network, it consists of multiple stacked Transformer layers.

5. The text-first multimodal sentiment analysis method according to claim 4, characterized in that, In step 3, the private sentiment feature vector and the text modality portion of the general sentiment feature vector are fused to generate the final sentiment feature vector of a single text, with the following relationship: ； in, This is the first LSTM unit used for fusing text information. This refers to the valid text information after input gate filtering. Let be the learnable initial state vector. This is a vector concatenation operation.

6. The text-first multimodal sentiment analysis method according to claim 5, characterized in that, In step 4, the sentiment analysis process is initiated primarily based on the text modality. Sentiment classification prediction is performed using the final sentiment feature vector of a single text to obtain the sentiment category, and the confidence level of the sentiment category is calculated. The following relationship exists: ； in, For classification networks that utilize only textual information, To predict the distribution of sentiment categories, This represents the confidence level of the prediction result.

7. The text-first multimodal sentiment analysis method according to claim 6, characterized in that, Steps 1 to 5 above are implemented using a text-first multimodal sentiment analysis model. The training method for the text-first multimodal sentiment analysis model includes the following steps: Given a training set, which includes several given instances and their corresponding true classes; Using the training set as input, execute steps 1 to 5 above. During the execution, disable the confidence calculation and comparison of sentiment categories, force full-path forward propagation, and obtain the first sentiment category, the second sentiment category, and the third sentiment category. Based on the given true categories and the first, second, and third sentiment categories, the second multi-task learning cost is defined, and the corresponding process has the following relationship: ； in, The training set is manually labeled. For input examples The true sentiment category, CE is a function that calculates the cross-entropy of two discrete probability distributions. The cost of the second multitasking learning; Taking the third emotion category as the teacher and the first and second emotion categories as the students, the cost of knowledge distillation is defined, and the corresponding process has the following relationship: ； in, The cost of knowledge distillation To calculate the KL divergence of two probability distributions; Using the information output by each LSTM unit, mutual information constraint terms are defined, and the corresponding process has the following relationship: ； in, This is a function for calculating the cosine similarity between two vectors. It is a multi-layer feedforward neural network; According to the second multi-task learning cost The cost of knowledge distillation Mutual information constraints Define the second total cost during the global training phase. The corresponding process has the following relationship: ； in, and These are the weights of the knowledge distillation cost and the mutual information constraint term, respectively. This is the second total cost; By minimizing the second total cost The process continues until convergence, optimizing the parameters in the text-first multimodal sentiment analysis model to train the model.

8. A text-first multimodal sentiment analysis system, characterized in that, The system applies the text-first multimodal sentiment analysis method as described in any one of claims 1 to 7, and the system comprises: A shared-private expert-based coding module is used for: Given an instance, extract preliminary semantic representations of the instance's text, video, and audio. Private expert networks are constructed for text, video, and audio modalities respectively. Private sentiment feature vectors for the corresponding modalities are generated through the private expert networks in the preliminary semantic representation of text, video, and audio. A shared expert network is used to generate general sentiment feature vectors for corresponding modalities in the preliminary semantic representations of text, video, and audio. The text-first incremental decoding module is used for: The private sentiment feature vector is fused with the text modality part of the general sentiment feature vector to generate the final sentiment feature vector of a single text. The sentiment analysis process is initiated primarily based on the text modality. The final sentiment feature vector of a single text is used to predict the sentiment category, and the confidence level of the sentiment category is calculated. Based on the confidence level of the sentiment category, private sentiment feature vectors corresponding to videos and audio, as well as general sentiment feature vectors, are incrementally introduced as needed for supplementation. Then, sentiment classification prediction is performed to achieve an on-demand decoding strategy consistent with human cognition and obtain the final sentiment category.

Images