Multi-modal sentiment analysis method and system based on modal quality perception and adaptive fusion

Abstract

This invention relates to the field of multimodal sentiment analysis technology, and in particular to a method and system for multimodal sentiment analysis based on modality quality perception and adaptive fusion. The method includes acquiring multimodal sentiment data; constructing a multimodal sentiment analysis model based on quality perception and adaptive fusion, including modality-specific feature projection, unimodal Transformer context encoding, modality semantic-based quality assessment, and modality reliability-based adaptive fusion; training the constructed multimodal sentiment analysis model; and using the trained multimodal sentiment analysis model to perform sentiment classification prediction. This invention can achieve reliability-perceived adaptive fusion even under dynamic fluctuations in the contributions of different modalities, improving the accuracy and flexibility of utilizing multi-source sentiment information.

Description

Technical Field

[0001] This invention relates to the field of multimodal sentiment analysis technology, and in particular to a multimodal sentiment analysis method and system based on modal quality perception and adaptive fusion. Background Technology

[0002] Traditional sentiment analysis primarily focuses on the text modality. However, with the development of mobile internet and social media, user-generated content is increasingly exhibiting multimodal characteristics, interwoven with text, speech, and visual elements. Compared to single-modality data, multimodal data can utilize complementary information between different modalities. For example, text provides semantic content, speech reflects emotional intensity and tone changes, and visuals present facial expressions and scene cues. Therefore, Multimodal Sentiment Analysis (MSA) has gradually become an important research direction in the field of sentiment computing and has been widely applied in scenarios such as opinion mining and human-computer interaction.

[0003] In recent years, deep learning methods have significantly improved the discriminative power of multimodal sentiment analysis through cross-modal feature interaction, self-attention mechanisms, and fusion representation learning. However, most existing methods are based on the assumption of modal integrity and stable quality. In real-world scenarios, modality loss or quality degradation frequently occurs due to factors such as background noise, visual occlusion, and text transcription errors. In such cases, using a unified fusion strategy not only fails to fully utilize effective modal information but may also introduce low-quality modal noise, leading to distorted fusion representations. Therefore, how to achieve robust multimodal sentiment analysis under conditions of modality loss and quality fluctuations has become a key issue in current research. Although existing studies have proposed various solutions for modeling missing modalities, overall, the following shortcomings remain:

[0004] (1) Existing research is mostly limited to focusing on the state of modality absence. It ignores the problem of modality quality instability in real-world scenarios (expressive degradation or insufficient emotional information). Therefore, how to accurately perceive the true usability of modalities in complex scenarios remains a very challenging problem.

[0005] (2) Existing MSA fusion methods mainly adopt static weighting or direct splicing strategies. When there are significant differences in modal quality, the model cannot fully utilize the role of high-quality modes. Therefore, how to adjust the fusion process according to the differences in modal quality is a new challenge. Summary of the Invention

[0006] To address the issues of modality loss, modality quality imbalance, and difficulty in adaptively adjusting fusion weights in multimodal emotion recognition under missing modality scenarios, this invention provides a multimodal emotion analysis method and system based on modality quality perception and adaptive fusion.

[0007] Firstly, the multimodal sentiment analysis method based on modal quality perception and adaptive fusion provided by this invention adopts the following technical solution:

[0008] Multimodal sentiment analysis methods based on modal quality perception and adaptive fusion include:

[0009] Acquire multimodal sentiment data;

[0010] Data preprocessing is performed based on the acquired multimodal sentiment data;

[0011] Construct a multimodal sentiment analysis model based on quality perception and adaptive fusion, including modality-specific feature projection, single-modal Transformer context encoding, modality semantic-based quality assessment, and modality reliability-based adaptive fusion;

[0012] The constructed multimodal sentiment analysis model is trained.

[0013] Sentiment classification and prediction are performed using a trained multimodal sentiment analysis model.

[0014] Secondly, a multimodal sentiment analysis system based on modal quality perception and adaptive fusion includes:

[0015] The data acquisition module is configured to acquire multimodal sentiment data;

[0016] The preprocessing module is configured to perform data preprocessing based on the acquired multimodal sentiment data;

[0017] The adaptive fusion module is configured to build a multimodal sentiment analysis model based on quality perception and adaptive fusion, including modality-specific feature projection, single-modal Transformer context encoding, modality semantic-based quality assessment, and modality reliability-based adaptive fusion.

[0018] The training module is configured to train the constructed multimodal sentiment analysis model.

[0019] The prediction module is configured to perform sentiment classification prediction using a trained multimodal sentiment analysis model.

[0020] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the multimodal sentiment analysis method based on modal quality perception and adaptive fusion.

[0021] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to perform the multimodal sentiment analysis method based on modal quality perception and adaptive fusion.

[0022] In summary, the present invention has the following beneficial technical effects:

[0023] This invention enables reliable modality quality perception under conditions of modality absence, noise interference, and quality fluctuations, reducing the interference of low-quality or missing modalities on sentiment recognition results. The invention extracts the contextual representation of each modality through a modality-specific encoder, uses a lightweight quality estimator to score the quality of the global representation, and constructs a reliability vector by combining the missing states. This mechanism accurately characterizes the differences in the effectiveness and reliability of each modality's representation, avoiding the indiscriminate use of low-quality or missing modalities during the fusion stage, and providing fundamental support for improving the robustness of multimodal sentiment analysis in scenarios with missing modalities.

[0024] This invention enables reliability-aware adaptive fusion under dynamic fluctuations in contributions from different modalities, improving the accuracy and flexibility of multi-source emotional information utilization. Based on quality scores, missing states, and reliability representation, this invention constructs an adaptive fusion mechanism and dynamically allocates fusion weights. Compared to simple concatenation, average fusion, or fixed-weighted fusion, this mechanism enhances the contribution of highly reliable modalities in the fusion representation while suppressing the negative impact of noise and missing modalities on the fusion result. It is a key support for improving the expressive power of multimodal fusion and the stability of emotion recognition. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a multimodal data missing scenario according to Embodiment 1 of the present invention.

[0026] Figure 2 This is a schematic diagram of the overall multimodal sentiment analysis model based on modal quality perception and adaptive fusion in Embodiment 1 of the present invention.

[0027] Figure 3 This is a schematic diagram of the modal quality assessment module structure in Embodiment 1 of the present invention.

[0028] Figure 4 This is a schematic diagram of the adaptive fusion module structure of Embodiment 1 of the present invention.

[0029] Figure 5 This is a schematic diagram showing the comparison results of M-F1 and ACC for different modal missing rates in a four-class classification task according to Embodiment 1 of the present invention.

[0030] Figure 6This is a schematic diagram showing the comparison results of M-F1 and ACC for different modal missing rates in a seven-class classification task according to Embodiment 1 of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the accompanying drawings.

[0032] Example 1

[0033] Reference Figure 1 The multimodal sentiment analysis method based on modal quality perception and adaptive fusion in this embodiment includes:

[0034] S1. Acquire multimodal sentiment data;

[0035] In this embodiment, the multimodal input is set as follows: ,in, , and These represent text, audio, and visual modalities, respectively. Unlike traditional full multimodal sentiment analysis, this paper considers an incomplete input setting that more closely resembles real-world scenarios. For any given sample, all three modalities are not always available; therefore, a modality existence indicator vector is introduced. .in, Indicates the first The existence state of each modality This indicates that the mode exists. This indicates that the mode is missing, and Based on this, the model needs to adaptively learn a stable fusion representation according to the availability and quality status of each modality in the current sample, and output the final sentiment prediction result. .

[0036] S2. Perform data preprocessing based on the acquired data.

[0037] Both datasets used in this paper contain information from three modalities: text, speech, and vision. To ensure consistency with existing multimodal sentiment analysis research, this paper adopts a feature extraction workflow similar to existing work, preprocessing each of the three modalities separately. Considering the differences in signal form and semantic carrying methods among the different modalities, this paper uses the pre-trained language model BERT, the Librosa toolkit, and the OpenFace 2.0 toolkit to construct text semantic representation, acoustic temporal representation, and visual dynamic representation, respectively.

[0038] Text Modality Feature Extraction: For text modality, this paper uses a pre-trained BERT model to perform contextual semantic encoding on text segments. Let the input text sequence be... in, Represents the original text sequence. Indicates the first Each word element, This indicates the length of the text sequence. After processing by the tokenizer, the text sequence is converted into a sequence of token indices: in, This represents the sequence of word indexes obtained by the word segmenter. This indicates the text segmentation and index mapping operation corresponding to BERT. This represents the input text sequence. Subsequently, the word index sequence is input into the BERT model to obtain a contextualized text representation: in, Contextualized semantic features representing text modalities This indicates the pre-trained BERT encoder. This represents the sequence of token indices input to BERT. Let the first term be... The BERT output corresponding to each word is Then the text modal sequence is represented as in, Indicates the first Contextualized text features corresponding to each lexical unit This represents the length of the text sequence. Since BERT's hidden layer dimension is 768, therefore... in, Represents a 768-dimensional text feature space. The text modality is represented by This representation consists of 768 semantic features per lexical position. It preserves the contextual semantic information within the text sequence, providing a semantic basis for subsequent sentiment recognition.

[0039] Audio Modal Feature Extraction: To reduce the differences in audio format, sampling rate, and channel settings across different datasets, this paper first uses the Librosa toolkit to perform uniform preprocessing on the original audio. Let the original audio signal be... in, Represents the original audio signal. Indicates the first One original audio sampling point, This indicates the number of original audio sample points. This article converts the original audio to a uniform 16 kHz mono signal: .in, This represents the resampled audio signal. This indicates a resampling operation. This represents the original audio signal, and 16000 indicates a target sampling rate of 16kHz. The resampled audio signal can be represented as... in, Indicates the number of resampling operations. One audio sampling point, This indicates the number of sampling points after resampling. Subsequently, the audio signal is divided into frames of 512 sampling points each. Each audio frame can be represented as in, Indicates the first One audio frame, This indicates the number of sampling points contained in each frame, which is set to 512 in this paper; and They represent the first The starting and ending sampling points of each frame are defined. For each audio frame, acoustic features such as zero-crossing rate, Mel-frequency cepstral coefficients, and constant Q-transform are extracted. The acoustic features of the frame are spliced ​​together as follows in, Indicates the first Acoustic characteristics of frame audio; Indicates feature concatenation operation; Indicates from the first Zero-crossing rate features extracted from frame audio; Indicates from the first Mel frequency cepstral coefficient features extracted from frame audio; Indicates from the first Constant Q-transform features extracted from frame audio. After concatenation, 33-dimensional frame-level audio features are obtained: in, This represents a 33-dimensional real-valued feature space, meaning that each frame of audio input consists of 33 acoustic feature values. Therefore, the audio modality is ultimately represented as... in, This represents the final input features of the audio modality. Indicates the first 33-dimensional audio features of the frame, This indicates the length of the audio sequence after frame division. Indicates audio modality by Each time step consists of 33 features. These features can reflect emotional information in the audio modality from aspects such as short-time spectral structure, zero-crossing changes, and speech prosody.

[0040] Visual modality feature extraction: For visual modality, this paper uses the OpenFace 2.0 toolkit to extract facial-related features from video frames. Let the original video sequence be represented as... in, Represents the original video sequence. Indicates the first Frame image, This represents the sequence length of the visual modality, i.e., the number of video frames. After performing facial analysis on each frame, OpenFace 2.0 can extract visual features such as facial keypoints, head pose, and eye movements to characterize dynamic facial changes during emotional expression. During the original feature extraction process, OpenFace also generates auxiliary information such as frame number, timestamp, and facial detection confidence score. Let the... The original visual features of the frame are represented as in, Indicates the first The frame contains raw visual features extracted by OpenFace; Indicates feature concatenation operation; Indicates the frame number; Represents a timestamp; Indicates the confidence level of facial detection; The core facial features related to emotional expression include facial landmarks, head pose, and eye movements. Since auxiliary fields such as frame number, timestamp, and confidence score are mainly used for recording and describing the detection process and do not directly participate in emotional semantic modeling, this paper removes them when constructing the visual modality input, retaining only the core visual features: in, Indicates the first The frames ultimately serve as the visual features used as model input. After field filtering and feature concatenation, each frame's visual features are represented as a 709-dimensional vector. Therefore, the final input of the visual modality can be represented as in, The final input features representing the visual modality. 709 represents the length of the visual sequence and the dimension of the visual features in each frame. This representation can describe the visual dynamics of emotional expression from aspects such as facial key points, head posture, and eye movements.

[0041] Feature extraction results: After the above preprocessing and feature extraction, the three modalities are finally represented as follows: in, , and These represent the final input features for text, audio, and visual modalities, respectively. , and 768, 33, and 709 represent the sequence lengths of the text, audio, and visual modalities, respectively; 768, 33, and 709 represent the feature dimensions of the text, audio, and visual modalities, respectively. These three modal features will serve as inputs to subsequent single-modal representation learning, modality quality assessment, and adaptive fusion modules.

[0042] S4. Model Building

[0043] This embodiment presents a multimodal sentiment analysis model based on quality perception and adaptive fusion. The model mainly consists of four parts: single-modal representation learning, quality estimation, adaptive fusion, and classification prediction. Specifically, the model first maps the three modalities to a unified feature space through a modality-specific projection layer, and then extracts contextualized sequence representations using a single-modal Transformer encoder. Subsequently, temporal average pooling is performed on the encoded features to obtain a modality-level global representation, and a lightweight quality estimator is used to generate quality scores for each modality. To simultaneously characterize modality missing states and modality quality, this paper constructs missing vectors and quality vectors, and obtains a reliability vector through element-wise multiplication, which is used to measure the actual credibility of each modality in the current sample. In the fusion stage, the model combines the modality global representation, quality information, and reliability information, and learns adaptive fusion weights through a reliability-aware weighted network and a relationship-aware network, thereby dynamically aggregating the three modal information to obtain a unified fused representation. Furthermore, to mitigate the representation bias caused by low-quality or unreliable modalities, this paper introduces a strongest modality residual anchoring mechanism, using the most reliable modality in the current sample as the anchor point to stabilize and correct the fusion result. Finally, the corrected fusion representation is fed into the classification layer after passing through a post-fusion Transformer encoder and attention aggregation module to achieve sentiment category prediction. The remaining modules of QAFMSA are described in detail below.

[0044] 1. Modal-specific feature projection

[0045] To address the inconsistency in the original feature dimensions of text, audio, and visual modalities, this paper introduces a modality-specific linear projection layer to map each of the three modalities to a unified latent space. Let the modal input be: ,in, , Representing modes The sequence length, This represents the original feature dimension of the modality. For any modality... Its modality-specific projection process is defined as follows: ,in, For modality Learnable projection matrix, Represents the unified latent space dimension. This represents the modal sequence representation after projection. For the three modalities of text, audio, and vision, it can be written as:

[0046] ,

[0047] ,

[0048] ,

[0049] in: Combining the model input dimensions, we have: ,therefore: The three modal representations after projection satisfy: Furthermore, since the projected input has already undergone modal masking, we have: Right now:

[0050] ,

[0051] Therefore, when a modality exists, the projection layer maps that modality to a unified latent space; when a modality is missing, its projected representation remains a zero matrix, thus avoiding interference from the missing modality with subsequent context encoding. Ultimately, the projections of the three modalities can be uniformly represented as: This module aligns the dimensions of different modes using modality-specific projection parameters, while preserving the distribution characteristics of each mode, thus providing a unified dimensional input for the subsequent modality-specific Transformer encoder.

[0052] 2. Single-modal Transformer context encoding

[0053] After completing feature projection, this paper employs a modality-specific Transformer encoder to perform context modeling for each modality sequence. Let the first... The input to the layer Transformer is Its multi-head self-attention sublayer and feedforward network sublayer can be represented as: in, This indicates a multi-head self-attention mechanism. This represents a location-based feedforward network. This represents the layer normalization operation. Residual connections are used to preserve input features and alleviate gradient decay during deep network training, while layer normalization is used to stabilize the hidden state distribution, thereby improving the stability of the model training process. Specifically, for the ... Each attention head first represents the input. These are mapped to the query matrix, key matrix, and value matrix, respectively: in, , , The first The learnable mapping matrix corresponding to each attention head , and These represent the query, key, and value representations, respectively. Based on the scaled dot product attention mechanism, the... The output of each attention head is calculated as follows: in, The feature dimension of a single attention head. This is a scaling factor used to mitigate the problem of excessively large values ​​for the dot product result as the dimension increases. This mechanism adaptively allocates attention weights based on the correlation between different time steps in the sequence, thereby capturing long-distance dependencies within a modality.

[0054] Then, the outputs of all attention heads are concatenated, and the final result of multi-head self-attention is obtained through the output mapping matrix: in, Indicates the number of attention heads. This represents the output mapping matrix. The multi-head structure enables parallel modeling of sequence dependencies from different subspaces, thereby enhancing the ability of single-modal representations to characterize multi-granular temporal information. After self-attention computation, the position-fedforward network further performs a nonlinear transformation on the hidden state at each time step, defined as... in, and The learnable weight matrix in the feedforward network. and For bias terms, This represents a non-linear activation function. The feedforward network operates independently at each time step to enhance the non-linear expressiveness of the hidden representation and further improve the discriminative power of single-modal features. After multiple layers of Transformer encoding, the text, audio, and visual modalities obtain their corresponding contextualized sequence representations: in, Representing modes The sequence-level context representation learned by the encoder. This representation integrates local features within the modality with temporal dependency information, providing an effective single-modal semantic foundation for subsequent quality assessment, adaptive fusion, and sentiment classification prediction.

[0055] 3. Quality Assessment Module

[0056] In missing modality multimodal sentiment analysis, the presence of a modality does not equate to its reliability. Even if a modality is not missing, its quality may be low due to noise, lighting conditions, recognition errors, or weakened sentiment cues. Therefore, this paper designs a quality assessment module based on unimodal encoding to evaluate the content quality of each modality and generate sample-level modality quality scores. For any modality... Let the contextualized sequence output by the single-mode encoder be represented as follows: in, Representing modes In the The hidden state at each time step. This indicates the sequence length of the mode. This represents a unified latent space dimension. To obtain a modality-level global semantic representation, this paper performs temporal average pooling on the sequence representation: This process can also be written in matrix form: This allows us to obtain the global representations for the three modes: After obtaining the modality-level global representation, this paper uses a two-layer perceptron to... Perform quality mapping. First, [the following steps are taken]. The first nonlinear transformation is input to obtain the quality implicit representation. in, This is the first-level mapping matrix. For bias vectors, Represents the dimension of the latent space of mass. Represents a non-linear activation function. This represents the implicit representation of modal quality. Subsequently, the implicit quality representation is further mapped to a scalar quality score. in, This is the second-level mapping matrix. For bias terms, This represents the quality score before normalization. Combining the above two steps, we get: Substituting further into the average pooling representation, the quality score can be written in the form of the sequence hidden state: To constrain the quality scores to a stable range, this paper first uses the Sigmoid function for normalization: in, This represents the initial quality probability score. Considering that excessively low quality scores may lead to numerical instability in subsequent reliability modeling and fusion weight calculations, this paper further introduces a lower bound on the quality score. And apply a truncation constraint to the quality score. in, Indicates the lower bound of the mass fraction. This represents the truncation function. Its equivalent form is: because The above formula can be further expressed as Therefore, the final quality score satisfies For the three modalities of text, audio, and vision, the calculations are as follows: Finally, the mass fraction vectors of the three modes are represented as follows: in, Representing modes Content quality and discriminant value in the current sample. Existence indication with binary modality. In comparison, quality score This quality vector can characterize the effectiveness of information within a modality, thereby further distinguishing between two cases: a modality exists but has low quality, and a modality exists and has high quality. This quality vector will serve as an important basis for subsequent reliability modeling and adaptive fusion weight allocation.

[0057] 4. Adaptive fusion module based on modal reliability

[0058] Modal reliability modeling

[0059] In missing modality multimodal sentiment analysis, the existence of a modality only characterizes its structural usability and cannot reflect its actual discriminative value. This paper models modality reliability based on modality presence states and quality scores. Let the modality presence vector be... The quality vector output by the quality assessment module is in, Representing modes Does it exist? Representing modes Content quality and discriminative value in the current sample. For any modality Its reliability is defined as Furthermore, the sample-level reliability vectors for the three modalities can be expressed as follows: Right now in, This represents element-wise multiplication. Representing modes Overall reliability in the current sample. Due to... ,and Therefore, reliability satisfy Specifically, when the mode When missing, there is When mode When it exists, there is Therefore, reliability can also be written in piecewise form: As shown in the above equation, modal reliability is affected not only by the existence of the mode but also by the quality of its content. Only when the mode exists and its quality is high can reliability be guaranteed. Only then will a larger value be obtained; when a mode is missing, its reliability is directly 0; when a mode exists but its quality is low, the reliability will also decrease accordingly. Considering that a reliability prior needs to be introduced in the subsequent adaptive fusion process, and logarithmic calculation may be involved, to avoid This leads to numerical instability, and this paper further defines smoothing reliability: in, This is a minimal constant. Correspondingly, the smooth reliability vector for the three modes can be expressed as: Will Substituting into the above equation, we can obtain

[0060] Further written in segment form as because ,when When, the above formula can be simplified to Based on this, if reliability needs to be used as a priori in the calculation of fusion weights, smoothed reliability can be used in the calculation: This processing can prevent the occurrence of zero reliability corresponding to the missing mode. This addresses the issue while preserving the constraint of reliability on the allocation of fusion weights. Through the above modeling, the reliability vector... It also integrates modal existence state and quality assessment results, which can more accurately characterize the actual reliability of different modalities in the current sample and provide a reliable prior for subsequent adaptive fusion.

[0061] Adaptive fusion representation

[0062] After completing modal reliability modeling, this paper further utilizes modal semantic representation, quality information, missing states, and reliability information to learn adaptive fusion representations, thereby achieving more robust multimodal information integration. First, a learnable transformation is performed on the contextualized sequence representations obtained from single-modal representation learning. Let the encoding representations of text, audio, and visual modalities be respectively... For any mode ,have in, Representing modes The sequence length, This represents a unified latent space dimension. Considering that although different modalities have been mapped to the same dimension, their semantic distribution and feature scale may still differ, this paper further introduces a modality-level learnable transformation matrix to perform a nonlinear mapping on the encoded sequence representation. in, For modality The corresponding learnable parameter matrix, This represents the hyperbolic tangent activation function. Let the transformed modal sequence be represented as follows. in, Representing modes In the The hidden state after transformation at each time step. Then, for... Average pooling is performed along the time dimension to obtain the modality-level semantic representation used for fusion: Its matrix form can be written as Further substitution , can be obtained This yields the fusion semantic representations for the three modalities: It should be noted that in the quality assessment module Primarily used for estimating modal content quality, while in this section... The representation is obtained by a learnable linear transformation from contextualized encoding and is mainly used for subsequent adaptive fusion. To avoid symbolic confusion, this paper uses... This represents the modal-level semantic representation during the fusion phase. After obtaining the fusion semantic representations of the three modalities, to enable the fusion module to simultaneously perceive modal semantic state, quality level, existence state, and reliability information, this paper constructs a fusion context vector: in, This represents a vector concatenation operation. Represents the mass vector. This indicates that the mode has a vector. Represents a reliability vector. Based on the fusion context vector. This paper uses linear mapping to generate initial preference scores for three modalities: in, and For learnable parameters, Representing modes The initial fusion preference score in the current sample. To further explicitly inject modal reliability into the fusion weight allocation process, this paper introduces a smooth reliability prior before normalization. Let the smooth reliability be... Then mode The fused logits can be represented as in, This represents a reliability prior. When the modal reliability is high, this term can improve the relative advantage of the corresponding mode before normalization; when the mode is missing or the reliability is low, A smaller logarithmic value will decrease the fusion score of that modality, thus suppressing its influence on the final fusion representation. Based on this, a Softmax function with a temperature coefficient is used to calculate the final fusion weights: in, This is a temperature coefficient used to adjust the smoothness of the fusion weight distribution. When When the weights are smaller, the weight distribution is more concentrated, and the model tends to assign more weights to high-reliability modes; when When the value is large, the weight distribution is smoother, and the weight differences between different modes are relatively reduced. This yields the fused weight vector for the three modes: Due to the Softmax normalization constraint, the fusion weights satisfy... Combination The above fusion weights can also be written as Therefore, the final modality fusion weights are determined by two parts: one is the initial preference score learned from the fusion context vector. Secondly, the prior constraints introduced by smooth reliability. Finally, this paper performs a weighted summation of the semantic representations of the three modalities based on the fusion weights to obtain an adaptive fusion representation: Its unified form can be expressed as in, This represents the dynamically fused multimodal representation. It integrates semantic information from text, audio, and visual modalities, and adaptively adjusts the contribution levels of different modalities through quality and reliability constraints, providing more robust fused features for subsequent sentiment classification.

[0063] S4. Model Training

[0064] This paper incorporates classification prediction loss, modality quality assessment loss, and fusion consistency loss into a joint optimization objective to achieve collaborative training of sentiment classification, modality quality estimation, and adaptive fusion processes. For the sentiment classification task, this paper uses labeled smoothing cross-entropy loss for supervision. Let the... The true label of each sample is one-hot encoded as follows: The predicted probability is ,in , Indicates the number of categories. Let be the label smoothing coefficient, then the smoothed label is represented as: The corresponding classification prediction loss is defined as: .in, Let represent the number of samples. This loss is used to constrain the model output to remain consistent with the true sentiment labels and to mitigate the problem of model overconfidence through label smoothing. In the early stages of training, the quality score output by the quality assessment branch may collapse or fluctuate significantly, thus affecting the subsequent reliability modeling and adaptive fusion process. To improve the stability of quality estimation, this paper introduces a lightweight regularization constraint on the modal quality score. Let the _i_ be the _i ... Each sample in modality The mass fraction on is ,in , representing text, audio, and visual modalities, respectively. To suppress excessive bias of quality scores towards 0 or 1 in the early stages of training, a reference center is introduced. and order The modal quality assessment loss is then defined as: ,in, This represents the number of samples. This term serves only as an auxiliary stability constraint, used to smooth the optimization process of the quality assessment branch, thereby improving the stability of subsequent reliability modeling and adaptive fusion. This paper constructs modal reliability jointly by modal existence states and quality scores, and learns adaptive fusion weights based on reliability priors. Considering that the fusion weights should maintain consistency with the modal reliability in the overall trend, this paper further designs a fusion consistency regularization term to impose soft constraints on the weight allocation process. Let the ... Each sample in modality The reliability of the above is To avoid instability caused by excessively small values, we first perform a smoothing process: .in, This is a very small constant. Subsequently, the smoothed reliability is normalized to obtain the sample-level reliability distribution: Let the first... Each sample in modality The adaptive fusion weights learned above are Then the fusion consistency loss is defined as: .in, This represents the number of samples. This loss function is used to constrain the adaptive fusion weights to maintain consistency with the modality reliability distribution, enabling the model to focus more on high-reliability modes during the fusion process and reduce the interference of low-reliability modes on the fused representation. Combining the above three parts, this paper ultimately uses the following joint loss function to optimize the model. .in, and This is a hyperparameter used to balance the importance of modal quality assessment loss and fusion consistency loss in the total loss.

[0065] S5. Model Prediction

[0066] To further refine the fusion representation, firstly... Expand into a sequence of length 1: ,in, This represents the initial input to the post-fusion Transformer encoder. This represents the dimension expansion operation. Let the first... The input to the post-layer blending Transformer is Then its query, key, and value matrices are as follows: , in, , , They represent the first In the post-blending Transformer, the first layer The query, key, and value representation of each attention head. , , This is the corresponding learnable parameter matrix. The output of each attention head is: ,in, Indicates the post-fusion Transformer, the first The output of each attention head, This represents the feature dimension of a single attention head. The output of multi-head attention is: in, Indicates the first Post-layer fusion multi-head attention output, This represents the output mapping matrix. Subsequently, residual connections and layer normalization are performed: ,in, Indicates the first The attention sublayer output is fused after each layer. The feedforward network is computed as follows: ,in, Indicates the first Post-layer fusion feedforward network output, , This represents the feedforward network weight matrix. , This indicates the bias term. The post-layer fusion encoding output is: ,in, Indicates the first The output of the Transformer is then blended after layers. After layer encoding, we get: in, This indicates the number of layers in the post-fusion Transformer encoder. This represents the refined fused sequence representation. Subsequently, a Sigmoid attention aggregation module is used to obtain the global fused representation: in, This represents the global fusion representation after aggregation. To further represent the attention aggregation process, let: in, Indicates the th in the post-fusion sequence The hidden state of each location, This represents the length of the post-fusion sequence. The attention interval is then represented as: in, Indicates the first Attention at each position is represented in the middle. , For learnable weight matrix, Let represent the Sigmoid function. The attention score is expressed as: ,in, Indicates the first Attention score for each position, This represents the attention score mapping matrix. The normalized attention weights are: ,in, Indicates the first Attention weights corresponding to each position, and satisfying Therefore, global fusion is represented as: in, This represents the fused representation after attention aggregation. Finally, Input classification layer. First, obtain the following through pre-classification projection: in, This indicates implicit representation before classification. This represents the projection matrix before classification. The category logits are calculated as follows: ,in, This represents the unnormalized category score. Represents the classification weight matrix. Indicates the category bias term. This represents the number of sentiment categories. The final predicted probability distribution is: Right now:

[0067] ,

[0068] in, Indicates that the sample belongs to the first The predicted probabilities of each sentiment category. The final sentiment category is determined by the category with the highest probability: in, This represents the sentiment category predicted by the model.

[0069] Experimental verification

[0070] To verify the effectiveness of the proposed model in both complete and missing modalities, this paper conducts experiments using two trimodal (text, speech, and vision) sentiment analysis datasets, IEMOCAP

[43] and CMU-MOSI

[42] . IEMOCAP is derived from controlled two-person interaction scenarios, while CMU-MOSI is derived from real online videos. The differences in data sources and interaction forms between the two datasets help to comprehensively evaluate the model's adaptability under different data distributions. A brief introduction to these datasets follows.

[0071] IEMOCAP is a typical multimodal emotion interaction dataset, containing approximately 12 hours of audio and video data, consisting of 5 pairs of conversations involving a total of 10 speakers. This dataset provides three aligned modalities: text, speech, and vision, and includes annotations for discourse-level emotion categories. It is commonly used for multimodal emotion recognition and emotion classification tasks.

[0072] CMU-MOSI is a commonly used video opinion sentiment dataset in the field of multimodal sentiment analysis. It contains 93 YouTube movie review videos, divided into 2,199 discourse segments. Each discourse segment includes three aligned modalities: text, audio, and visual, and is labeled with the following: arrive The continuous sentiment score is used to represent the sentiment tendency from strong negative to strong positive.

[0073] To align with the sentiment classification task, this paper uses accuracy (ACC) and macro-average F1 score (M-F1) as the primary evaluation metrics. ACC measures the overall classification accuracy of the model, while M-F1 comprehensively reflects precision and recall across different classes, making it particularly suitable for scenarios with imbalanced class distributions. Their definitions are as follows:

[0074]

[0075] in, This indicates the number of samples that were correctly predicted. Represents the total number of samples. Indicates accuracy. This indicates the recall rate.

[0076] The model parameters used in this invention are shown in Table 1.

[0077] Table 1 QAFMSA Model Parameter Settings

[0078]

[0079] benchmark model

[0080] This paper selects the following 10 state-of-the-art benchmark models for comparative experiments and verifies the performance of QAFMSA. These models are described below:

[0081] AE: A model based on a nonlinear encoder-decoder structure that learns the latent features of input data by mapping the input to a latent space and reconstructing the original representation. CRA: A method that utilizes a stacked residual autoencoder structure to model the difference between the predicted result and the original input, thereby estimating missing modal information. MCTN: A model that translates between different modalities through forward and backward recurrent propagation mechanisms, enabling the learning of joint representations between two modalities during cross-modal transformation. TransM: An end-to-end model that uses Transformer to fuse source and target modal information and proposes parallel translation. MRAN: A model that mitigates the impact of modality loss through modality reconstruction and cross-modal alignment, aiming to recover missing modal information and enhance semantic consistency between modalities, thereby improving the robustness of multimodal sentiment analysis. MMIN: A network that models sentiment relevance from sentence-level representations and predicts missing modalities based on available modalities. TATE_C: A model that assists in joint representation learning by covering missing cases using label encoding techniques. MTMSA: A model that translates visual and audio modalities into text modalities through modal translation, aiming to improve the quality of feature fusion and thus enhance the model's robustness by leveraging text modalities. TATE_J: This model further assigns differentiated weights to different modalities based on TATE_C, thereby more fully exploring the contributions of each modality. SMCMSA: A model that uses a pre-built pool of similar samples and cosine similarity to filter similar modalities for missing data completion.

[0082] Performance comparison experiment

[0083] On the CMU-MOSI dataset, we conducted three-class classification experiments; on the IEMOCAP dataset, we conducted two-class classification experiments. To comprehensively evaluate the effectiveness of the proposed model in scenarios with missing modalities, we selected 10 benchmark models as comparison methods and conducted experiments under both single-modal and multi-modal missing modalities. In the experiments, the modality missing rate was set to 0.0–0.5, and the evaluation metrics were M-F1 and ACC. The results are shown in Tables 2 and 3, respectively.

[0084] Analysis of Single-Modal Missing Data: As shown in Table 2, QAFMSA achieves superior overall results on both datasets under single-modal missing data conditions. Although its M-F1 score is slightly lower than MMIN at a missing rate of 0.0 and slightly lower than SMCMSA at a missing rate of 0.2, the differences are small, and its ACC remains optimal or tied for optimal. When the missing rate increases to 0.5, QAFMSA's M-F1 and ACC reach 56.45% and 83.33%, respectively, representing improvements of 1.73% and 3.65% compared to the second-best model. On the IEMOCAP dataset, QAFMSA achieves optimal performance under all single-modal missing rate settings, maintaining an M-F1 score of 80.81% and an ACC of 85.81% at a missing rate of 0.5, indicating good robustness in single-modal missing data scenarios. Analysis of Multimodal Missing Data:

[0085] As shown in Table 3, QAFMSA still exhibits a strong performance advantage under multimodal missing data conditions. Multimodal missing data simultaneously weakens the information integrity of multiple modalities, thus increasing the task difficulty. On the CMU-MOSI dataset, QAFMSA's advantage is more pronounced under high missing rate settings, especially achieving the best M-F1 and ACC at missing rates of 0.3, 0.4, and 0.5. On the IEMOCAP dataset, QAFMSA achieves the best results under all multimodal missing rate settings, even at a missing rate of 0.5, still achieving an M-F1 score of 77.68% and an ACC score of 84.12%, indicating that the model can maintain strong discriminative ability under complex missing data environments.

[0086] Theoretical Analysis: Overall, as the missing data rate increases from 0.0 to 0.5, the performance of all models on both datasets shows a downward trend, indicating that modality loss weakens the discriminative ability of multimodal sentiment analysis models. In contrast, QAFMSA achieves optimal or tied-optimal results under most missing data rate settings, especially under medium-to-high missing data rates and multimodal loss conditions, demonstrating its stronger adaptability to complex missing data environments. In summary, QAFMSA's advantage lies not only in its overall superior performance but also in its ability to maintain strong discriminative ability even with increasing levels of missing data. This suggests that in scenarios with missing modalities, simply considering the existence of modalities is insufficient; further characterizing the quality differences of available modalities and adaptively adjusting the fusion process accordingly is key to improving the robustness of multimodal sentiment analysis.

[0087] Table 2 Experimental results under single-modal missing conditions

[0088]

[0089] Table 3 Experimental results under multimodal missing conditions

[0090]

[0091] Multiclass Validation

[0092] We conducted 4-class and 7-class classification experiments on the IEMOCAP dataset to verify the robustness of QAFMSA in multi-class tasks and under modality missing conditions. The experiments set the missing rate to 0–0.5, and selected TATE_C, TATE_J, MRAN, MTMSA, and SMCMSA as comparison models. The results are shown in Figures 5 and 6. Figure 5 shows that in the 4-class classification task, QAFMSA outperforms all the comparison models overall. In the M-F1 metric, when the missing rate is 0.4 and 0.5, QAFMSA improves performance by 2.93% and 2.39% compared to the second-best model, SMCMSA, respectively; in the ACC metric, when the missing rate is 0.2 and 0.3, it improves performance by 2.49% and 2.21%, respectively. This indicates that QAFMSA has good performance retention in 4-class missing scenarios. Figure 6 shows that QAFMSA still demonstrates a strong advantage in the more challenging 7-class classification task. On the M-F1 metric, QAFMSA outperformed SMCMSA by 1.85% and 1.02% at missing rates of 0.2 and 0.5, respectively. On the ACC metric, while slightly lower than SMCMSA by 0.02% at a missing rate of 0%, it outperformed SMCMSA by 2.39% and 1.47% at missing rates of 0.1 and 0.5, respectively. This indicates that QAFMSA remains robust in fine-grained sentiment classification and under high missing rate conditions. Overall, as the number of classification categories increases, the boundaries between sentiment categories become more blurred, and model performance generally deteriorates more easily. QAFMSA assesses the reliability of each modality through a quality-aware mechanism and adaptively adjusts the fusion weights based on modality quality. This allows the model to prioritize the use of highly reliable modal information when modalities are missing or their quality deteriorates, reducing interference from missing modalities. Therefore, QAFMSA demonstrates good effectiveness and robustness in both four-class and seven-class classification tasks.

[0093] ablation experiment

[0094] To verify the effectiveness of each key module of QAFMSA, this paper constructs three ablation variants based on the complete model: QAFMSA-QE removes the modality quality assessment module and only uses missing modality states in subsequent fusion; QAFMSA-AF removes the adaptive fusion module and replaces weight allocation with average fusion; QAFMSA-RA removes the most reliable modality anchoring mechanism and directly uses the adaptive fusion results for subsequent encoding and classification. The ablation experiment results are shown in the table.

[0095] After removing the modal quality assessment module, the performance of QAFMSA-QE significantly decreased. When the missing value rate was 0%, F1 and ACC decreased by 2.06% and 0.64% respectively compared to QAFMSA; when the missing value rate was 0.5%, the decreases were 2.39% and 3.65% respectively. This indicates that the quality assessment module can effectively characterize modal reliability and prevent the model from introducing unreliable features under missing or low-quality modal conditions.

[0096] The performance degradation of QAFMSA-AF is most pronounced after removing the adaptive fusion module. When the missing data rate is 0%, F1 and ACC decrease by 3.38% and 2.72%, respectively; when the missing data rate is 0.2%, they decrease by 2.36% and 4.08%, respectively. This indicates that fixed-average fusion is insufficient to dynamically adjust modal contributions based on sample states, while adaptive fusion can more rationally allocate modal weights, which is key to improving model robustness.

[0097] After removing the most reliable modal anchoring mechanism, the performance of QAFMSA-RA also decreased. When the missing data rate was 0, 0.2, and 0.4, both the F1 and ACC scores were lower than the complete model, indicating that reliable modal anchoring can utilize highly reliable single-modal representations to correct fusion features and alleviate representation shift caused by low-quality modalities. Overall, the complete QAFMSA model achieved the best F1 and ACC scores across all missing data rate settings, demonstrating that modal quality assessment, adaptive fusion, and reliable modal anchoring all have a positive effect on improving model performance in scenarios with missing modalities.

[0098] Overall, the complete model QAFMSA achieved the best F1 score and ACC at all missing rate settings. As the missing rate increased from 0.0 to 0.5, the performance of all models generally declined, but QAFMSA consistently maintained superior performance, indicating that the three modules designed in this paper—modal quality assessment, adaptive fusion, and most reliable modal anchoring—all played a positive role in mitigating the performance degradation caused by missing modes.

[0099] Table 4 Module Ablation Experiment Results

[0100]

[0101] Example 2

[0102] This embodiment provides a multimodal sentiment analysis system based on modal quality perception and adaptive fusion.

[0103] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the multimodal sentiment analysis method based on modal quality perception and adaptive fusion.

[0104] A terminal device includes a processor and a computer-readable storage medium, the processor being configured to implement various instructions; the computer-readable storage medium being configured to store multiple instructions adapted for loading and execution by the processor of the multimodal sentiment analysis method based on modal quality perception and adaptive fusion.

[0105] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multimodal sentiment analysis method based on modal quality perception and adaptive fusion, characterized in that, include: Acquire multimodal sentiment data; Data preprocessing is performed based on the acquired multimodal sentiment data; Construct a multimodal sentiment analysis model based on quality perception and adaptive fusion, including modality-specific feature projection, single-modal Transformer context encoding, modality semantic-based quality assessment, and modality reliability-based adaptive fusion; The constructed multimodal sentiment analysis model is trained. Sentiment classification and prediction are performed using a trained multimodal sentiment analysis model.

2. The multimodal sentiment analysis method based on modal quality perception and adaptive fusion according to claim 1, characterized in that, The data preprocessing based on the acquired multimodal sentiment data includes feature extraction of text, audio, and visual modalities using a pre-trained BERT model, the Librosa toolkit, and the OpenFace 2.0 toolkit, respectively. For the text modality, let the input text sequence be: in, Indicates the first Each word element, The length of the text sequence is represented by the word segmentation algorithm, which maps the word index sequence to the given word index. This is then fed into the BERT encoder to obtain a text modal contextual representation: ,in, Indicates the first The contextualized text features corresponding to each lexical unit are as follows: For audio modes, let the original audio signal be... in, Indicates the first One original audio sampling point, This represents the original number of sampling points. Resampling it to a 16kHz mono signal is represented as: in, Indicates the number of resampled samples. One audio sampling point, Indicates the number of sampling points after resampling; followed by The frame is divided into frames based on each sampling point. Each audio frame is represented as: For each frame of audio, the zero-crossing rate, Mel frequency cepstral coefficients, and constant Q transform features are extracted and concatenated to obtain... in, This represents the feature concatenation operation, and has the following properties: Therefore, the audio modality is ultimately represented as: in, This represents the length of the audio sequence after frame division; for the visual modality, let the original video sequence be... in, Indicates the first Frame image, Indicates the length of the visual modality sequence, extracted using OpenFace 2.

0. Original visual features of the frame: in, , , These represent the frame number, timestamp, and face detection confidence level, respectively. This represents the core facial features, including facial landmarks, head posture, and eye movements; after removing auxiliary fields, the following is obtained: Therefore, the visual modality is ultimately represented as: After the above feature extraction, the three modalities are finally represented as follows: ,in, , and These represent the final input features for text, audio, and visual modalities, respectively.

3. The multimodal sentiment analysis method based on modal quality perception and adaptive fusion according to claim 2, characterized in that, The modality-specific feature projection includes introducing a modality-specific linear projection layer to eliminate the inconsistency in the original feature dimensions of text, audio, and visual modalities, mapping the three modalities to a unified latent space. Let the modality input be: ,in, , Representing modes The sequence length, This represents the original feature dimension of the modality, for any modality. Its modality-specific projection process is defined as follows: ,in, For modality Learnable projection matrix, Represents the unified latent space dimension. The modal sequence representation after projection is as follows: For text, audio, and visual modalities, they are: ， ， ， in: Combining the model input dimensions, we get: ,therefore: The three modal representations after projection satisfy: Since the projected input has already undergone modal masking, we have: Right now: Ultimately, the projections of the three modes are uniformly represented as: .

4. The multimodal sentiment analysis method based on modal quality perception and adaptive fusion according to claim 3, characterized in that, The single-modal Transformer context encoding includes, after completing feature projection, using a modality-specific Transformer encoder to perform context modeling on each modal sequence, assuming the ... The input to the layer Transformer is Its multi-head self-attention sublayer and feedforward network sublayer are represented as follows: in, This indicates a multi-head self-attention mechanism. This represents a location-based feedforward network. Representation layer normalization operation; for the first layer Each attention head first represents the input. These are mapped to the query matrix, key matrix, and value matrix, respectively: ,in, , , The first The learnable mapping matrix corresponding to each attention head , and Representing query, key, and value representations respectively; based on the scaled dot product attention mechanism, the... The output of each attention head is calculated as follows: ,in, The feature dimension of a single attention head. The scaling factor is used; then the outputs of all attention heads are concatenated, and the final result of multi-head self-attention is obtained through the output mapping matrix: ,in, Indicates the number of attention heads. This indicates the output mapping matrix.

5. The multimodal sentiment analysis method based on modal quality perception and adaptive fusion according to claim 4, characterized in that, The modal semantics-based quality assessment includes evaluating the content quality of each modality and generating a sample-level modal quality score for any modality. Let the contextualized sequence output by the single-mode encoder be represented as follows: in, Representing modes In the The hidden state at each time step. This indicates the sequence length of the mode. To represent a unified latent space dimension; to obtain a modality-level global semantic representation, temporal average pooling is performed on the sequence representation: After obtaining the modal-level global representation, a two-layer perceptron is used to... To perform quality mapping, first... The first nonlinear transformation is input to obtain the quality implicit representation. in, This is the first-level mapping matrix. For bias vectors, Represents the dimension of the latent space of mass. Represents a non-linear activation function. The modal quality implicit representation is then used to map the quality implicit representation to a scalar quality score. in, This is the second-level mapping matrix. For bias terms, This represents the quality score before normalization. Combining the two steps, we get: Then, substituting this into the average pooling representation, the quality score can be expressed in terms of the sequence hidden state: To constrain the quality scores to a stable range, the Sigmoid function is used for normalization: ,in, This represents the initial quality probability score. Considering that excessively low quality scores can lead to numerical instability in reliability modeling and fusion weight calculation, a lower bound for quality is introduced. And apply a truncation constraint to the quality score. ,in, Indicates the lower bound of the mass fraction. This represents the truncation function, and its equivalent form is: For the three modalities of text, audio, and vision, the following calculations were performed respectively: Finally, the mass fraction vectors of the three modes are represented as follows: ,in, Representing modes Content quality and discriminative value in the current sample.

6. The multimodal sentiment analysis method based on modal quality perception and adaptive fusion according to claim 5, characterized in that, The adaptive fusion based on modal reliability includes modeling modal reliability based on modal existence states and quality scores, whereby the modal existence vector is denoted as . The quality vector output by the quality assessment module is ,in, Representing modes Does it exist? Representing modes The content quality and discriminant value in the current sample, for any modality Its reliability is defined as Therefore, the sample-level reliability vectors of the three modalities are represented as follows: ,Right now ,in, This represents element-wise multiplication. Representing modes The overall reliability in the current sample, due to ,and Therefore, reliability satisfy Specifically, when the mode When missing, there is When the mode When it exists, there is Therefore, reliability is in segmented form: To avoid This leads to numerical instability; a smoothing reliability is defined as follows: in, Since is a minimal constant, the corresponding smooth reliability vectors for the three modes are expressed as: Will Substituting into the above equation, we can obtain Therefore, it can be written in segment form as follows because ,when When, it is simplified to Based on this, when reliability needs to be used as a priori in the calculation of fusion weights, smoothed reliability is used in the calculation: .

7. The multimodal sentiment analysis method based on modal quality perception and adaptive fusion according to claim 6, characterized in that, The adaptive fusion based on modal reliability further includes, after completing modal reliability modeling, learning an adaptive fusion representation using modal semantic representation, quality information, missing states, and reliability information. Let the encoding representations for text, audio, and visual modalities be respectively... For any mode ,have in, Representing modes The sequence length, To represent a unified latent space dimension, and considering the differences in semantic distribution and feature scale across different modalities, a modality-level learnable transformation matrix is ​​introduced to perform a nonlinear mapping on the encoded sequence representation. in, For modality The corresponding learnable parameter matrix, This represents the hyperbolic tangent activation function. Let the transformed modal sequence be represented as follows: in, Representing modes In the The hidden state after transformation at each time step, followed by... Average pooling is performed along the time dimension to obtain the modality-level semantic representation used for fusion: Its matrix form is Substitute ,have to Thus, the fused semantic representations corresponding to the three modalities are obtained: To enable the fusion module to simultaneously perceive modal semantic state, quality level, existence state, and reliability information, a fusion context vector is constructed. ,in, This represents a vector concatenation operation. Represents the mass vector. This indicates that the mode has a vector. Represents a reliability vector, based on the fusion context vector. Initial preference scores for the three modalities are generated using a linear mapping: in, and For learnable parameters, Representing modes The initial fusion preference score in the current sample is used to explicitly inject modal reliability into the fusion weight allocation process. Before normalization, a smooth reliability prior is introduced, assuming the smooth reliability is... Then mode The fused logits are represented as ,in, This represents the reliability prior; then, the Softmax function with a temperature coefficient is used to calculate the final fusion weights. ,in, The temperature coefficient is used to obtain the fusion weight vector for the three modes: Finally, the semantic representations of the three modalities are weighted and summed according to the fusion weights to obtain the adaptive fusion representation: Its unified form is expressed as ,in, This represents the multimodal representation after dynamic fusion.

8. The multimodal sentiment analysis method based on modal quality perception and adaptive fusion according to claim 7, characterized in that, The training of the constructed multimodal sentiment analysis model includes incorporating classification prediction loss, modality quality assessment loss, and fusion consistency loss into a joint optimization objective to achieve collaborative training of sentiment classification, modality quality estimation, and adaptive fusion processes. For the sentiment classification task, labeled smooth cross-entropy loss is used for supervision. Let the first... The true label of each sample is one-hot encoded as follows: The predicted probability is ,in , Indicates the number of categories. Let be the label smoothing coefficient, then the smoothed label is represented as: The corresponding classification prediction loss is defined as: ,in, Let represent the sample size; to improve the stability of the quality estimation, a lightweight regularization constraint is introduced for the modal quality score, assuming the th... Each sample in modality The mass fraction on is ,in , representing text, audio, and visual modalities respectively. To suppress excessive bias of quality scores towards 0 or 1 in the early stages of training, a reference center is introduced. The modal quality assessment loss is defined as: ,in, Let the number of samples be represented; further, a fusion consistency regularization term is constructed to impose soft constraints on the weight allocation process, assuming the first... Each sample in modality The reliability of the above is To avoid instability caused by excessively small values, a smoothing process is performed: ,in, The constant is a very small constant; then the smoothed reliability is normalized to obtain the sample-level reliability distribution: Let the first Each sample in modality The adaptive fusion weights learned above are Then the fusion consistency loss is defined as: ,in, The sample size represents the total number of samples. Combining these three components, a joint loss function is ultimately used to optimize the model, expressed as follows: ,in, and This is a hyperparameter used to balance the importance of modal quality assessment loss and fusion consistency loss in the total loss.

9. The multimodal sentiment analysis method based on modal quality perception and adaptive fusion according to claim 8, characterized in that, The process of using a trained multimodal sentiment analysis model for sentiment classification and prediction includes, firstly, refining the fusion representation by... Expand into a sequence of length 1: ,in, This represents the initial input to the post-fusion Transformer encoder. Let the dimension expansion operation be represented by the first dimension. The input to the post-layer blending Transformer is Then its query, key, and value matrices are as follows: , in, , , They represent the first In the post-blending Transformer, the first layer The query, key, and value representation of each attention head. , , For the corresponding learnable parameter matrix, the first... The output of each attention head is: ,in, Indicates the post-fusion Transformer, the first The output of each attention head, The feature dimension of a single attention head is represented by the following: The output of multi-head attention is: in, Indicates the first Post-layer fusion multi-head attention output, This represents the output mapping matrix; subsequently, residual connections and layer normalization are performed. ,in, Indicates the first The output of the attention sublayer is fused after the layer, and the feedforward network is computed as follows: ,in, Indicates the first Post-layer fusion feedforward network output, , This represents the feedforward network weight matrix. , Indicates the bias term, the first The post-layer fusion encoding output is: ,in, Indicates the first The output of the Transformer is then blended after layers, and... After layer encoding, we get: in, This indicates the number of layers in the post-fusion Transformer encoder. The refined fused sequence representation is then used; subsequently, a Sigmoid attention aggregation module is employed to obtain the global fused representation. in, To represent the global fusion representation after aggregation; to represent the attention aggregation process, let: in, Indicates the th in the post-fusion sequence The hidden state of each location, Let the length of the post-fusion sequence be represented, then the attention intermediate is represented as: in, Indicates the first Attention at each position is represented in the middle. , For learnable weight matrix, The attention score is represented by the Sigmoid function as follows: ,in, Indicates the first Attention score for each position, The attention score mapping matrix is ​​represented by the normalized attention weights as follows: ,in, Indicates the first Attention weights corresponding to each position, and satisfying Therefore, global fusion is represented as: in, This represents the fused representation after attention aggregation. Finally, Input the classification layer, and obtain it through pre-classification projection: in, This indicates implicit representation before classification. This represents the projection matrix before classification, and the category logits are calculated as follows: ,in, This represents the unnormalized category score. Represents the classification weight matrix. Indicates the category bias term. The number of sentiment categories is represented, and the final predicted probability distribution is as follows: Right now: ， in, Indicates that the sample belongs to the first The predicted probabilities of each sentiment category are given, and the final sentiment category is determined by the category with the highest probability. in, This represents the sentiment category predicted by the model.

10. A multimodal sentiment analysis system based on modal quality perception and adaptive fusion, characterized in that, include: The data acquisition module is configured to acquire multimodal sentiment data; The preprocessing module is configured to perform data preprocessing based on the acquired multimodal sentiment data; The adaptive fusion module is configured to build a multimodal sentiment analysis model based on quality perception and adaptive fusion, including modality-specific feature projection, single-modal Transformer context encoding, modality semantic-based quality assessment, and modality reliability-based adaptive fusion. The training module is configured to train the constructed multimodal sentiment analysis model. The prediction module is configured to perform sentiment classification prediction using a trained multimodal sentiment analysis model.

Images