A multimodal sentiment analysis method and related device

Abstract

This invention belongs to the field of multimodal sentiment analysis and discloses a multimodal sentiment analysis method and related apparatus. It achieves efficient multimodal sentiment analysis through time alignment, a capacity-aware routing network, and τ-threshold optimization decision-making. Capacity matching routing is performed based on modal feature dimensions to avoid overfitting to low-dimensional non-verbal modalities. A linear classifier and τ-threshold optimization are used instead of the traditional Transformer and fixed threshold. Non-verbal modalities with feature dimensions no greater than a preset threshold are routed to a lightweight network to suppress noise propagation. Audio and visual modalities with feature dimensions greater than the preset threshold are modeled using a dual-path state-space model. The τ-threshold optimization inference engine obtains the optimal threshold through validation set optimization, making the decision boundary adaptable to long-tail distribution and improving the discriminative power of boundary samples. This method reduces computational complexity, enhances robustness to modal noise, improves classification balance in long-tail scenarios, and is more suitable for high-throughput real-time inference and edge deployment requirements.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically to the field of multimodal sentiment analysis, and particularly relates to a multimodal sentiment analysis method and related apparatus. Background Technology

[0002] Multimodal Sentiment Analysis (MSA) is an important research branch of sentiment computing in the field of artificial intelligence. Its core goal is to integrate information from multiple heterogeneous modalities such as text, audio, and vision to accurately identify human emotional intentions. It has wide applications in various industrial and public service scenarios, including intelligent customer service, public opinion monitoring, edge intelligent devices, and large-scale long video stream processing, demonstrating high practical value and promising industrial prospects. In recent years, with the rapid development of deep learning technology, multimodal fusion models based on the Transformer architecture have become a research hotspot in the MSA field. Mainstream models such as MulT and MISA have emerged, achieving significant progress in cross-modal information alignment and sentiment feature extraction. This has promoted the industrialization of multimodal sentiment analysis technology and provided new technical paths for sentiment identification in complex scenarios.

[0003] However, in practical industrial applications, existing multimodal sentiment analysis technologies based on the Transformer architecture still have many shortcomings that need to be addressed, making it difficult to meet the high-performance requirements of real-world applications. Firstly, the quadratic complexity of the global attention mechanism leads to a severe computational bottleneck. Existing mainstream multimodal networks heavily rely on the Transformer's self-attention mechanism to achieve cross-modal alignment. When processing long-term multimodal signals, its computational complexity is quadratic of the input sequence length (…). First, the excessive growth in data volume leads to high memory usage and large inference latency, failing to meet the high-throughput real-time inference requirements of hundreds of frames per second in industry. Second, blindly stacking features leads to the problem of "modal noise overfitting." In real-world natural datasets such as CMU-MOSI, audio and visual modalities are easily affected by environmental noise and sudden changes in illumination, exhibiting low-dimensionality and high-noise characteristics. Existing large-capacity models force the application of complex networks of equal depth to all modalities, which not only fails to effectively extract core emotional information but also causes the model to overfit to the noisy features of inferior modalities, resulting in catastrophic performance collapse during cross-modal fusion. Finally, the static threshold decision that ignores extreme long-tail distributions has obvious defects. Emotional data in real-world scenarios often exhibits an extremely unbalanced long-tail distribution, such as the vast majority of samples being neutral emotions and a few being strong emotions. Existing technologies generally use fixed static thresholds for class classification, resulting in extremely low recall rates for positive class samples when the model processes samples with blurred boundaries or long tails. The macro-F1 score is far lower than the accuracy, greatly limiting the actual robustness of the model.

[0004] It is evident that existing multimodal sentiment analysis technologies based on the Transformer architecture have significant shortcomings in terms of computing power adaptation, resistance to modal noise interference, and long-tail sample processing, and cannot meet the high-performance requirements of practical application scenarios such as edge device deployment and large-scale long video stream processing. Summary of the Invention

[0005] This invention provides a multimodal sentiment analysis method and related apparatus. This method can effectively solve the problems of computing power adaptation, anti-modal noise interference and long-tail sample processing in multimodal sentiment analysis technology based on Transformer architecture. It can meet the high performance requirements of practical application scenarios such as edge device deployment and large-scale long video stream processing.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal sentiment analysis method includes: The text, audio, and visual multimodal input signals of the target object's emotional expression are acquired, heterogeneous feature sequences are extracted, and an alignment algorithm based on sequence truncation and padding is used to map the heterogeneous feature sequences into temporal feature matrices of the same time dimension. Based on the feature dimensions of the temporal feature matrices of each modality, data of different modalities are routed to a backbone network with matching capacity in order to extract the single-modal high-level semantic representations corresponding to the data of different modalities. The single-modal high-level semantic representations corresponding to different modal data are concatenated along the feature dimension. The fused features obtained by concatenation are input into a pre-trained multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix. Based on the optimal decision threshold, the sentiment prediction probability distribution or logits matrix is ​​binarized or multi-class activation is determined, and the corresponding sentiment classification label is generated according to the determination result to output the final sentiment analysis result; wherein, the optimal decision threshold is obtained by optimizing the inference engine on the validation set based on the τ threshold.

[0007] Furthermore, the step of acquiring text, audio, and visual multimodal input signals of the target object's emotional expression, extracting heterogeneous feature sequences, and mapping the heterogeneous feature sequences into temporal feature matrices of the same time dimension using an alignment algorithm based on sequence truncation and padding includes: Acquire text, audio, and visual multimodal input signals of the target object's emotional expression; The initial multimodal feature set is constructed from text, audio, and visual multimodal input signals; During the model training phase, a neutral bandwidth filtering threshold is introduced to filter out noisy samples in the initial multimodal feature set. The filtering principle is as follows:

[0008] In the formula, This indicates the filtered sample; Indicates the original continuous sentiment label; Indicates the neutral bandwidth filtering threshold; After filtering, the neutral interval The samples are discarded, resulting in truncated heterogeneous feature sequences; After aligning the truncated heterogeneous feature sequences, a temporal feature matrix with the same time dimension is obtained.

[0009] Furthermore, the step of routing different modal data to a backbone network with matching capacity based on the feature dimensions of each modality's temporal feature matrix to extract single-modality high-level semantic representations corresponding to different modal data includes: The text modal data is routed to a deep temporal network built on a bidirectional state-space model to obtain the first unimodal high-level semantic representation. Non-linguistic modal data with a feature dimension not greater than a preset threshold are routed to a multilayer perceptron projection network to obtain a second single-modal high-level semantic representation. Audio modal data with feature dimensions greater than a preset threshold are routed to a pre-constructed time-frequency dual-path state space model, and visual modal data with feature dimensions greater than a preset threshold are routed to a pre-constructed spatiotemporal dual-path state space model. Joint modeling is performed on both models to obtain a third single-modal high-level semantic representation.

[0010] Furthermore, the step of routing the text modal data to a deep temporal network constructed based on a bidirectional state-space model to obtain a first unimodal high-level semantic representation includes: A deep temporal network employing a bidirectional continuous-time state-space model is used to discretize and model text modal data, where: For a given discretization step size State transition matrix and input matrix After zero-order preservation discretization, it can be represented as:

[0011]

[0012] in, It is the identity matrix; This is the discretized state transition matrix; The input matrix after discretization; It is a matrix exponential function; The specific operations for forward state update and output mapping are as follows:

[0013]

[0014] In the formula, This is the updated forward hidden state vector; This is the forward hidden state vector of the previous time step; This represents the text modal data at the i-th time step; This is the output matrix; Forward output features; The reverse output features of the reverse state are calculated in the same way as the forward state update and output mapping. ; The forward output features and the backward output features are added together to obtain the dynamic hidden state sequence of the text modality data; Global average pooling is performed along the time dimension on the dynamic hidden state sequence to extract the global semantic representation. The global semantic representation is output as the first unimodal high-level semantic representation, and the specific formula is as follows:

[0015] In the formula, This represents the total time step. For feature space; To unify the feature dimensions of the hidden layer; The step of routing non-linguistic modal data with a feature dimension not greater than a preset threshold to a multilayer perceptron projection network to obtain a second unimodal high-level semantic representation includes: When the feature dimension of the non-linguistic modality data is not greater than a preset threshold, the temporal state update is automatically cut off, and the corresponding non-linguistic modality data is routed to a multilayer perceptron projection network. High-frequency noise is filtered out using linear projection and nonlinear activation to obtain a regularized global feature f. The regularized global feature is then output as the second unimodal high-level semantic representation, as shown in the following formula:

[0016] In the formula, This is the time step mean vector of the original sequence; , All are learnable weight matrices; It is the bias vector; Use the GELU activation function; For Dropout regularization operations; The process of routing audio modal data with feature dimensions greater than a preset threshold to a pre-constructed time-frequency dual-path state space model, and routing visual modal data with feature dimensions greater than a preset threshold to a pre-constructed spatiotemporal dual-path state space model, and performing joint modeling to obtain a third single-modal high-level semantic representation, includes: When the feature dimension of audio modal data or visual modal data exceeds a preset threshold, a parallel state-space model branch is activated for deep decoupling modeling, where: The audio modal data is divided into a time domain matrix and a frequency domain matrix. The time domain matrix and the frequency domain matrix are then fused with dynamic gated features in the time domain and frequency domain to obtain the first fused gated feature. Visual modal data are decoupled into spatial static features and temporal dynamic changes using the same dual-path mechanism. The spatial static features and temporal dynamic changes are then processed by a dual-path state space module and gating aggregation to obtain the second fused gating feature. The first fusion-gated feature and the second fusion-gated feature are respectively used as the output of the single-modal high-level semantic representation of the corresponding modality.

[0017] Furthermore, the step of cascading the single-modal high-level semantic representations corresponding to different modal data along the feature dimension, and inputting the fused features obtained from the cascaded model into a multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix includes: The high-level semantic representations of single modalities corresponding to different modalities are concatenated along the feature dimension to obtain fused features. The specific formula is as follows:

[0018] In the formula, , , These represent the single-modal high-level semantics corresponding to different modal data; Indicates a splicing operation; The fused features are input into a pre-trained multimodal linear classifier to obtain a fused classification head, which yields the sentiment prediction probability distribution or logits matrix, represented as:

[0019] In the formula, For classification weights, This is a positive sentiment prediction probability. This is the Sigmoid activation function.

[0020] Furthermore, the multimodal linear classifier calculates the network error based on a focal loss function with class weights. To complete the training, the specific formula is as follows:

[0021] In the formula, This is the category weight matrix; For focusing parameters; This is a real label.

[0022] Furthermore, the optimal decision threshold is obtained by optimizing the inference engine on the validation set based on the τ threshold, including: The τ-dynamic optimization inference engine performs optimization operations on the validation set to obtain the optimal decision threshold, including: After the validation set inference is completed, an optimization function is constructed with the macro average F1 score as the sole objective. Perform a fine-grained sliding window search within the continuous interval [0,1] to solve for the dynamic decision boundary τ that maximizes the macro-average F1 score of the current fusion distribution. :

[0023] in, and These represent the threshold values ​​respectively. Next category Precision and recall rates; The dynamic decision boundary obtained through the optimization operation is output as the optimal decision threshold. Among them, if If the result is positive, it is considered a positive sentiment sample; otherwise, it is considered a negative sentiment sample.

[0024] A multimodal sentiment analysis system, comprising: The data acquisition module is used to acquire text, audio and visual multimodal input signals of the target object's emotional expression, extract heterogeneous feature sequences, and use an alignment algorithm based on sequence truncation and padding to map the heterogeneous feature sequences into a temporal feature matrix with the same time dimension. The capacity-aware routing module is used to route data of different modalities to a backbone network with matching capacity based on the feature dimensions of the temporal feature matrix of each modality, so as to extract the single-modal high-level semantic representations corresponding to the data of different modalities. The concatenation module is used to concatenate the single-modal high-level semantic representations corresponding to different modal data along the feature dimension, and input the fused features obtained by concatenation into a pre-trained multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix. The decision module is used to binarize or multi-class activation of the sentiment prediction probability distribution or logits matrix based on the optimal decision threshold, and generate corresponding sentiment classification labels according to the decision results to output the final sentiment analysis results; wherein, the optimal decision threshold is obtained by optimization on the validation set based on the τ threshold optimization inference engine.

[0025] A multimodal sentiment analysis device, comprising: Memory, used to store computer programs; A processor is used to implement the above-described multimodal sentiment analysis method when executing the computer program.

[0026] A computer-readable storage medium storing a computer program that, when executed by a processor, is used to implement the above-described multimodal sentiment analysis method.

[0027] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a multimodal sentiment analysis method that achieves efficient multimodal sentiment analysis through temporal alignment, a capacity-aware routing network, and τ-threshold optimization decision-making. Capacity-matched routing is performed based on modality feature dimensions to avoid overfitting to low-dimensional non-verbal modalities; and a linear classifier and τ-threshold optimization are used instead of the traditional Transformer and fixed threshold. In this method, non-verbal modalities with feature dimensions no greater than a preset threshold are routed to a lightweight network to suppress noise propagation; audio and visual modalities with feature dimensions greater than the preset threshold are modeled using a dual-path state-space model; and the τ-threshold optimization inference engine obtains the optimal threshold τ through validation set optimization. This method adapts the decision boundary to a long-tail distribution, improving the discriminative power of boundary samples. By reducing computational complexity, this method enhances robustness to modal noise and improves classification balance in long-tail scenarios, thus better meeting the needs of high-throughput real-time inference and edge deployment. Attached Figure Description

[0028] Figure 1 A schematic diagram illustrating the principle architecture of the multimodal sentiment analysis method provided in this embodiment of the invention; Figure 2 A radar chart comparing the multimodal collaborative performance of the CH-SIMS dataset provided in this embodiment of the invention; Figure 3 The following are comparison curves of model training and validation set loss provided in the embodiments of the present invention; wherein, (a) is a comparison curve of validation loss; and (b) is a comparison curve of macroscopic F1 values ​​of validation. Figure 4 This is a bar chart comparing the performance of different modal combinations provided in the embodiments of the present invention. Figure 5 This is a core flowchart of a multimodal sentiment analysis method provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a multimodal sentiment analysis system provided in an embodiment of the present invention. Detailed Implementation

[0029] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0030] For example, addressing the shortcomings of existing technologies such as high computational complexity, susceptibility to noise overfitting in poor / low-dimensional modalities, and the failure of static thresholds in long-tailed distributions, this embodiment provides a multimodal sentiment analysis method: a lightweight multimodal sentiment analysis method based on a State Space Model (SSM). This method aims to overcome the computational bottleneck of long-sequence multimodal data by utilizing a linearly complex State Space Model (SSM), pioneering a "Capacity-Aware Routing" mechanism to achieve cross-modal regularization, and introducing a τ-threshold optimization inference engine to achieve efficient, robust, and real-time classification and recognition of extremely imbalanced multimodal sentiment features.

[0031] like Figure 5 As shown, this embodiment provides a multimodal sentiment analysis method, including: The text, audio, and visual multimodal input signals of the target object's emotional expression are acquired, heterogeneous feature sequences are extracted, and an alignment algorithm based on sequence truncation and padding is used to map the heterogeneous feature sequences into temporal feature matrices of the same time dimension. Based on the feature dimensions of the temporal feature matrices of each modality, data of different modalities are routed to a backbone network with matching capacity in order to extract the single-modal high-level semantic representations corresponding to the data of different modalities. The single-modal high-level semantic representations corresponding to different modal data are concatenated along the feature dimension. The fused features obtained by concatenation are input into a pre-trained multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix. Based on the optimal decision threshold, the sentiment prediction probability distribution or logits matrix is ​​binarized or multi-class activation is determined, and the corresponding sentiment classification label is generated according to the determination result to output the final sentiment analysis result; wherein, the optimal decision threshold is obtained by optimizing the inference engine on the validation set based on the τ threshold.

[0032] The multimodal sentiment analysis method provided in this embodiment will now be described in detail with reference to the accompanying drawings: For example, such as Figure 1 As shown in the figure, this embodiment provides a multimodal sentiment analysis method, and the specific steps are as follows: Step 1: Alignment and Preprocessing of Heterogeneous Multimodal Features Acquire multimodal input streams containing human (target object) emotional expressions, including text, audio, and visual data. Obtain heterogeneous feature sequences using pre-trained feature extractors, and map the text, audio, and visual features into temporal feature matrices with the same time dimension using an alignment algorithm based on sequence truncation and padding.

[0033] Step 2: The Capacity-Aware Routing Backbone, based on feature dimensions, is the core noise reduction mechanism in this embodiment. In this mechanism, the system routes data from different modalities to a backbone network with matching capacity for processing based on the feature dimensions of each modal input. The specific routing rules are as follows: For text modal data, the system routes it to a deep temporal network based on a bidirectional state space model for advanced feature extraction; for non-linguistic modal data with a feature dimension no greater than a preset threshold, the system routes it to a lightweight multilayer perceptron projection network for processing; for audio modal data and visual modal data with a feature dimension greater than a preset threshold, the system routes them to a time-frequency dual-path state space model and a spatiotemporal dual-path state space model for joint modeling, respectively.

[0034] Explained, this capacity-matching routing design not only effectively filters high-frequency noise in the time series, but also acts as a natural cross-modal strong regularizer, fundamentally suppressing the overfitting of complex sequence models to low-dimensional non-verbal modal noise features.

[0035] Step 3: Dual-Path Spatio-Temporal Modeling (SSM-TF) When the feature dimensions of the audio and visual modal exceed a preset threshold, the system activates a customized state space branch designed specifically for non-textual modalities—the Audio Dual-Frequency Path (SSM-TF): Audio features are decoupled into time domain and frequency domain features, scanned through two independent state space modules, and then fused using a learnable gating mechanism. Visual Spatio-Temporal Path (SSM-ST): Visual features are decoupled into spatial static residuals and temporal dynamic changes. It also employs dual-path state space modules and gating aggregation technology to extract the deep spatio-temporal evolution of facial micro-expressions and body movements.

[0036] Step 4: Cross-modal feature fusion and The threshold-optimized inference engine (Cross-Modal Fusion and τ-Threshold-Optimized Inference) concatenates the single-modal high-level semantic representations output from steps two and three along the feature dimension, inputting them into the multimodal fusion classification head, and outputting the basic continuous probability distribution or logits matrix. To address the long-tail class imbalance problem, the system integrates... - Threshold optimization inference engine. This engine constructs a non-convex optimization function with the macro-average F1 score as the sole optimization objective during the validation phase. The system... Fine-grained threshold sliding window sampling is performed within the interval to automatically search for the dynamic decision boundary threshold that optimizes the current fusion distribution. In the final testing and deployment phase, the system abandons the traditional static... Threshold, forcing the adoption of this optimal decision threshold. Binarization or multi-class activation is performed, which greatly improves the recall of weak sentiment positive samples.

[0037] For example, the multimodal sentiment analysis method provided in the above embodiments is further implemented in a specific manner, and the implementation steps are as follows: Step 1: Task Formalization and Multimodal Preprocessing: Acquire text, audio, and visual multimodal input signals from the target object to construct an initial multimodal feature set. During the model training phase, a neutral bandwidth filtering threshold is introduced to eliminate noise interference from ambiguous sentiment in the dataset. (For example The threshold is dynamically determined based on the prior or validation set optimization results of the sentiment intensity distribution of different datasets to filter ambiguous noise samples; while for very large datasets that require evaluation of the model's long-tail robustness (such as CMU-MOSEI), the threshold is adaptively set. The full sample size was retained for stress resistance training.

[0038] Let the original continuous sentiment label be The training set is binarized and filtered using the following rules:

[0039] neutral range Samples that do not meet the criteria will be discarded. After preprocessing and feature alignment, the text feature matrix is ​​denoted as... The audio feature matrix is ​​denoted as The visual feature matrix is ​​denoted as ;in These represent the sequence lengths of the three modes, respectively. These represent the feature dimensions of the three modalities, respectively.

[0040] 2. Text Modal Coding: Bi-SSM Encoder (Bi-SSM model) Targeting high-dimensional text features A bidirectional continuous-time state-space model (SSM) is used for discretized sequence modeling. For a given discretization step size... State transition matrix and input matrix After discretization using zero-order hold (ZOH), it is represented as:

[0041]

[0042] in, It is an identity matrix.

[0043] The specific operations for forward state update and output mapping are as follows:

[0044]

[0045] Similarly, calculate the reverse state output. The dynamic hidden state sequence representation of the text sequence is obtained by summing the bidirectional outputs. Global average pooling is performed along the time dimension to extract the global semantic representation of the fixed-length text. :

[0046] in, A unified hidden layer feature dimension is set for the model.

[0047] 3. Audio and video modal coding: Capacity-Aware Routing: Non-linguistic modal feature dimensions vary greatly in natural scenes. This embodiment presets a modal dimension threshold. (like The threshold setting rule is determined based on the statistical distribution of the feature extractor's output dimension and hyperparameter search. Routing decisions are automatically performed based on the dimension of the input features. Path 1: Downgrade to strongly regularized route (when) or ): When non-verbal modalities (such as 5D audio and 20D vision in CMU-MOSI) exhibit low-dimensional, high-noise characteristics, the system automatically cuts off complex temporal state updates and routes the modal data to a lightweight multilayer perceptron (MLP) network, utilizing its linear projection and nonlinear activation to filter out high-frequency noise.

[0048]

[0049] in, and This is the time step mean vector of the original sequence; For learnable weight matrix, This is the bias vector. This path, while extracting shallow statistical features, also naturally provides cross-modal regularization.

[0050] Path 2: Dual-path spatiotemporal / time-frequency advanced routing (when...) or ): When the feature dimension of audio or visual modal data exceeds a preset threshold, a parallel, customized state space module is activated for deep decoupled modeling. Taking audio modality TF-SSM as an example, [the module is used to perform this]. Divided into time domain matrices With frequency domain matrix :

[0051] Then, dynamic gating feature fusion in the time and frequency domains is performed. First, the global mean pooling vectors of the two branches are calculated separately. and Then, the fusion gating factor is calculated through linear projection. :

[0052] This gating factor is used to perform soft attention-weighted fusion, outputting the final audio representation. :

[0053] in, It is the Sigmoid activation function. This is an element-wise multiplication operation.

[0054] The visual modality (ST-SSM) employs the same dual-path mechanism, decoupling it into spatial static features and temporal dynamic changes. The output is then processed through a dual-path state-space module and gated aggregation. .

[0055] 4. Cross-modal feature fusion and Focal Loss optimization: Obtain the high-level semantic features of the output of the aforementioned three routes. Then, the data is concatenated along the feature dimension and input into a multimodal linear classifier:

[0056]

[0057] in, This indicates a splicing operation. For classification weights, This represents the predicted probability of a sample belonging to positive sentiment. To address the long-tail bias caused by the imbalance of positive and negative samples in the training set, this system employs a focal loss function with class weights to calculate the network error.

[0058]

[0059] in, The class weight matrix is ​​dynamically calculated based on the prior distribution of the training set. To adjust the focusing parameters for the proportion of loss in difficult samples (set in this embodiment) ).

[0060] 5. -Threshold Optimization Inference Engine: To overcome the performance bottleneck caused by the fixed threshold during the testing phase, this model was designed... Threshold inference engine. After the validation set inference is complete, an optimization function is constructed with the macro-average F1 score as the sole objective. The system then... Perform a fine-grained sliding window search within a continuous interval to find the dynamic decision boundary that maximizes the Macro-F1 score of the current fusion distribution.

[0061]

[0062] in, and These represent the threshold values ​​respectively. Next category Precision and recall.

[0063] In the final testing or actual industrial deployment phase, the system extracts the pre-defined threshold parameters. If and only if the network outputs a probability At that time, it was judged as a positive sentiment sample. Conversely, negative emotions ( .

[0064] For example, in order to verify the effectiveness of the multimodal sentiment analysis method and its various innovative modules proposed in this embodiment, this embodiment conducted a comprehensive experimental evaluation on three industry-recognized multimodal sentiment analysis benchmark datasets (CH-SIMS, CMU-MOSI, CMU-MOSEI).

[0065] The overall performance comparison experiment is as follows: To verify the absolute performance advantage and ultra-lightweight features of this embodiment, a rigorous horizontal comparison was conducted with six mainstream / cutting-edge baseline models in the current multimodal sentiment analysis field, including: TFN, LMF, MULT, MISA, as well as MAG-BERT and Self-MM, which rely on large pre-trained models with hundreds of millions of parameters.

[0066] Therefore, the technical effect analysis is as follows: First, exceptional efficiency-performance trade-off: As shown in Table 1, this embodiment utilizes only a minimal number of 14.2M parameters, comprehensively surpassing earlier lightweight and mid-range models (such as TFN and MISA) in all metrics. More importantly, in comparison with large pre-trained frameworks (MAG-BERT, Self-MM) with over 110M parameters, this embodiment achieves state-of-the-art (SOTA) recognition accuracy with less than one-tenth of their computational cost, demonstrating extremely high industrial application value.

[0067] Table 1 compares the performance of this embodiment with existing mainstream multimodal models. Models Parameter magnitude CH-SIMS (Acc) CH-SIMS (Macro-F1) CMU-MOSI (Acc) CMU-MOSI (Macro-F1) CMU-MOSEI (Acc) CMU-MOSEI (Macro-F1) TFN Lightweight 72.73% 72.37% 79.08% 79.11% 81.89% 81.74% LMF Lightweight 72.05% 71.49% 79.18% 79.15% 83.48% 83.36% MULT medium amount 80.66% 80.75% 80.20% 80.10% 84.63% 84.52% MISA medium amount 76.40% 76.52% 80.50% 80.25% 83.70% 83.60% MAG-BERT Weight (110M+) 76.02% 76.12% 84.30% 84.30% 85.20% 85.10% Self-MM Weight (110M+) 76.02% 78.28% 84.00% 84.42% 85.30% 85.30% SSM-Lite (Ours) Ultra-lightweight (14.2M) 83.42% 83.34% 80.64% 80.11% 84.20% 83.06% Second, strong generalization across multiple languages ​​and scenarios: On the high-quality Chinese dataset CH-SIMS, this embodiment demonstrates absolute dominance, with Acc and Macro-F1 scores reaching 83.42% and 83.34% respectively, achieving an absolute performance leap of nearly 3%; it also maintains excellent generalization performance on the English datasets MOSI and MOSEI, which are extremely challenging.

[0068] Third, Modality Ablation Study. To verify the synergistic gain effect of non-linguistic modalities (audio and vision) in this embodiment framework, modality stripping experiments were conducted on the CH-SIMS dataset. Simultaneously, a τ-threshold inference engine was enabled for threshold optimization, as shown in Table 2. The comparison of multimodal synergistic performance on the CH-SIMS dataset and the comparison of model training and validation set losses are detailed below. Figure 2 and Figure 3 As shown in (a) and (b) in the figure. The performance ablation comparison of different modal combinations is detailed below. Figure 4 As shown.

[0069] Table 2 shows the experimental results of multimodal collaborative ablation on the CH-SIMS dataset. Experimental setup Text (T) Audio (A) Visual (V) Acc(Tuned) Macro-F1 (Tuned) Optimal τ value Text Only √ × × 79.03% 78.87% 0.765 T + A √ √ × 77.28% 76.89% 0.43 T + V √ × √ 82.99% 82.95% 0.38 T + A + V (Full) √ √ √ 83.42% 83.34% 0.41 like Figures 2 to 4 As shown, further technical effect analysis is as follows: 1) Multimodal collaboration brings significant gains: Compared with single-text (Text Only) and full-modal (T+A+V), the model accuracy improved by 4.39% in absolute terms after adding audio and video features. This proves that the multimodal fusion mechanism in this embodiment effectively captures cross-modal complementary information.

[0070] 2) Noise-resistant regularization effect of visual modal: The validation set loss and Macro-F1 curves show that when only text and audio (T+A) are used, the model exhibits significant overfitting (a surge in validation loss) in the later stages of training. However, when the visual modal (T+A+V) is introduced, the loss curve shows a perfectly U-shaped "robust" characteristic with a smooth decline. This demonstrates that in the architecture design of this embodiment, visual features not only provide incremental information but also play a regularization role in suppressing high-frequency audio noise and stabilizing network gradients.

[0071] Fourth, regarding the capacity-aware routing architecture ablation study: For the core innovation of this embodiment—"Capacity-Aware Routing"—ablation testing was conducted at the architecture level on the CMU-MOSI dataset, which has extremely low feature dimensionality and high noise floor. The performance difference between forcibly enabling complex networks (Full SSM) and adaptive degradation networks (MLP) was tested, as shown in Table 3.

[0072] Table 3 shows the ablation experiment results of the capacity-aware routing architecture on the CMU-MOSI dataset. Model Configuration T A V Acc (Argmax) Acc (Tuned) Macro-F1 (Tuned) Full SSM (no downgrade) Bi-SSM TF-SSM ST-SSM 79.27% 79.27% 78.22% Bi-SSM+MLP+ST-SSM Bi-SSM MLP ST-SSM 79.42% 79.88% 78.97% Bi-SSM+TF-SSM+MLP Bi-SSM TF-SSM MLP 81.02% 79.27% 77.78% SSM-Lite (Downgrade) Bi-SSM MLP MLP 80.64% 80.64% 80.11% Therefore, the specific technical effects can be analyzed as follows: In the CMU-MOSI dataset, the audio and video dimensions are extremely low (e.g., audio is only 5-dimensional and visual is only 20-dimensional).

[0073] 1. If all modalities are forced to take the advanced complex route (Full SSM: Bi-SSM+TF-SSM+ST-SSM), the model suffers from severe overfitting, with a Tuned Accuracy of only 79.27%.

[0074] 2. When this embodiment starts "capacity-aware routing" and determines that the non-verbal modality dimension is less than the set threshold, it intelligently downgrades and guides it to Lightweight MLP (Mamba+MLP+MLP configuration), the accuracy rises to 79.73% (and even the peak Acc exceeds 81.02% in some cross-downgrade combinations), and Macro-F1 reaches as high as 79.47%.

[0075] like Figure 6 As shown, exemplarily, this embodiment also provides a multimodal sentiment analysis system, including: The data acquisition module is used to acquire text, audio and visual multimodal input signals of the target object's emotional expression, extract heterogeneous feature sequences, and use an alignment algorithm based on sequence truncation and padding to map the heterogeneous feature sequences into a temporal feature matrix with the same time dimension. The capacity-aware routing module is used to route data of different modalities to a backbone network with matching capacity based on the feature dimensions of the temporal feature matrix of each modality, so as to extract the single-modal high-level semantic representations corresponding to the data of different modalities. The concatenation module is used to concatenate the single-modal high-level semantic representations corresponding to different modal data along the feature dimension, and input the fused features obtained by concatenation into a pre-trained multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix. The decision module is used to binarize or multi-class activation of the sentiment prediction probability distribution or logits matrix based on the optimal decision threshold, and generate corresponding sentiment classification labels according to the decision results to output the final sentiment analysis results; wherein, the optimal decision threshold is obtained by optimization on the validation set based on the τ threshold optimization inference engine.

[0076] The present invention also provides a multimodal sentiment analysis device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the multimodal sentiment analysis method.

[0077] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the multimodal sentiment analysis method.

[0078] When the processor executes the computer program, it implements the steps of the multimodal sentiment analysis described above, for example: The text, audio, and visual multimodal input signals of the target object's emotional expression are acquired, heterogeneous feature sequences are extracted, and an alignment algorithm based on sequence truncation and padding is used to map the heterogeneous feature sequences into temporal feature matrices of the same time dimension. Based on the feature dimensions of the temporal feature matrices of each modality, data of different modalities are routed to a backbone network with matching capacity in order to extract the single-modal high-level semantic representations corresponding to the data of different modalities. The single-modal high-level semantic representations corresponding to different modal data are concatenated along the feature dimension. The fused features obtained by concatenation are input into a pre-trained multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix. Based on the optimal decision threshold, the sentiment prediction probability distribution or logits matrix is ​​binarized or multi-class activation is determined, and the corresponding sentiment classification label is generated according to the determination result to output the final sentiment analysis result; wherein, the optimal decision threshold is obtained by optimizing the inference engine on the validation set based on the τ threshold.

[0079] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing preset functions, the instruction segments describing the execution process of the computer program in the multimodal emotion analysis device. For example, the computer program can be divided into a data acquisition module, a capacity-aware routing module, a concatenation module, and a decision module; the specific functions are as follows: The data acquisition module is used to acquire text, audio, and visual multimodal input signals of the target object's emotional expression, extract heterogeneous feature sequences, and use an alignment algorithm based on sequence truncation and padding to map the heterogeneous feature sequences into temporal feature matrices of the same time dimension; The capacity-aware routing module is used to route different modal data to a backbone network with matching capacity based on the feature dimension of each modal temporal feature matrix, so as to extract the single-modal high-level semantic representations corresponding to different modal data; The concatenation module is used to concatenate the single-modal high-level semantic representations corresponding to different modal data along the feature dimension, and input the fused features obtained by concatenation into a pre-trained multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix; The decision module is used to binarize or perform multi-class activation determination on the sentiment prediction probability distribution or logits matrix based on the optimal decision threshold, generate corresponding sentiment classification labels according to the determination results, and output the final sentiment analysis results; wherein, the optimal decision threshold is obtained by optimization on the validation set based on the τ threshold optimization inference engine.

[0080] The multimodal sentiment analysis device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The multimodal sentiment analysis device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above are examples of multimodal sentiment analysis devices and do not constitute a limitation on them. The device may include more components than described above, or combine certain components, or use different components. For example, the multimodal sentiment analysis device may also include input / output devices, network access devices, buses, etc.

[0081] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or any conventional processor. The processor is the control center of the multimodal sentiment analysis, connecting various parts of the multimodal sentiment analysis device via various interfaces and lines.

[0082] The memory can be used to store the computer program and / or modules. The processor implements various functions of the multimodal emotion analysis device by running or executing the computer program and / or modules stored in the memory and by calling the data stored in the memory.

[0083] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.). The data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0084] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multimodal sentiment analysis method described above.

[0085] If the modules / units integrated in the multimodal sentiment analysis system are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0086] Based on this understanding, the present invention can implement all or part of the processes in the above-described multimodal sentiment analysis method, or it can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above-described multimodal sentiment analysis method. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or a preset intermediate form, etc.

[0087] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0088] It should be noted that the content contained in the computer-readable storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0089] Compared with existing sentiment analysis methods, this invention provides a multimodal sentiment analysis method with the following advantages: This invention addresses three major pain points in the current field of multimodal sentiment analysis: the bottleneck of global attention computation, overfitting due to poor modality noise, and the static decision-making defects of long-tail data. It proposes a multimodal sentiment analysis method based on capacity-aware routing and a state-space model. This embodiment employs a pioneering "capacity-aware routing mechanism," assigning a lightweight perceptron for strong regularization and degradation to non-linguistic modalities with feature dimensions not exceeding a preset threshold, and assigning a bidirectional / bidirectional linear complexity state-space model for deep decoupling modeling to high-dimensional modalities, fundamentally solving the overfitting problem caused by modality noise. Simultaneously, it combines Focal Loss with... The threshold-optimized inference engine breaks through the bottleneck of long-tail recognition under extremely imbalanced data. Comprehensive experiments on three recognized international benchmark datasets (CH-SIMS, CMU-MOSI, CMU-MOSEI) show that this invention, with an extremely small computational cost of only 14.2M parameters, not only meets the industrial-grade real-time processing requirements with ultra-fast inference of 400+ FPS, but also comprehensively surpasses existing giant multimodal models with more than ten times the number of parameters in terms of recognition accuracy and Macro-F1 index.

[0090] The above embodiments are merely one of the implementation methods for achieving the technical solution of the present invention. The scope of protection claimed by the present invention is not limited to this embodiment, but also includes any variations, substitutions and other implementation methods that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that, include: The text, audio, and visual multimodal input signals of the target object's emotional expression are acquired, heterogeneous feature sequences are extracted, and an alignment algorithm based on sequence truncation and padding is used to map the heterogeneous feature sequences into temporal feature matrices of the same time dimension. Based on the feature dimensions of the temporal feature matrices of each modality, data of different modalities are routed to a backbone network with matching capacity in order to extract the single-modal high-level semantic representations corresponding to the data of different modalities. The single-modal high-level semantic representations corresponding to different modal data are concatenated along the feature dimension. The fused features obtained by concatenation are input into a pre-trained multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix. Based on the optimal decision threshold, the sentiment prediction probability distribution or logits matrix is ​​binarized or multi-class activation is determined, and the corresponding sentiment classification label is generated according to the determination result to output the final sentiment analysis result; wherein, the optimal decision threshold is obtained by optimizing the inference engine on the validation set based on the τ threshold.

2. The multimodal sentiment analysis method according to claim 1, characterized in that, The process of acquiring text, audio, and visual multimodal input signals of the target object's emotional expression, extracting heterogeneous feature sequences, and mapping these sequences to a temporal feature matrix with the same time dimension using an alignment algorithm based on sequence truncation and padding includes: Acquire text, audio, and visual multimodal input signals of the target object's emotional expression; The initial multimodal feature set is constructed from text, audio, and visual multimodal input signals; During the model training phase, a neutral bandwidth filtering threshold is introduced to filter out noisy samples in the initial multimodal feature set. The filtering principle is as follows: In the formula, This indicates the filtered sample; Indicates the original continuous sentiment label; Indicates the neutral bandwidth filtering threshold; After filtering, the neutral range The samples are discarded, resulting in truncated heterogeneous feature sequences; After aligning the truncated heterogeneous feature sequences, a temporal feature matrix with the same time dimension is obtained.

3. The multimodal sentiment analysis method according to claim 1, characterized in that, Based on the feature dimensions of each modality's temporal feature matrix, different modal data are routed to a backbone network with matching capacity to extract single-modal high-level semantic representations corresponding to different modal data, including: The text modal data is routed to a deep temporal network built on a bidirectional state-space model to obtain the first unimodal high-level semantic representation. Non-linguistic modal data with a feature dimension not greater than a preset threshold are routed to a multilayer perceptron projection network to obtain a second single-modal high-level semantic representation. Audio modal data with feature dimensions greater than a preset threshold are routed to a pre-constructed time-frequency dual-path state space model, and visual modal data with feature dimensions greater than a preset threshold are routed to a pre-constructed spatiotemporal dual-path state space model. Joint modeling is performed on both models to obtain a third single-modal high-level semantic representation.

4. The multimodal sentiment analysis method according to claim 3, characterized in that, The step of routing text modal data to a deep temporal network constructed based on a bidirectional state-space model to obtain a first unimodal high-level semantic representation includes: A deep temporal network employing a bidirectional continuous-time state-space model is used to discretize and model text modal data, where: For a given discretization step size State transition matrix and input matrix After zero-order preservation discretization, it can be represented as: in, It is the identity matrix; This is the discretized state transition matrix; The input matrix after discretization; It is a matrix exponential function; The specific operations for forward state update and output mapping are as follows: In the formula, This is the updated forward hidden state vector; This is the forward hidden state vector of the previous time step; This represents the text modal data at the i-th time step; This is the output matrix; Forward output features; The reverse output features of the reverse state are calculated in the same way as the forward state update and output mapping. ; The forward output features and the backward output features are added together to obtain the dynamic hidden state sequence of the text modality data; Global average pooling is performed along the time dimension on the dynamic hidden state sequence to extract the global semantic representation. The global semantic representation is output as the first unimodal high-level semantic representation, and the specific formula is as follows: In the formula, This represents the total time step. For feature space; To unify the feature dimensions of the hidden layer; The step of routing non-linguistic modal data with a feature dimension not greater than a preset threshold to a multilayer perceptron projection network to obtain a second unimodal high-level semantic representation includes: When the feature dimension of the non-linguistic modality data is not greater than a preset threshold, the temporal state update is automatically cut off, and the corresponding non-linguistic modality data is routed to a multilayer perceptron projection network. High-frequency noise is filtered out using linear projection and nonlinear activation to obtain a regularized global feature f. The regularized global feature is then output as the second unimodal high-level semantic representation, as shown in the following formula: In the formula, This is the time step mean vector of the original sequence; , All are learnable weight matrices; It is the bias vector; Use the GELU activation function; For Dropout regularization operations; The process of routing audio modal data with feature dimensions greater than a preset threshold to a pre-constructed time-frequency dual-path state space model, and routing visual modal data with feature dimensions greater than a preset threshold to a pre-constructed spatiotemporal dual-path state space model, and performing joint modeling to obtain a third single-modal high-level semantic representation, includes: When the feature dimension of audio modal data or visual modal data exceeds a preset threshold, a parallel state-space model branch is activated for deep decoupling modeling, where: The audio modal data is divided into a time domain matrix and a frequency domain matrix. The time domain matrix and the frequency domain matrix are then fused with dynamic gated features in the time domain and frequency domain to obtain the first fused gated feature. Visual modal data are decoupled into spatial static features and temporal dynamic changes using the same dual-path mechanism. The spatial static features and temporal dynamic changes are then processed by a dual-path state space module and gating aggregation to obtain the second fused gating feature. The first fusion-gated feature and the second fusion-gated feature are respectively used as the output of the single-modal high-level semantic representation of the corresponding modality.

5. The multimodal sentiment analysis method according to claim 1, characterized in that, The step of cascading the single-modal high-level semantic representations corresponding to different modal data along the feature dimension, and inputting the fused features obtained from the cascaded model into a multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix includes: The high-level semantic representations of single modalities corresponding to different modalities are concatenated along the feature dimension to obtain fused features. The specific formula is as follows: In the formula, , , These represent the single-modal high-level semantics corresponding to different modal data; Indicates a splicing operation; The fused features are input into a pre-trained multimodal linear classifier to obtain a fused classification head, which yields the sentiment prediction probability distribution or logits matrix, represented as: In the formula, For classification weights, Probability of positive sentiment prediction; This is the Sigmoid activation function.

6. The multimodal sentiment analysis method according to claim 5, characterized in that, The multimodal linear classifier calculates the network error based on a focal loss function with class weights. To complete the training, the specific formula is as follows: In the formula, This is the category weight matrix; For focusing parameters; This is a real label.

7. The multimodal sentiment analysis method according to claim 1, characterized in that, The optimal decision threshold is obtained by optimizing the inference engine on the validation set based on the τ threshold, including: The τ-dynamic optimization inference engine performs optimization operations on the validation set to obtain the optimal decision threshold, including: After the validation set inference is completed, an optimization function is constructed with the macro average F1 score as the sole objective. Perform a fine-grained sliding window search within the continuous interval [0,1] to solve for the dynamic decision boundary τ that maximizes the macro-average F1 score of the current fusion distribution. : in, and These represent the threshold values ​​respectively. Next category Precision and recall rates; The dynamic decision boundary obtained through the optimization operation is output as the optimal decision threshold. Among them, if If the result is positive, it is considered a positive sentiment sample; otherwise, it is considered a negative sentiment sample.

8. A multimodal sentiment analysis system, characterized in that, include: The data acquisition module is used to acquire text, audio and visual multimodal input signals of the target object's emotional expression, extract heterogeneous feature sequences, and use an alignment algorithm based on sequence truncation and padding to map the heterogeneous feature sequences into a temporal feature matrix with the same time dimension. The capacity-aware routing module is used to route data of different modalities to a backbone network with matching capacity based on the feature dimensions of the temporal feature matrix of each modality, so as to extract the single-modal high-level semantic representations corresponding to the data of different modalities. The concatenation module is used to concatenate the single-modal high-level semantic representations corresponding to different modal data along the feature dimension, and input the fused features obtained by concatenation into a pre-trained multimodal linear classifier to obtain the sentiment prediction probability distribution or logits matrix. The decision module is used to binarize or multi-class activation of the sentiment prediction probability distribution or logits matrix based on the optimal decision threshold, and generate corresponding sentiment classification labels according to the decision results to output the final sentiment analysis results; wherein, the optimal decision threshold is obtained by optimization on the validation set based on the τ threshold optimization inference engine.

9. A multimodal sentiment analysis device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the multimodal sentiment analysis method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it is used to implement the multimodal sentiment analysis method according to any one of claims 1-7.

Images