A multi-modal sentiment analysis method based on frequency-aware multi-stage fusion

Abstract

The application relates to the technical field of sentiment analysis, and discloses a multi-modal sentiment analysis method based on frequency perception multi-stage fusion, which comprises the following steps: decoupling the features of input text, audio and visual modalities to obtain shared features and private features of each modality; performing frequency domain decomposition on the private features to obtain high-frequency components and low-frequency components; using a frequency perception expert module to process the high-frequency components and the low-frequency components respectively to obtain enhanced frequency domain representations; using a modality interaction module to perform cross-modality fusion on the enhanced modality representations to obtain fused multi-modal representations; and using a hierarchical prediction module to generate a final sentiment prediction result based on the fused multi-modal representations. The application combines frequency perception, feature decoupling and multi-stage interaction fusion organically, and comprehensively improves the performance, robustness and explainability in a multi-modal sentiment analysis task.

Description

Technical Field

[0001] This invention relates to the field of sentiment analysis technology, and more specifically, to a multimodal sentiment analysis method based on frequency-aware multi-stage fusion. Background Technology

[0002] With the rapid proliferation of social media platforms, users are increasingly expressing emotions through multiple means, including text, audio, and video. This has spurred the rise of multimodal sentiment analysis (MSA), which aims to infer human emotional states by modeling heterogeneous multimodal data sources. MSA has shown broad application prospects in fields such as healthcare, social media monitoring, and human-computer interaction. Compared to unimodal methods, MSA offers stronger robustness and richer semantic understanding by utilizing complementary cross-modal cues.

[0003] Early research primarily relied on textual data, but the increasing prevalence of user-generated videos has shifted the focus to fully leveraging multimodal representations. Recent studies have explored representation learning to enhance the expressiveness and decoupling capabilities of modality-shared and modality-specific features, employing strategies such as shared-private decomposition, contrastive learning, and multi-task learning. Simultaneously, fusion strategies have evolved from early splicing to more expressive mechanisms, including tensor-based fusion, convolutional and recurrent architectures, and cross-modal attention networks. However, most existing methods neglect the inherent frequency characteristics within modalities: audio signals exhibit rich temporal variations across frequency bands, while visual and textual features typically contain low-frequency semantic trends interspersed with sparse high-frequency fluctuations. Directly modeling these heterogeneous frequency patterns in the original feature space can dilute task-relevant information and obscure subtle but crucial modal dynamics.

[0004] Multimodal fusion aims to construct high-quality joint representations through efficient cross-modal integration. Despite significant progress, existing methods still struggle to model the complexity and heterogeneity of cross-modal interactions. Most methods emphasize atomic-level dependencies while neglecting compositional and hierarchical semantic structures. Attention-based and graph-structured models can capture local dependencies but lack mechanisms for modeling cross-modal hierarchical relationships. Furthermore, modal encoders often introduce redundant or irrelevant content—especially in visual streams—leading to high noise in fused representations. While high-capacity tensor or multinomial fusion frameworks offer richer expressive power, they suffer from excessive computational cost and poor scalability. Even recent approaches employing mutual information maximization and top-down feedback have failed to effectively capture bidirectional semantic streams.

[0005] In general, current fusion strategies are insufficient in modeling the heterogeneity, structure, and hierarchy of multimodal expressions, thus necessitating a more flexible and interpretable fusion mechanism to capture both fine-grained and coarse-grained intermodal relationships. Recent research has further explored dynamic weighting strategies to adaptively adjust modality importance. In contrast, this invention proposes FEMF, which follows a widely accepted assumption that textual modalities provide the most reliable and fine-grained sentiment cues, and therefore serve as the dominant modality to guide multimodal fusion. Summary of the Invention

[0006] In view of this, the present invention proposes a multimodal sentiment analysis method based on frequency-aware multi-stage fusion to solve the problems existing in the prior art.

[0007] To achieve the above objectives, this invention proposes a multimodal sentiment analysis method based on frequency-aware multi-stage fusion, characterized by the following steps: Feature decoupling is performed on the modal features of the input text, audio, and visual data to obtain the shared and private features of each modality; For each modality's private features, frequency domain decomposition is performed to obtain high-frequency and low-frequency components; The high-frequency and low-frequency components are processed separately using a frequency-aware expert module to obtain an enhanced frequency domain representation; A modal interaction module is used to perform cross-modal fusion on the enhanced modal representation to obtain a fused multimodal representation; Based on the fused multimodal representation, the final sentiment prediction result is generated using a hierarchical prediction module.

[0008] Furthermore, the process of decoupling the modal features of the input text, audio, and visual data includes: A shared encoder is used to extract shared features for each modality. A modality-specific private encoder is used to extract private features for each modality feature; A similarity loss function is used to constrain the distribution consistency of shared features among different modalities; Orthogonal loss functions are used to constrain the non-redundancy between shared and private features; Improving the semantic consistency and reconstruction quality of representations by using reconstruction loss functions and cycle consistency loss functions.

[0009] Furthermore, the process of frequency domain decomposition of the private features of each modality includes: For each mode's private features, a discrete Fourier transform is applied along the time dimension to convert the signal to the frequency domain; The spectrum is decomposed into low-frequency and high-frequency components according to the preset frequency cutoff coefficient. Each of the low-frequency components and the high-frequency components is processed by a dedicated frequency-specific mode expert, which consists of a multi-layer Transformer encoder.

[0010] Furthermore, the frequency-aware expert module further includes: A soft router is used, with the current modal features and text modal features as input, to dynamically calculate the weights of each expert's output; By weighted fusion of the expert outputs, the low-frequency and high-frequency information features of each modality are obtained; Hyperparameters are used to control the contribution ratio of the low-frequency information features and the high-frequency information features in the final representation.

[0011] Furthermore, the process of cross-modal fusion of the enhanced modal representation using a modal interaction module includes: A multi-head cross-attention mechanism is adopted to perform pairwise fusion of different modalities in stages; Using the target modality as the query, attention interactions are sequentially performed with the other modalities. The above fusion process is repeated in a multi-layer structure, and the fusion features are refined using a feedforward network and layer normalization.

[0012] Furthermore, the hierarchical prediction module includes: Perform modality-specific predictions for each fused multimodal representation; The multimodal representations are concatenated and then jointly predicted; Adaptive weighted fusion of modality-specific predictions and joint predictions is performed using learnable weights; Introducing consistency constraints on shared representations improves the robustness and interpretability of predictions.

[0013] Furthermore, the overall learning objective of the method is constituted by the following loss functions: Feature decoupling losses include similarity loss, orthogonality loss, reconstruction loss, and cycle consistency loss; Frequency-aware expert training loss, including prediction errors on the outputs of each expert; Consistency alignment loss between shared representation and final prediction; The main loss function for multimodal sentiment analysis tasks.

[0014] Furthermore, after training is completed, the parameters of all frequency-aware experts are frozen and reused during the inference phase.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention decomposes modal features into high-frequency and low-frequency components using Discrete Fourier Transform, and processes them separately by a dedicated expert network. This allows the model to simultaneously capture stable sentiment trends and instantaneous sentiment fluctuations, effectively solving the problem of semantic information dilution caused by neglecting the internal frequency characteristics of modalities in traditional methods. Experiments on the CMU-MOSI and CMU-MOSEI benchmark datasets show that this method outperforms existing mainstream models in key metrics, achieving more accurate sentiment prediction.

[0016] The shared-private feature decoupling mechanism employed in this invention, combined with hierarchical prediction and adaptive fusion strategies, enables the model to exhibit stronger robustness when faced with real-world scenarios involving partial modality loss. The model can dynamically adjust its dependence on different modalities and prediction heads, prioritizing the trust of more reliable information sources. Consequently, even with a modality loss rate as high as 40%, the performance degradation is significantly less than that of traditional methods.

[0017] This invention employs a multi-stage, paired cross-modal attention interaction mechanism, overcoming the shortcomings of traditional fusion methods in modeling hierarchical semantic structures. This mechanism can explicitly and progressively integrate complementary cues between different modalities to form a high-quality joint representation. Simultaneously, the attention patterns of frequency experts towards signals in different frequency bands can be clearly visualized, enhancing the transparency and interpretability of the model's decision-making process. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a schematic diagram of the overall process of a multimodal sentiment analysis method based on frequency-aware multi-stage fusion proposed in this invention; Figure 2 This is a schematic diagram of the overall structure of the FEMF frame in an embodiment of the present invention; Figure 3 This is a schematic diagram of the frequency sensing MoE module structure in an embodiment of the present invention; Figure 4 This is a schematic diagram of the modal interaction module structure in an embodiment of the present invention; Figure 5 This is a schematic diagram of parameter sensitivity analysis in an embodiment of the present invention; Figure 6 This is a schematic diagram of modal interaction sequence analysis on MOSI in an embodiment of the present invention; Figure 7 This is a schematic diagram of robustness analysis under missing modes in an embodiment of the present invention; Figure 8This is to visualize the attention distribution of the low-frequency and high-frequency branches in the three modalities in the embodiments of the present invention. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] This embodiment proposes a multimodal sentiment analysis method based on frequency-aware multi-stage fusion, such as... Figure 1 As shown, it includes the following steps: Feature decoupling is performed on the modal features of the input text, audio, and visual data to obtain the shared and private features of each modality; For each modality's private features, frequency domain decomposition is performed to obtain high-frequency and low-frequency components; The high-frequency and low-frequency components are processed separately using a frequency-aware expert module to obtain an enhanced frequency domain representation; A modal interaction module is used to perform cross-modal fusion on the enhanced modal representation to obtain a fused multimodal representation; Based on the fused multimodal representation, the final sentiment prediction result is generated using a hierarchical prediction module.

[0021] Specifically, this embodiment proposes a frequency-aware expert-based multi-stage fusion framework, FEMF, for multimodal sentiment analysis. FEMF is inspired by the observation that different frequency bands encode different sentiment cues—high-frequency components capture sudden shifts in sentiment, while low-frequency components reflect stable sentiment trends. FEMF decomposes modality-specific features into high-frequency and low-frequency components using the Discrete Fourier Transform (DFT). These components are processed by expert personnel and adaptively fused to form enhanced representations. This frequency-aware design preserves both global sentiment consistency and transient changes, thus mitigating semantic ambiguity caused by neglecting frequency cues. Furthermore, FEMF integrates shared-private information decoupling guided by cross-modal reconstruction to enhance multimodal representation learning and promote robust fusion.

[0022] Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmark datasets demonstrate the effectiveness and robustness of the proposed method. The results validate the practicality of frequency-aware expert modeling and highlight its potential for interpretable and robust multimodal sentiment analysis.

[0023] In the MSA task, the input data consists of three modalities: text (t), audio (a), and visual (v). Multimodal discourse is represented by triples. It means that among them Representing modes sequence features, Represents the sequence length. This refers to the feature dimension. Audio, text, and visual features are extracted using pre-trained wav2vec-large, DeBERTa-large, and MA-Net (after face alignment via MTCNN), respectively. The prediction target is the sentiment score. The values ​​greater than 0, equal to 0, and less than 0 correspond to positive, neutral, and negative emotions, respectively.

[0024] Feature decoupling module Each modality-specific representation obtained from the pre-trained encoder Both are further decomposed into two complementary components. Modality-shared representations encode information in a common latent space while enforcing cross-modal distributional similarity, thus mitigating heterogeneity differences and enabling more efficient fusion. In contrast, modality-private representations preserve modality-specific features crucial for capturing unique informational cues. These two components are jointly optimized within a unified framework to balance cross-modal consistency and modality uniqueness.

[0025] like Figure 2 As shown, this embodiment uses a shared multimodal encoder. and three independent single-mode encoders ,in Let's learn the mapping representation, defined as follows:

[0026] Shared multimodal encoder and three independent single-mode encoders It consists of linear layers.

[0027] To reduce the differences among the shared representations of the three modalities, this embodiment uses cosine similarity to quantify the differences between their respective distributions, denoted as . Therefore, in the shared representation space of all modalities, the objective of this embodiment is to minimize the following function:

[0028]

[0029] Furthermore, an orthogonal loss function is introduced to ensure that the modality-shared representation and the modality-private representation focus on different attributes of the input data, respectively. This non-redundancy is achieved by imposing orthogonal constraints between the two representations. The corresponding orthogonal loss function is calculated as follows:

[0030] This indicates orthogonal operations. Although introduced... While this approach helps to enforce the separation of shared and private representations, the risk remains that modality-specific encoders may learn trivial or meaningless solutions. To address this issue, this embodiment introduces a modality reconstruction loss, which encourages hidden representations to retain key information from each modality and accurately reflect their respective characteristics.

[0031] Since the text modality has the highest confidence level among the three modalities, this embodiment represents a specific modality for each modality. Modality sharing representation with text modality Connect them together as a single decoder The input, where The corresponding representation is reconstructed. The difference between the original multimodal representation and its reconstructed counterpart is quantified as follows:

[0032]

[0033] To ensure that the reconstructed modal representation retains the sentiment semantic information of the original modality, this embodiment introduces a sentiment consistency constraint mechanism to promote consistency in sentiment semantics between the original representation and its reconstructed version. Inspired by the concept of cycle consistency in the generative model domain, we adopt a cycle consistency loss function. To improve the quality and robustness of representation reconstruction.

[0034] Specifically, the reconstructed modal representation is input into its corresponding dedicated encoder for re-encoding, thereby recovering the unique representation of its original modality. The difference between the reconstructed representation and the unique representation is defined as the cycle consistency loss, as shown in the following equation:

[0035] in, This represents the reconstructed modal representation. Representing modes Private encoder, Representing modes The private representation of the text modality. This loss term effectively ensures that the reconstructed path is semantically consistent with the original modality, thereby enhancing the model's ability to preserve sentiment semantics. Furthermore, since the reconstruction of the text modality only occurs within that modality, we do not use this loss term to constrain the text modality.

[0036] Finally, this embodiment combines the above constraints to construct the overall loss function of the module, which is defined as follows:

[0037] This work uses private features from each modality as the primary input to subsequent modules. Shared features are mainly used for alignment and supervision, while private features contain more modality-specific information, such as intonation, visual details, or differences in language style. Using private features as input to subsequent modules enhances the complementary representation between modalities, improves the model's ability to model fine-grained sentiment differences, and thus improves the performance and robustness of the final sentiment analysis.

[0038] Frequency Domain Awareness Expert Module To further enhance the modeling capability of each modality representation, this embodiment decomposes each modality representation into low-frequency and high-frequency components, such as... Figure 3 As shown. This design is inspired by the observation that emotional semantics may simultaneously exhibit global (low-frequency) trends and sudden local changes (high-frequency cues).

[0039] Formally, this embodiment applies a real-valued Discrete Fourier Transform (DFT) along the time dimension to project the signal into the frequency domain. The Discrete Fourier Transform (DFT) is a fundamental tool in digital signal processing (DSP) that transforms discrete-time signals from the time domain to the frequency domain. This embodiment represents the DFT operation as a linear mapping: Its inverse transform (called the inverse discrete Fourier transform (IDFT)) is expressed as: .Will Applied to real-valued sequences Equivalent to Multiply by DFT matrix Each row of this matrix is ​​a Fourier basis vector. Defined as:

[0040] in It is the imaginary unit, and Let the signal spectrum be: This embodiment decomposes the spectrum into low-frequency components and high-frequency components: the low-frequency component... and high frequency part This decomposition is achieved through hyperparameters in the FourierLayer. Implemented, and independently applied to each modality's private representation:

[0041] Each frequency band obtained from the previous module is processed by a dedicated frequency-specific mode expert. Each expert is... It consists of stacked Transformer encoder modules—each module containing a multi-head self-attention layer and a feedforward network, followed by residual connections and layer normalization. Unless otherwise stated, =4.

[0042]

[0043]

[0044] The training objective for each expert is to minimize the mean squared error (MSE) between their predictions and the true values:

[0045] After training, the parameters of all modality frequency experts are frozen and reused in downstream models. This design ensures that each expert can capture the discriminative patterns specific to the stable (low-frequency) or dynamic (high-frequency) components of modality-specific emotional signals. Through this process, each modality frequency expert can reinterpret the input representation from its own modal perspective and integrate complementary cues learned from other modalities, thereby forming a unified cross-modal understanding. Given the low-frequency components of the m-th modality... Input it into each expert In order to obtain modality expert-specific embeddings, as shown below:

[0046] here, Indicates corresponding to the first Each modality (audio, visual, or text) has an expert. This process allows each expert to reinterpret the input features from their own modality perspective, thereby generating a set of cross-expert representations. These representations capture complementary knowledge from multiple modalities.

[0047] To further enhance the expressive power of the representation, this embodiment introduces a lightweight soft router, which is a two-layer multilayer perceptron (MLP). Since text modalities have empirical reliability in sentiment understanding, they are treated as high-confidence semantic anchors. The router takes the current modality features and text features as input to estimate the contribution scores of all experts, thereby achieving dynamic optimization and rebalancing of contributions from different experts (taking low-frequency signals as an example):

[0048]

[0049]

[0050] That It represents the weight of each expert's output. It is modal The low-frequency information characteristics. Similarly, after expert processing, a pattern can be obtained. Representation of high-frequency information .

[0051] Subsequently, this embodiment introduces hyperparameters. To control the contributions of high-frequency and low-frequency components, they are fused by weighted summation as follows:

[0052] Unlike traditional feature segmentation methods that arbitrarily divide latent dimensions, this embodiment uses a Discrete Fourier Transform (DFT) decomposition method to perform signal-level separation based on the time-frequency domain characteristics of multimodal data. Through DFT, each modality is decomposed into low-frequency components representing stable contextual trends and high-frequency components capturing transient emotional changes. This principle-based spectral separation allows frequency experts to model complementary dynamic processes based on interpretable frequency semantics, rather than arbitrary feature subsets. The resulting frequency-aware representation is then fed into a cross-modal attention mechanism and a hierarchical prediction layer to enhance semantic alignment and decision robustness.

[0053] Modal interaction module After generating a modality-specific representation refined by a frequency-aware expert mechanism, this embodiment introduces a fusion module based on a multi-head attention mechanism (e.g., Figure 4 As shown, this module explicitly models the interactions between different modalities in a pairwise and sequential manner. The mechanism uses stacked bidirectional multi-head attention layers to iteratively align each target modality with complementary information from the remaining modalities.

[0054] For each modality, the fusion process is performed in two phases. First, the target modality uses its own features as a query and the target modality's features as key-value pairs to focus on the target modality. Then, the resulting features are fused with the third modality in the same way. This phased attention mechanism allows each modality to progressively integrate semantic cues from other modalities. This process is applied symmetrically to all modalities and is stacked... Repeated execution at higher layers facilitates deeper, more structured cross-modal interactions. Unless otherwise stated, It is considered an adjustable hyperparameter.

[0055] Taking text modality as an example, let Indicates the first The text representation of the layer, let These represent audio and visual modal features, respectively. The layer fusion process includes: First, text features are used as query vectors, and audio features are used as keys and values ​​in a multi-head cross-attention module to model text-audio interaction.

[0056]

[0057]

[0058] in This is the projection matrix.

[0059] Regarding the visual fusion, similar to the previous step, the fused text-audio features are used as queries, while the visual features are used as keys and values ​​in the second cross-attention module to model the text-visual interaction, ultimately yielding the fused features. .

[0060] A feedforward network (FFN) with residual connections and layer normalization is used to refine the fused features and achieve a deeper level of representation.

[0061]

[0062]

[0063] in, For the projection matrix, This is the bias parameter.

[0064] Hierarchical prediction module To further enhance the discriminative power of each modality and the robustness of the final decision, this embodiment designs a hierarchical prediction module that integrates modality-specific prediction and multimodal fusion prediction.

[0065] Specifically, for each refined modal representation ,in A two-layer feedforward network with residual connections is applied to obtain a high-level representation, and then modality-specific predictions are obtained by applying a sigmoid activation projection.

[0066] To leverage the synergistic effects between different modalities, this embodiment connects refined modal representations to form joint features and computes fusion-based predictions:

[0067] To achieve adaptive fusion of hierarchical predictions, this embodiment uses learnable fusion to compute the weights. Specifically, this embodiment employs a lightweight multilayer perceptron (MLP) to estimate the importance of each prediction head, including three modality-specific predictions and one joint fusion prediction. Given four prediction vectors, they are first concatenated into a combined vector, which is then input into an MLP. Finally, a softmax layer is used to generate a set of normalized weights. :

[0068] Final predicted value It is obtained by weighting and combining all the individual predicted values:

[0069] This design allows the model to adaptively emphasize more reliable specific modes or joint outputs under different input conditions.

[0070] Furthermore, this embodiment aggregates the modal shared representations obtained from the decoupling module (these representations are not directly used in subsequent prediction steps) and uses them to generate an auxiliary prediction. Then, consistency constraints are applied to this auxiliary prediction and the final output.

[0071]

[0072]

[0073] This constraint encourages shared representations to retain task-related semantics and to keep their predictive behavior consistent with the final decision, thereby improving the robustness and interpretability of the shared feature space.

[0074] Overall learning objectives Final MSA output loss Defined as:

[0075] in For MSA tags, The sample size. Total modal expert training loss. Defined as:

[0076] The proposed framework integrates decoupling loss Modal expert training loss Shared consistency loss Total MSA mission losses To form overall learning objectives:

[0077] Verification Experiment This experiment evaluates the proposed model on two widely used benchmark datasets: CMU-MOSI and CMU-MOSEI.

[0078] CMU-MOSI contains 2199 monologue video clips of movie reviews, each accompanied by audio and video streams at sampling rates of 12.5Hz and 15Hz, respectively. The dataset consists of 1284 training samples, 229 validation samples, and 686 test samples.

[0079] As a larger-scale extension, CMU-MOSEI contains 22,856 video clips collected from YouTube, with audio feature sampling at 20Hz and visual feature sampling at 15Hz. It includes 16,326 training samples, 1,871 validation samples, and 4,659 test samples. For both datasets, sentiment annotations range from -3 to +3, representing a fine-grained continuum from strong negative to strong positive sentiment.

[0080] Evaluation indicators Referring to previous research, this experiment employed a comprehensive set of evaluation metrics to assess model performance, including binary accuracy (Acc-2), F1 score, Pearson correlation coefficient (Corr) between predicted sentiment score and actual sentiment score, and mean absolute error (MAE). For binary accuracy, this experiment used… "The symbols report negative / non-negative accuracy and negative / positive accuracy, with the left-hand value representing the former and the right-hand value representing the latter, thus ensuring fair and detailed comparisons between different studies."

[0081] Experimental details All experiments were implemented using PyTorch and run on an NVIDIA RTX 3090 GPU with 24 GB of VRAM. Model training used a batch size of 64 and a hidden layer dimension of 256, consistent with the feature dimensions for modality specificity and cross-modal attention embeddings. The Adam optimizer with a learning rate of 0.0001 was used, and each experiment was run three times; average performance is reported.

[0082] To ensure fair comparison, all pre-trained encoders in FEMF are the same as those used in MoMKE: text features use DeBERTa-large, audio features use wav2vec-large, and visual features use MA-Net, thus ensuring complete consistency across modalities.

[0083] Baseline Self-MM performs joint unimodal and multimodal learning through a self-supervised labeling module. MMIM enhances fusion by maximizing cross-modal mutual information. CubeMLP models cross-modal interactions using a cube-structured MLP. ConFEDE applies contrastive decoupling and knowledge distillation to achieve robust fusion. ALMT introduces a language-guided adaptive module to filter irrelevant signals. TMSON models modal uncertainty through Gaussian fusion of ordinal sentiment spaces. HyDiscGAN employs a hybrid discriminator for global-local alignment. MoMKE integrates unimodal and joint features through a soft routing strategy. DEVA generates text sentiment descriptions from audiovisual cues to enhance semantic reasoning capabilities. MFMB-Net employs macro-micro fusion and modality reconstruction techniques to improve robustness.

[0084] Main Experiment As shown in Table 1, the proposed model achieves competitive or better results on most evaluation metrics of the CMU-MOSI and CMU-MOSEI datasets. Compared with previous state-of-the-art methods, FEMF significantly improves Pearson correlation coefficient, classification accuracy, and F1 score, indicating stronger consistency between predicted sentiment scores and actual sentiment scores. These improvements demonstrate that FEMF is better able to capture subtle dynamic changes in sentiment across multimodal inputs.

[0085] Under both negative / non-negative and negative / positive binary classification protocols, our model consistently achieves higher accuracy and F1 scores, demonstrating stronger discriminative power and robustness across datasets. Compared to mutual information or generative models such as Self-MM, MMIM, and HyDiscGAN, FEMF achieves more stable improvements in both relevance and classification metrics. While confidence-aware methods such as ConFEDE and TMSON enhance robustness through uncertainty modeling and ordinal representations, FEMF further strengthens representation learning by jointly modeling shared and modality-specific features under hierarchical supervision. Even compared to advanced semantic alignment or routing frameworks such as DEVA and MoMKE, FEMF maintains good overall performance, confirming its effectiveness in balancing fine-grained regression accuracy and robust classification in multimodal sentiment analysis.

[0086] ablation experiment Ablation of modules and losses To evaluate the contribution of each component, an ablation experiment was conducted in this embodiment, removing the Decoupling Module (DTM), Frequency-Aware Modal Evaluation Module (FAE), Modal Interaction Module (MIM), Hierarchical Prediction Module (HPM), and five auxiliary objective functions. , , , , (This step was repeated, while keeping the rest of the architecture unchanged.) The results are summarized in Table 2.

[0087] Removing the DTM leads to a significant drop in all metrics, confirming its role in generating informative and decoupled modality-specific representations. Excluding FAE (i.e., omitting frequency decomposition and expert selection) results in higher errors and weaker consistency, validating the advantages of frequency-aware modeling in capturing both stable and transient emotional cues. Removing the MIM significantly reduces accuracy and relevance, demonstrating that explicit cross-modal interaction modeling is crucial for robust fusion. Removing the HPM and using single-stage prediction also degrades performance, highlighting the value of hierarchical supervision in complementary unimodal-multimodal learning and improved generalization ability.

[0088] Table 2

[0089] Regarding secondary targets, remove This will weaken the consistency of shared representations, and omitting... or This will moderately reduce performance, highlighting their role in stable reconstruction and cross-modal consistency. Excluding... This will increase redundancy between shared and private spaces, while removing... This will slightly reduce accuracy, indicating its contribution to high-level relationship alignment.

[0090] In summary, these findings confirm that building components and auxiliary losses make unique and complementary contributions to the final performance of the model, validating the necessity and effectiveness of a complete FEMF design.

[0091] Ablation of different modes Table 3 compares the performance of different modality combinations on the CMU-MOSI and CMU-MOSEI datasets, where T = text modality, A = audio modality, and V = visual modality. As shown in Table 3, the text modality exhibits the strongest single-modal performance, significantly outperforming both visual and auditory modalities, highlighting the rich semantic information inherent in language for sentiment understanding. In the bimodal setting, any combination of two modalities consistently improves performance, outperforming the single-modal baseline model, demonstrating the complementarity of multimodal cues. Furthermore, the trimodal configuration achieves the best overall results in both benchmarks, confirming that our framework effectively utilizes heterogeneous but complementary information. The progressive improvement from single-modal to bimodal to trimodal fusion demonstrates the robustness of our multimodal fusion strategy in capturing cross-modal dependencies and enhancing fine-grained sentiment discrimination.

[0092] Table 3

[0093] Parameter sensitivity experiment like Figure 5 As shown, this embodiment performs sensitivity analysis on three key parameters: the number of layers in the modal interaction module, the fusion weight in the frequency-aware hybrid expert (MoE) branch, and the fusion weight in the frequency-aware hybrid expert (MoE) branch. and frequency cutoff coefficient —To examine their impact on the performance of CMU-MOSI and CMU-MOSEI models.

[0094] For the number of layers ( Figure 5 (a) CMU-MOSI performs best with a single-layer architecture, while deeper architectures lead to a slight performance drop, which may be due to overfitting or redundant modeling. CMU-MOSEI, due to its larger data scale, exhibits higher stability at different layer depths.

[0095] Change the fusion coefficient within the range of [0.1, 0.9]. ( Figure 5 (b) shows that the optimal value is approximately 0.3, at which point assigning appropriate weights to high-frequency components produces a more discriminative representation. Extreme values ​​degrade performance, confirming the importance of equalized frequency fusion.

[0096] Finally, the cutoff coefficient ( Figure 5 (c) determines the cutoff frequency between the low-frequency and high-frequency components in the Fourier transform-based decomposition, with a peak value of [value missing] in CMU-MOSI. In CMU-MOSEI, the peak value is approximately This suggests that moderately preserving high-frequency information can enhance sensitivity to subtle emotional cues, while larger, more diverse datasets are less affected by specific cutoff values.

[0097] Modal interaction sequence analysis experiment To evaluate the impact of the modal interaction order within each query-dominant branch of the modal interaction module, this embodiment conducted an experiment to explore different fusion paths between modalities, such as... Figure 6 As shown.

[0098] For the text query branch, first interact with the audio modality in all... It performed best in all metrics, which suggests that early integration of temporally aligned acoustic cues can better refine the semantic foundation before integrating higher-level visual information.

[0099] Within the audio query branch, the optimal order is A→T→V, indicating that associating audio with linguistic information is more effective in enhancing emotional interpretation compared to using visual features that are not aligned with time.

[0100] For visual query branches, the V→T→A order is always superior to its reverse, suggesting that incorporating textual context early helps to eliminate ambiguity in visual signals before adding acoustic nuances.

[0101] Overall, these findings demonstrate that the order of modal interactions plays a crucial role. The proposed multi-branch interaction module benefits from a carefully designed fusion sequence, effectively capturing diverse and complementary cross-modal dependencies.

[0102] Robustness analysis experiment under missing modes To evaluate the robustness of the model to incomplete multimodal inputs, we conducted experiments in which input features were randomly masked with missing rates ranging from 0% to 40%, simulating real-world scenarios such as sensor failure or transmission noise.

[0103] like Figure 7 As shown, the performance of all models decreases with increasing missing rates, revealing the inherent difficulty of incomplete multimodal learning. However, compared to representative baseline models, our proposed model exhibits a significantly slower rate of decline in both F1 score and Pearson correlation coefficient, demonstrating stronger robustness against missing modalities. This robustness stems from our decoupled representation learning framework, which jointly captures complementary information from shared and modality-specific spaces; furthermore, it arises from our confidence-aware fusion and hierarchical prediction strategy, which adaptively emphasizes reliable modalities.

[0104] Furthermore, the robustness analysis also reflects the model's data generalization ability: even with a large number of missing modes, FEMF maintains stable performance, indicating its strong adaptability under conditions of limited data or small sample sizes. The frequency-aware multi-stage fusion mechanism further promotes cross-modal alignment and enhances the model's resistance to noise or missing signals, highlighting the practicality of this method in real-world multimodal scenarios.

[0105] Visualization Analysis Experiment To evaluate the effectiveness of the proposed Fourier perceptual expert model, this experiment visualized the attention distribution of high-frequency and low-frequency branches under different modalities. Figure 8 The results showed significant modality-specific differences: high-frequency components in audio focused on prosodic changes, while low-frequency components captured overall intonation; high-frequency branches in text focused on emotionally rich words, while low-frequency branches reflected stable semantic structures; high-frequency visual responses highlighted local edges and rapid facial changes, while low-frequency responses captured overall appearance and scene context.

[0106] These complementary patterns demonstrate that frequency-aware modeling can achieve adaptive and semantically meaningful attention allocation. To further elucidate the semantic meaning of frequency decomposition, this embodiment notes that high-frequency components capture transient and emotionally intense changes, such as sudden changes in speech or facial expression, while low-frequency components encode smoother, more stable emotional profiles, representing calm or sustained tone of voice. This observation aligns with the expected design of frequency-aware experts and supports the effectiveness of the proposed decomposition method.

[0107] In summary, this embodiment proposes a multimodal sentiment analysis framework called FEMF, which is based on a frequency-aware expert model and a multi-stage fusion technique. FEMF combines discrete Fourier transform for frequency decomposition and utilizes expert models to model high-frequency and low-frequency components, effectively capturing stable and transient sentiment cues across modalities. Furthermore, FEMF integrates shared-private decoupling, modal interaction modules, and hierarchical prediction mechanisms, further enhancing the model's robustness and interpretability. Extensive experiments on the CMU-MOSI and CMU-MOSEI datasets demonstrate that FEMF significantly improves upon other strong baseline models. Ablation experiments and robustness analysis also validate the contributions of each component and the robustness of the overall architecture.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal sentiment analysis method based on frequency-aware multi-stage fusion, characterized in that, Includes the following steps: Feature decoupling is performed on the modal features of the input text, audio, and visual data to obtain the shared and private features of each modality; For each modality's private features, frequency domain decomposition is performed to obtain high-frequency and low-frequency components; The high-frequency and low-frequency components are processed separately using a frequency-aware expert module to obtain an enhanced frequency domain representation; A modal interaction module is used to perform cross-modal fusion on the enhanced modal representation to obtain a fused multimodal representation; Based on the fused multimodal representation, the final sentiment prediction result is generated using a hierarchical prediction module.

2. The method according to claim 1, characterized in that, The process of decoupling the modal features of the input text, audio, and visual features includes: A shared encoder is used to extract shared features for each modality. A modality-specific private encoder is used to extract private features for each modality feature; A similarity loss function is used to constrain the distribution consistency of shared features among different modalities; Orthogonal loss functions are used to constrain the non-redundancy between shared and private features; Improving the semantic consistency and reconstruction quality of representations by using reconstruction loss functions and cycle consistency loss functions.

3. The method according to claim 1, characterized in that, The process of frequency domain decomposition of the private features of each modality includes: For each mode's private features, a discrete Fourier transform is applied along the time dimension to convert the signal to the frequency domain; The spectrum is decomposed into low-frequency and high-frequency components according to the preset frequency cutoff coefficient. Each of the low-frequency components and the high-frequency components is processed by a dedicated frequency-specific mode expert, which consists of a multi-layer Transformer encoder.

4. The method according to claim 1, characterized in that, The frequency sensing expert module further includes: A soft router is used, with the current modal features and text modal features as input, to dynamically calculate the weights of each expert's output; By weighted fusion of the expert outputs, the low-frequency and high-frequency information features of each modality are obtained; Hyperparameters are used to control the contribution ratio of the low-frequency information features and the high-frequency information features in the final representation.

5. The method according to claim 1, characterized in that, The process of cross-modal fusion of the enhanced modal representation using a modal interaction module includes: A multi-head cross-attention mechanism is adopted to perform pairwise fusion of different modalities in stages; Using the target modality as the query, attention interactions are sequentially performed with the other modalities. The above fusion process is repeated in a multi-layer structure, and the fusion features are refined using a feedforward network and layer normalization.

6. The method according to claim 1, characterized in that, The hierarchical prediction module includes: Perform modality-specific predictions for each fused multimodal representation; The multimodal representations are concatenated and then jointly predicted; Adaptive weighted fusion of modality-specific predictions and joint predictions is performed using learnable weights; Introducing consistency constraints on shared representations improves the robustness and interpretability of predictions.

7. The method according to claim 1, characterized in that, The overall learning objective of the method is comprised of the following loss functions: Feature decoupling losses include similarity loss, orthogonality loss, reconstruction loss, and cycle consistency loss; Frequency-aware expert training loss, including prediction errors on the outputs of each expert; Consistency alignment loss between shared representation and final prediction; The main loss function for multimodal sentiment analysis tasks.

8. The method according to claim 1, characterized in that, After training is complete, the parameters of all frequency-aware experts are frozen and reused during the inference phase.

Images