A cross-time hierarchical temporal memory network construction method for multi-modal sentiment analysis

Abstract

This invention discloses a method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis. The network includes a multimodal feature encoder, a hierarchical temporal memory network, a cross-temporal attention recall module, a gated fusion layer, a multimodal fusion and prediction head, and a multi-task joint training strategy. By constructing a three-layer differentiated memory architecture (short-term, medium-term, and long-term), it achieves dynamic hierarchical modeling of sentiment across multiple time scales. A cross-temporal attention recall mechanism is designed to adaptively retrieve historical sentiment context. A gated fusion layer dynamically balances current features and historical memory. Joint knowledge distillation and InfoNCE contrastive learning achieve pre-trained knowledge transfer and strong cross-modal alignment. This invention effectively solves problems such as insufficient fine-grained sentiment cue capture and weak temporal dynamic modeling. Its performance on the CMU-MOSI and CMU-MOSEI datasets significantly outperforms existing methods, making it suitable for scenarios such as natural human-computer interaction and intelligent sentiment recognition.

Description

Technical Field

[0001] This invention relates to the fields of sentiment computing and multimodal machine learning, specifically to a method for constructing a cross-temporal hierarchical temporal memory network (HTMN) for multimodal sentiment analysis. Background Technology

[0002] Multimodal sentiment analysis aims to integrate heterogeneous modal information such as text, vision, and audio to accurately identify human emotional states, and is an important research direction in the field of affective computing. Current mainstream methods have made significant progress in multimodal feature alignment and fusion, but still face challenges in fine-grained emotional cue capture and cross-modal temporal dynamic modeling: on the one hand, modality-specific noise can easily mask subtle emotional signals; on the other hand, traditional methods struggle to effectively characterize emotional evolution patterns across multiple time scales.

[0003] Existing multimodal sentiment analysis methods mostly focus on single-moment or global-level modal interactions, neglecting the dynamic evolution of emotional states over time. Some methods employ single-timescale modeling, failing to simultaneously characterize instantaneous micro-expression fluctuations and long-term emotional baseline evolution. From a cognitive science perspective, human emotional memory exhibits hierarchical characteristics: short-term memory processes immediate stimuli, medium-term memory maintains emotional trends, and long-term memory stores the overall emotional baseline. However, existing memory networks do not differentiate between instantaneous perception, trend maintenance, and baseline storage functions from this perspective. Furthermore, existing methods often use knowledge distillation and contrastive learning separately, failing to fully leverage the advantages of pre-trained model knowledge and self-supervised feature alignment, resulting in limited cross-modal semantic alignment and noise suppression effects.

[0004] Therefore, there is an urgent need for a multimodal sentiment analysis method that can achieve multi-timescale dynamic hierarchical modeling of sentiment, adaptive retrieval of historical sentiment context, dynamic fusion of current features and historical memory, and combined knowledge distillation and contrastive learning to achieve efficient pre-trained knowledge transfer and cross-modal alignment. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis. This method aims to solve the problems of insufficient fine-grained sentiment cue capture, weak cross-modal temporal dynamic modeling capabilities, and insufficient utilization of pre-trained knowledge in existing methods, thereby improving the accuracy and robustness of multimodal sentiment analysis.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis includes a multimodal feature encoder, a hierarchical temporal memory network (HTMN), a cross-temporal attention recall module (CTAR), a gated fusion layer, multimodal fusion and sentiment prediction, and a multi-task joint training strategy. The overall approach adopts a Teacher-Student framework, wherein:

[0008] A multimodal feature encoder includes a text feature extraction unit, a visual feature extraction unit, and an audio feature extraction unit. These units extract features and perform linear projection on text, visual, and audio inputs respectively, mapping them uniformly to a shared feature space.

[0009] Hierarchical Temporal Memory Network (HTMN): It maintains a three-layer memory structure—Short-Term Memory (STM), Medium-Term Memory (MTM), and Long-Term Memory (LTM)—for both visual and audio modalities. Each layer of memory is updated using an exponential moving average (EMA), and emotional dynamics at different time scales are captured through differentiated decay rates.

[0010]

[0011] in, Indicates at time , No. The memory vector of the layer; These represent short-term, medium-term, and long-term memory, respectively. For each layer of memory projection function; It is the feature vector at the current moment; By integrating historical information with the current state through exponential moving averages, a comprehensive sentiment representation is formed; The decay rate for different levels determines the effective time window of the corresponding memory layer; based on the mathematical properties of EMA, the decay rate... The corresponding effective window length is approximately It is used to capture information at different time scales; the three layers of memory together cover the complete time scale from instantaneous to long-term: short-term memory (STM) captures immediate emotional expression, medium-term memory (MTM) maintains emotional trends, and long-term memory (LTM) stores the overall emotional baseline.

[0012] After the three layers of memory are updated independently, they are combined to form a unified memory pool for subsequent attention retrieval. The visual and audio modal memory pools are defined as follows:

[0013]

[0014]

[0015] in These are unified memory pool matrices for visual and audio modalities, respectively, and are derived from the three-layer memory of the visual modality in the formula. and audio modality three-layer memory The vector is obtained by concatenating rows; each row corresponds to a memory vector at a time scale, for a total of 3 rows; subscripts... The visual and audio modalities are identified respectively, and the superscripts STM, MTM, and LTM are used to identify the short-term, medium-term, and long-term memory layers respectively. This is a row-wise concatenation (stacking) operator.

[0016] Cross-Time Attention Recall (CTAR) module: Employs scaled dot product attention, using the current time-stack features as the query vector and the memory pool as a key-value pair vector. It adaptively retrieves the historical sentiment context most relevant to the current emotional state, projecting the current features as the query vector and the memory pool as key and value vectors. Attention scores are calculated through the dot product of the query and key, and scaled to stabilize the gradient. Attention weights are obtained after Softmax normalization.

[0017]

[0018] The final recalled historical sentiment context is a weighted sum of attention weights and the value vector:

[0019]

[0020]

[0021] in It retrieves context for visual modalities; leveraging the adaptive capabilities of this cross-temporal attention mechanism, attention automatically shifts towards STM when emotional states undergo abrupt changes; and when it is necessary to grasp the overall trend, more weight is allocated to MTM and LTM.

[0022] This provides recall context for audio modalities. Leveraging the adaptive capabilities of this cross-temporal attention mechanism, when there are transient fluctuations in speech intonation (such as sudden pauses or stresses), attention automatically shifts towards STM (Short-Term Memory) to capture fine-grained features. When it is necessary to understand the speaker's overall emotional tone (such as persistent sadness or excitement), more weight is allocated to MTM (Medium-Term Memory) and LTM (Long-Term Memory). These are the query matrix, key matrix, and value matrix of the scaled dot product attention mechanism, respectively. For row Normalization operation; To query the dot product score matrix of the key; The key vector dimension is used to avoid excessively large dot product values ​​in high-dimensional dimensions. Gradient saturation; Learnable visual modalities Projection matrix; Learnable audio modalities Projection matrix.

[0023]

[0024]

[0025] in For the Sigmoid activation function, For gated projection matrix; It is the visual features extracted from video frames at the current moment. It is the auditory feature of the current moment extracted from a speech segment; This indicates vector concatenation; These are the corresponding biases, and the fused visual and audio features are as follows:

[0026]

[0027]

[0028] Where ⊙ represents element-wise multiplication; This represents the fusion weights that control the visual modalities. This represents the fusion weights controlling the acoustic modalities; subsequently, Layer Normalization and Dropout regularization are applied to obtain the final visual and audio feature representations.

[0029]

[0030]

[0031] The gating mechanism behavior can be explained as follows: when g≈1, the model believes that the current emotion is significantly different from the historical emotion and mainly relies on the current information; when g≈0, the model believes that the emotion is continuous and mainly relies on the historical context; when g≈0.5, the two are integrated in a balanced way.

[0032] Multimodal fusion and sentiment prediction: integrating text features Enhanced visual features Enhanced audio features The data is then concatenated and processed by a three-layer fully connected MLP regressor to output the predicted sentiment intensity value. :

[0033]

[0034] The fused multimodal features are input into a three-layer fully connected MLP regressor, which ultimately outputs a predicted sentiment intensity value.

[0035]

[0036] The layer-by-layer dimensionality reduction design of MLP can achieve a smooth mapping from high-dimensional multimodal representations to scalar emotion intensity.

[0037] Multi-task joint training strategy: total loss function Fusion regression loss Knowledge distillation loss Comparative learning loss with cross-modal modes :

[0038]

[0039] in and Hyperparameters are used to balance the contributions of each loss term.

[0040] Furthermore,

[0041] The text feature extraction unit uses text modality as the core carrier of sentiment expression, typically containing the most explicit semantic information. Following the standard experimental setup of CMUMOSI / MOSEI, a pre-trained BERT model is used to extract text features, given an input text sequence. Where 𝑛 represents the sequence length, the BERT encoder converts it into a hidden state sequence:

[0042]

[0043] in For the hidden layer dimension of BERT, the hidden state corresponding to the [CLS] position is taken as the sentence-level feature, and mapped to the unified encoding space through a linear projection layer:

[0044]

[0045] in, The projected text sentence-level feature vector; For learnable projection matrices, For bias vectors, To unify the encoding dimensions. It is worth noting that text features also serve as semantic anchors in cross-modal contrastive learning, guiding visual and audio features to align with the text semantic space.

[0046] Visual Feature Extraction Unit: The visual modality contains rich facial expression and body movement information. Following the standard configuration of the CMU dataset, it uses facial action unit features pre-extracted by the Facet tool, given a visual feature sequence. Where 𝑚 is the visual frame count, The features of each frame are projected into a unified space through two layers of MLP, and then the global visual representation is obtained through average pooling.

[0047]

[0048]

[0049] in The visual features after projection of the i-th frame; This is the current visual global representation obtained through average pooling; For the projection matrix, For hidden layer dimensions; For the first Original visual features of the frame; These are the corresponding biases; the superscript cur indicates the feature representation at the current moment, used to distinguish it from historical information in subsequent memory mechanisms.

[0050] Audio Feature Extraction Unit: The audio modality contains rich paralinguistic features such as pitch, speech rate, and volume. The COVAREP tool is used to pre-extract acoustic features, including fundamental frequency, Mel-frequency cepstral coefficients, and glottal parameters. Given an audio feature sequence... Where l is the number of audio frames, , For the audio feature encoding dimension, a projection strategy consistent with that of the visual modality is adopted:

[0051]

[0052]

[0053] in The audio features are those projected onto the i-th frame. This is the global audio representation of the current moment obtained through average pooling. This is the projection matrix; through the above extraction process, the three modal features are uniformly mapped to... The current audio features are obtained by sharing the representation space.

[0054] The regression loss, with sentiment intensity prediction as the core task, is supervised using mean squared error loss.

[0055]

[0056] in for , For the first The true sentiment intensity label of each sample This corresponds to the predicted value;

[0057] The knowledge distillation loss is based on the KL divergence constraint, which approximates the frozen teacher network output using the student network output.

[0058]

[0059]

[0060]

[0061] in For modal indexing, distillation is performed only for the visual and audio modalities (text-side students have already used it). Encoding, therefore no distillation is required); For the student network (HTMN branch), in modality The original output vector of the last level of the branch; For pre-trained and parameter-frozen teacher networks in modality The original output vector on the branch, which is directly obtained by the teacher network through forward propagation of the input data; , These are the probability distributions of the student and teacher networks after temperature softening, respectively; both are obtained through... calculate; For distillation temperature parameters, The larger the value, the smoother the distribution; in this example, T = 4. for Divergence, defined as .

[0062] The cross-modal contrastive learning loss uses text features as semantic anchors to achieve strong semantic alignment between visual, audio, and text features, thus suppressing modal noise.

[0063]

[0064]

[0065] in Temperature hyperparameter used to control the smoothness of the temperature distribution; superscript They represent Inner A sample; cross-modal feature pairs of the same sample constitute a positive sample. Other sample features constitute negative samples. The sum of the visual and audio contrast loss components; the numerator is the similarity of positive sample pairs (cross-modal feature pairs of the same sample), and the denominator is... The sum of similarities among all candidates (including one positive sample and N-1 negative samples).

[0066] The beneficial effects of this invention are:

[0067] 1. Construct a three-layer differentiated emotional memory architecture to achieve multi-scale temporal modeling of instantaneous emotional fluctuations, mid-term evolutionary trends, and long-term emotional baselines, breaking through the limitations of traditional single-scale modeling;

[0068] 2. The cross-temporal attention recall mechanism can adaptively retrieve highly relevant historical sentiment contexts, significantly improving the ability to model long-term temporal dependencies;

[0069] 3. The gating fusion layer dynamically balances current and historical information to adapt to different emotional scenarios, such as sudden emotional changes and continuous stability;

[0070] 4. By combining knowledge distillation and contrastive learning, we can efficiently transfer knowledge from pre-trained models, achieve strong semantic alignment across modalities, and suppress modal noise.

[0071] 5. On the CMU-MOSI and CMU-MOSEI datasets, the fine-grained emotion recognition, classification accuracy, regression error and other metrics are all superior to existing mainstream methods. Attached Figure Description

[0072] Figure 1 This is an overall framework diagram of the construction method of the present invention;

[0073] Figure 2 This is a heatmap of the loss function weights in an embodiment of the present invention;

[0074] Figure 3 The heatmap of CTAR attention weights in this embodiment of the invention: the horizontal axis is the time step, the vertical axis is the three-layer memory (STM / MTM / LTM), and the color intensity indicates the magnitude of the attention weight.

[0075] Figure 4 The T-SNE feature space visualization comparison diagram of the embodiment of the present invention is as follows: (a) original feature (b) MulT (c) MMML (d) HTMN, with different colors representing different emotional polarities;

[0076] Figure 5 This is a distribution diagram of the gating thresholds for the test set in an embodiment of the present invention. Detailed Implementation

[0077] The present invention will be further described in detail below with reference to specific embodiments.

[0078] Example:

[0079] This invention discloses a method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis, aiming to achieve high-precision multimodal sentiment analysis through a hierarchical temporal memory mechanism. For example... Figure 1As shown, the overall architecture adopts the Teacher-Student framework, which mainly consists of the following core modules: multimodal feature encoder, hierarchical temporal memory network (HTMN), cross-temporal attention recall module (CTAR), gated fusion layer, multimodal fusion and prediction head, and frozen teacher network.

[0080] In terms of overall framework design, the model comprises two parallel processing streams: the teacher network uses CLIP-ViT-Large / 14 and Wav2Vec 2.0-Large, pre-trained on large-scale data, as visual and audio feature extractors, respectively, with parameters frozen during training to generate high-quality soft labels to guide student network learning; the student network uses pre-extracted features from the CMU-MOSI / MOSEI dataset and performs temporal sentiment modeling through a lightweight projection layer and the proposed HTMN module. This design fully utilizes the strong representational capabilities of the pre-trained large model while achieving refined modeling of multi-scale temporal sentiment dynamics through HTMN.

[0081] In the information processing flow, the student network first performs linear projection on the pre-extracted features of the three modalities (text, visual, and audio) to map them to a unified feature space. Subsequently, the visual and audio features are input into their respective HTMN modules for multi-scale temporal modeling—this module maintains a three-layer memory structure consisting of short-term memory (STM), medium-term memory (MTM), and long-term memory (LTM), capturing emotional dynamics at different time scales through differentiated decay rates. After memory updates, the CTAR module uses the current moment's features as the query to adaptively retrieve contextual information related to the current emotional state from the historical memory pool. The retrieved historical context and current features are dynamically weighted by a gated fusion layer, concatenated with the text features, and output as a predicted emotional intensity value through a multilayer perceptron (MLP) regressor.

[0082] Multimodal feature extraction

[0083] A two-stream structure is adopted: the student network uses standard pre-extracted features from the CMU dataset to ensure fair comparison with existing methods; the teacher network uses a pre-trained large model to provide high-quality knowledge distillation signals.

[0084] Text Feature Extraction. As a core carrier of emotional expression, text modalities typically contain the most explicit semantic information. Following the standard CMUMOSI / MOSEI experimental setup, a pre-trained BERT model is used to extract text features. Given an input text sequence... Where 𝑛 represents the sequence length, the BERT encoder converts it into a hidden state sequence:

[0085]

[0086] in This represents the hidden layer dimension of BERT. The hidden state corresponding to the [CLS] position is taken as the sentence-level feature and mapped to a unified encoding space through a linear projection layer.

[0087]

[0088] in, The projected text sentence-level feature vector; For learnable projection matrices, For bias vectors, To unify the encoding dimensions. It is worth noting that text features also serve as semantic anchors in cross-modal contrastive learning, guiding visual and audio features to align with the text semantic space.

[0089] Visual Feature Extraction. Visual modalities contain rich information about facial expressions and body movements. Following the standard configuration of the CMU dataset, facial motion unit features pre-extracted using the Facet tool are employed. Given a sequence of visual features... Where 𝑚 is the visual frame count, The features of each frame are projected into a unified space through two layers of MLP, and then the global visual representation is obtained through average pooling.

[0090]

[0091]

[0092] in The visual features after projection of the i-th frame; This is the current visual global representation obtained through average pooling. For the projection matrix, For hidden layer dimensions; For the first Original visual features of the frame; These are the corresponding biases; the superscript cur indicates the feature representation at the current moment, used to distinguish it from historical information in subsequent memory mechanisms.

[0093] Audio feature extraction. Audio modalities contain rich paralinguistic features such as pitch, speech rate, and volume. The COVAREP tool is used to pre-extract acoustic features, including fundamental frequency, Mel-frequency cepstral coefficients, and glottal parameters. Given an audio feature sequence... Where l is the number of audio frames, , For the audio feature encoding dimension, a projection strategy consistent with that of the visual modality is adopted:

[0094]

[0095]

[0096] in The audio features after projection of the i-th frame; This is the global audio representation of the current moment obtained through average pooling. This is the projection matrix; through the above extraction process, the three modal features are uniformly mapped to... The current audio features are obtained by sharing the representation space.

[0097] Through the above extraction process, the features of the three modalities are uniformly mapped to the di-dimensional shared representation space, laying the foundation for subsequent multimodal fusion.

[0098] Hierarchical Temporal Memory Network (HTMN)

[0099] Emotional expression exhibits significant temporal dynamics, with instantaneous micro-expression fluctuations, mid-term emotional trend evolution, and long-term emotional baseline changes collectively constituting a complete emotional trajectory. Inspired by the hierarchical theory of human memory in cognitive science, this application proposes a Hierarchical Temporal Memory Network (HTMN), which models multi-timescale emotional dynamics through a three-layered differentiated memory structure.

[0100] The emotional memory mechanism HTMN module maintains a three-layer memory structure for both visual and audio modalities. Each memory layer is updated using an exponential moving average (EMA), but captures information at different time scales through varying decay rates. Taking the visual modality as an example, each memory layer is equipped with an independent projection network that transforms the current input features into a stored representation adapted to that time scale. Considering that different time scales may focus on different feature patterns, this example designs a two-layer feedforward network with independent parameters for each memory layer. The three-layer memory update rules are as follows:

[0101]

[0102] in, Indicates at time , No. The memory vector of the layer; These represent short-term, medium-term, and long-term memory, respectively. For each layer of memory projection function; It is the feature vector at the current moment; By integrating historical information with the current state through exponential moving averages, a comprehensive sentiment representation is formed; The decay rate for different levels determines the effective time window of the corresponding memory layer; based on the mathematical properties of EMA, the decay rate... The corresponding effective window length is approximately It is used to capture information at different time scales; the three layers of memory together cover the complete time scale from instantaneous to long-term: short-term memory (STM) captures immediate emotional expression, medium-term memory (MTM) maintains emotional trends, and long-term memory (LTM) stores the overall emotional baseline.

[0103] After the three layers of memory are updated independently, they are combined to form a unified memory pool for subsequent attention retrieval. The visual and audio modal memory pools are defined as follows:

[0104]

[0105]

[0106] in These are unified memory pool matrices for visual and audio modalities, respectively, and are derived from the three-layer memory of the visual modality in the formula. and audio modality three-layer memory The vector is obtained by concatenating rows; each row corresponds to a memory vector at a time scale, for a total of 3 rows; subscripts... The visual and audio modalities are identified respectively, and the superscripts STM, MTM, and LTM are used to identify the short-term, medium-term, and long-term memory layers respectively. This is a row-wise concatenation (stacking) operator.

[0107] Cross-time attention recall mechanism

[0108] The memory pool stores rich, multi-scale historical emotional information, but efficient retrieval of context relevant to the current emotion is a key issue. This application proposes a Cross-Time Attention Recall (CTAR) mechanism, which achieves adaptive retrieval of historical memories through standard Scaled Dot-Product Attention. The core process of the CTAR module is as follows:

[0109] Taking visual modalities as an example, CTAR first projects the current features as a Query vector, and simultaneously projects the memory pool as Key and Value vectors. The attention score is calculated using the dot product of the Query and Key vectors, and then scaled to stabilize the gradient; after Softmax normalization, the attention weights are obtained.

[0110]

[0111] The final recalled historical sentiment context is a weighted sum of attention weights and the value vector:

[0112]

[0113]

[0114] in It retrieves context for visual modalities; leveraging the adaptive capabilities of this cross-temporal attention mechanism, attention automatically shifts towards STM when emotional states undergo abrupt changes; and when it is necessary to grasp the overall trend, more weight is allocated to MTM and LTM.

[0115] This provides recall context for audio modalities. Leveraging the adaptive capabilities of this cross-temporal attention mechanism, when there are transient fluctuations in speech intonation (such as sudden pauses or stresses), attention automatically shifts towards STM (Short-Term Memory) to capture fine-grained features. When it is necessary to understand the speaker's overall emotional tone (such as persistent sadness or excitement), more weight is allocated to MTM (Medium-Term Memory) and LTM (Long-Term Memory). These are the query matrix, key matrix, and value matrix of the scaled dot product attention mechanism, respectively. For row Normalization operation; To query the dot product score matrix of the key; The key vector dimension is used to avoid excessively large dot product values ​​in high-dimensional dimensions. Gradient saturation; Learnable visual modalities Projection matrix; Learnable audio modalities Projection matrix.

[0116] Gating fusion mechanism

[0117] The importance of current information and historical context varies significantly across different emotional scenarios: in scenarios of sudden emotional shifts (such as a sudden emotional outburst), the model should rely more on current information; while in scenarios of emotional inertia (such as persistent sadness), historical context provides crucial reference. To achieve an adaptive balance between the two, this application designs a gating fusion layer. The gating signal is obtained by concatenating the current features and recall context, followed by a linear transformation and sigmoid activation.

[0118]

[0119]

[0120] in For the Sigmoid activation function, For gated projection matrix, It is the visual information of the current moment extracted from video frames. It is the auditory information of the current moment extracted from the speech segment; This represents vector concatenation. These are the corresponding biases, and the fused visual and audio features are as follows:

[0121]

[0122]

[0123] in and This is the intermediate representation after gating and fusion; ⊙ represents element-wise multiplication; This represents the fusion weights that control the visual modalities. This represents the fusion weights controlling the acoustic modalities; subsequently, Layer Normalization and Dropout regularization are applied to obtain the final visual and audio feature representations.

[0124]

[0125]

[0126] The gating mechanism behavior can be explained as follows: when g≈1, the model believes that the current emotion is significantly different from the historical emotion and mainly relies on the current information; when g≈0, the model believes that the emotion is continuous and mainly relies on the historical context; when g≈0.5, the two are integrated in a balanced way.

[0127] Multimodal fusion and sentiment prediction

[0128] After enhancing the features of each modality, a simple and effective post-fusion strategy is adopted to combine the text features. Enhanced visual features Enhanced audio features The data is then concatenated and processed by a three-layer fully connected MLP regressor to output the predicted sentiment intensity value. :

[0129]

[0130] The fused multimodal features are input into a three-layer fully connected MLP regressor, which ultimately outputs a predicted sentiment intensity value.

[0131]

[0132] The layer-by-layer dimensionality reduction design of MLP can achieve a smooth mapping from high-dimensional multimodal representations to scalar emotion intensity.

[0133] Multi-task joint training strategy

[0134] To fully utilize the knowledge of pre-trained large models and enhance cross-modal semantic alignment capabilities, this application designs a multi-task joint training framework that integrates knowledge distillation and contrastive learning. During training, three types of loss functions are optimized simultaneously to form complementary supervision signals.

[0135] The main task is regression loss. Sentiment intensity prediction is the core task, and mean squared error loss is used for supervision.

[0136]

[0137] Among them, 𝑁 is , This is the true sentiment intensity label for the nth sample. This corresponds to the predicted value.

[0138] Knowledge distillation loss: To transfer the strong feature extraction capabilities of CLIP and Wav2Vec to lightweight student networks, a knowledge distillation loss based on KL divergence is introduced. This loss constrains the student network output distribution to approximate the teacher network output distribution.

[0139]

[0140]

[0141]

[0142] in For modal indexing, distillation is performed only for the visual and audio modalities (text-side students have already used it). Encoding, therefore no distillation is required); For student networks (HTMN branch) in modality Output on ; To freeze the teacher network in modal Output on ; , These are the probability distributions of the student and teacher networks after temperature softening, respectively. calculate; For distillation temperature parameters, The larger the value, the smoother the distribution; in this example, T = 4. for Divergence, defined as .

[0143] Cross-modal contrastive learning loss: In multimodal data, visual and audio signals often contain task-irrelevant background noise, while text typically carries the most explicit emotional semantics. To achieve strong cross-modal semantic alignment and filter out noise, this application introduces InfoNCE contrastive learning loss, using text features as semantic anchors:

[0144]

[0145]

[0146] in These represent the contrast loss for the visual modality and the audio modality, respectively. Temperature hyperparameter used to control the smoothness of the temperature distribution; superscript They represent Inner A sample; cross-modal feature pairs of the same sample constitute a positive sample. Other sample features constitute negative samples. The sum of the visual and audio contrast loss components; the numerator is the similarity of positive sample pairs (cross-modal feature pairs of the same sample), and the denominator is... The sum of similarities among all candidates (including one positive sample and N-1 negative samples).

[0147] Total Loss Function: Combining the three types of losses mentioned above, the total loss function of HTMN is defined as follows:

[0148]

[0149] in and Hyperparameters are used to balance the contributions of each loss term.

[0150] Experimental results

[0151] This application is experimentally validated on two widely used multimodal sentiment analysis benchmark datasets: CMU-MOSI and CMU-MOSEI. Specific sample information is shown in Table 1.

[0152] Table 1. Statistical information of CMU-MOSI and CMU-MOSEI datasets

[0153]

[0154] CMU-MOSI contains 2,199 video clips from 93 YouTube video reviews. Each clip is manually labeled with an emotion intensity score ranging from [−3, +3], where −3 represents strong negative emotion and +3 represents strong positive emotion. The dataset is divided into a training set (1,284 clips), a validation set (229 clips), and a test set (686 clips) according to standard criteria.

[0155] CMU-MOSEI is currently the largest multimodal sentiment analysis dataset, containing 23,454 video clips from 1,000 different speakers. Compared to MOSI, MOSI offers greater speaker diversity and more complex emotional expression patterns. The dataset is divided into a training set (16,326 clips), a validation set (1,871 clips), and a test set (4,659 clips).

[0156] Evaluation indicators

[0157] Following standard evaluation protocols in the field of multimodal sentiment analysis, the following metrics are used to evaluate model performance:

[0158] Binary classification accuracy ( The classification accuracy is calculated by labeling samples with an emotion intensity greater than 0 as positive and samples with an emotion intensity less than or equal to 0 as negative. The report includes results for both zero-value samples (Has0) and zero-value samples excluded (Non0).

[0159] F1 score: Calculate the weighted F1 score for the binary classification task, and report it under both Has0 and Non0 settings.

[0160] Multiclass accuracy (Acc5 / Acc7): Discretize the sentiment intensity into 5 or 7 classes and calculate the multiclass accuracy. and It is an evaluation of fine particles

[0161] The key indicator of emotion recognition ability reflects the model's ability to distinguish subtle differences in emotion intensity.

[0162] Mean Absolute Error (MAE): Measures the average absolute deviation between the predicted value and the actual value. The lower the value, the more accurate the prediction.

[0163] Pearson correlation coefficient (Corr): measures the linear correlation between predicted and actual values. A higher value indicates a better consistency between the predicted trend and the actual trend.

[0164] Experimental setup

[0165] The experimental environment consisted of an Ubuntu 20.04 operating system and an NVIDIA RTX 4090 GPU (24GB VRAM). The model was implemented using PyTorch 2.0 and trained using the Adam optimizer. Specific hyperparameter settings were as follows: learning rate set to 1e-4 with cosine annealing scheduling; batch size of 32; 100 training epochs with early stopping (patience=10) to prevent overfitting; and a Dropout ratio of 0.3.

[0166] In the HTMN module, the decay rate of the three-layer memory is set as follows: short-term memory =0.1 (effective window approximately 1.1 time steps), intermediate memory =0.5 (effective window approximately 2 time steps), long-term memory =0.9 (effective window approximately 10 time steps). Uniform coding dimension =256.

[0167] The weights of the loss function are set as follows: =0.5、 =0.3. Knowledge distillation temperature R=4, comparative learning temperature P=0.07.

[0168] To comprehensively evaluate the effectiveness of the proposed method, this application selects the following representative baseline methods for comparison:

[0169] MulT: The first work to introduce the Transformer architecture into multimodal sentiment analysis, achieving implicit alignment of unaligned sequences through a cross-modal attention mechanism.

[0170] SPECTRA: A multimodal fusion method based on spectral decomposition that enhances modal interaction through frequency domain features.

[0171] KuDA: A knowledge-enhanced dynamic attention network that incorporates external knowledge graphs to assist in sentiment understanding.

[0172] SFTTR: Adaptive Feature Transformer, which optimizes the modality fusion path through a dynamic routing mechanism.

[0173] MMML: Multimodal Multilayer Learning Framework, which captures emotional features at different granularities through a hierarchical attention mechanism.

[0174] Table 2 presents a performance comparison of the proposed HTMN with various baseline methods on CMU-MOSI and CMU-MOSEI. Experimental results show that HTMN achieves optimal or near-optimal performance on most evaluation metrics.

[0175] Table 2. Performance Comparison of CMU-MOSI and CMU-MOSEI Datasets

[0176]

[0177] In the best reflection of the ability to distinguish fine-grained emotions In terms of metrics, HTMN achieved 51.87% on CMU-MOSI, a 2.16 percentage point improvement over the strongest baseline MMML; and 55.86% on CMU-MOSEI, a 2.47 percentage point improvement over MMML. This improvement validates that hierarchical temporal memory mechanisms can effectively capture subtle changes in emotional intensity, thereby achieving more refined emotional classification.

[0178] In binary classification accuracy ( In terms of both the F1 and CMU-MOSI metrics, HTMN also achieved the best performance. The scores reached 88.25% / 89.85%, and the F1 scores reached 88.17% / 89.93%; on CMU-MOSEI, The accuracy reached 87.35% / 88.72%, and the F1 score reached 87.21% / 88.63%. In addition, the MAE of the two datasets decreased to 57.15 and 50.48, respectively, and the Corr improved to 87.92% and 82.15%, respectively.

[0179] Advantages of temporal modeling: Compared with methods that rely solely on modal-level attention enhancement, HTMN achieves more effective dynamic modeling of emotional temporal sequence through a hierarchical temporal memory network. Experiments show that the multi-scale temporal memory structure can simultaneously capture instantaneous emotional fluctuations (short-term memory), emotional evolution trends (medium-term memory), and the overall emotional baseline (long-term memory), thus forming a complete characterization of the temporal features of emotions.

[0180] To verify the effectiveness of each core module of HTMN, this application also designed a series of ablation experiments. The experiments were conducted on the CMU-MOSI and CMU-MOSEI datasets, and the results are shown in Table 3. The meanings of each variant in the table are as follows: w / o HTMN means removing the hierarchical temporal memory network and directly using average pooling feature fusion; w / o CTAR means removing the cross-temporal attention recall mechanism and replacing it with simple memory pooling average as historical context; w / o Gate means removing the gated fusion layer and using simple feature concatenation; Single Memory means retaining only single-temporal-scale memory (intermediate memory, α=0.5); TwoMemories means using a two-layer memory structure (short-term + long-term, removing intermediate memory).

[0181] Table 3. Ablation experimental results of CMU-MOSI and CMU-MOSEI datasets

[0182]

[0183] Effectiveness of hierarchical temporal memory networks. After removing the complete HTMN module (w / o HTMN), CMU-MOSI... The percentage of positive emotional data decreased by 3.34 percentage points (51.87% → 48.53%), while the MAE increased by 4.13 (57.15 → 61.28). This significant performance degradation validates the importance of dynamic emotional time-series modeling, indicating that relying solely on static feature fusion is insufficient to fully capture the temporal evolution of emotions.

[0184] Contribution of cross-temporal attention recall mechanism: After removing the CTAR mechanism (without CTAR), the model performance deteriorates significantly, especially in... The indicator decreased by 2.15 percentage points. This indicates that adaptive retrieval of historical sentiment context is crucial for accurately predicting the current emotional state. Compared to simple memory pool averaging, attention-based retrieval can more accurately locate historical information related to the current emotion.

[0185] The role of gating fusion mechanism: After removing the gating fusion layer (w / o Gate), the performance degradation is relatively small but still significant. (A decrease of 1.59 percentage points). This indicates that the gating mechanism plays a crucial role in balancing current information with historical context, and can dynamically adjust the integration ratio of the two according to different emotional scenarios.

[0186] Impact of Memory Layer Number: From Single Memory to Two Memory Layers and then to the full three-layer memory (HTMNFull), model performance improves in a stepwise manner. A single-layer memory can only capture information at a fixed time scale. =49.18%); dual-layer memory covers a wider time range ( =50.41%, but still lacks representation of medium-term sentiment trends; complete three-layer structure ( =51.87%), achieving comprehensive coverage from instantaneous fluctuations to long-term baselines, demonstrating the necessity of multi-timescale modeling.

[0187] Hyperparameter sensitivity analysis

[0188] The impact of hyperparameters on model performance includes memory decay rate and loss function weights. Memory decay rate determines the effective time window of each memory layer and is a core hyperparameter of HTMN. Table 4 shows the model performance under different decay rate configurations.

[0189] Table 4. Performance Comparison under Different Memory Decay Rate Configurations (CMU-MOSI)

[0190]

[0191] Experimental results show that:

[0192] (1) Config-2 The best performance corresponds to an effective time window of approximately 1.1, 2, and 10 time steps, which can cover the instantaneous, medium-term, and long-term time scales well.

[0193] (2) When the interval between the decay rates of the three memory layers is too small (such as Config-4), the time scale differentiation of different memory layers decreases, resulting in performance degradation;

[0194] (3) When the short-term memory decay rate is too low (e.g., Config-1, When the STM time window is too narrow, it is difficult to capture the complete instantaneous emotional pattern;

[0195] (4) The model is robust to the attenuation rate configuration. Within a reasonable range, parameter changes will not cause drastic performance fluctuations.

[0196] The joint loss function in this application consists of three components, and the weight configuration of each component has a significant impact on the model performance. Figure 2 This demonstrates the performance variation under different weight configurations. When the knowledge distillation weights ( When the optimal learning weight is set to 0.5, too small a weight will lead to insufficient utilization of the teacher's network knowledge, while too large a weight may cause the student network to overfit the teacher's output and ignore the true labels. (Comparative learning weights...) The optimal value is 0.3. Moderate contrastive learning can enhance cross-modal semantic alignment, but excessive weights will cause the model to focus too much on modal alignment and weaken the main task of sentiment prediction. The regression loss is used as the main task loss, with a fixed weight of 1.0 to ensure that the model always focuses on sentiment prediction as its core objective.

[0197] To visually demonstrate the working mechanism and advantages of HTMN, this application provides multi-dimensional visualization analysis.

[0198] CTAR Attention Weight Analysis: Visualizing the CTAR attention distribution of a typical sample at each time step of a video sequence. Figure 3 It can be observed that: (1) at time steps 3–5 (emotional stabilization period), attention is mainly focused on LTM, and the model relies on long-term memory to maintain stable emotion prediction; (2) at time steps 8–9 (emotional turning point), attention quickly shifts to STM, the model detects emotional mutations and prioritizes recent information; (3) at time steps 12–15 (emotional recovery period), the weight of MTM gradually increases, and the model captures the gradual calming trend after the emotional outburst. This dynamic attention allocation mechanism enables HTMN to adaptively respond to different emotional time sequence patterns.

[0199] Feature space visualization: visualized using T-SNE. Figure 4 The feature representation quality of different methods was compared. It is evident that the original features exhibit a chaotic trimodal distribution with significant overlap between positive and negative sentiment samples, making them difficult to distinguish. While MulT's distribution improved somewhat after cross-modal Transformer processing, the boundaries between positive and negative samples remained unclear. MMML, benefiting from a multi-level attention mechanism, further enhanced feature separability, with positive and negative samples beginning to form distinct clusters. In contrast, the features learned by HTMN in this application demonstrate the clearest inter-class separation, with more compact positive and negative sample clusters and a progressively distributed pattern of samples with different sentiment intensities in space, reflecting the model's ability to accurately characterize fine-grained sentiment differences.

[0200] Gating value distribution analysis: To more intuitively analyze the impact of the threshold, this application presents the gating value distribution on the test set, and the results are as follows: Figure 5As shown, the following conclusions can be drawn: (1) The gating values ​​show a bimodal distribution, with peak values ​​close to 0.3 and 0.7 respectively, indicating that the model will dynamically adjust the fusion ratio of current information and historical context according to the emotional scene; (2) The visual modality gating values ​​are more dispersed, reflecting the larger range of facial expression changes; the audio modality gating values ​​are relatively concentrated, indicating that the speech features have stronger temporal continuity; (3) There are few extreme gating value samples close to 0 or 1, indicating that the model rarely relies entirely on a single information source, but tends to make comprehensive use of current information and historical information.

[0201] In summary, this application proposes a Hierarchical Temporal Memory Network (HTMN) for multimodal sentiment analysis. Through a three-layer memory architecture with differentiated decay rates, the model achieves hierarchical modeling of sentiment dynamics across multiple time scales. A cross-temporal attention recall mechanism adaptively retrieves relevant historical context, while a gated fusion layer dynamically balances current and historical information. Combining a joint training framework of knowledge distillation and contrastive learning, this method achieves effective cross-modal alignment while simultaneously transferring pre-trained knowledge. Extensive experiments on the CMU-MOSI and CMU-MOSEI datasets validate the effectiveness of the proposed method, demonstrating significant performance improvements over existing state-of-the-art methods, particularly in fine-grained sentiment recognition tasks.

Claims

1. A method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis, characterized by: It includes a multimodal feature encoder, a hierarchical temporal memory network (HTMN), a cross-temporal attention and recall module (CTAR), a gated fusion layer, multimodal fusion and sentiment prediction, and a multi-task joint training strategy. The overall framework adopts the Teacher-Student framework, where: A multimodal feature encoder includes a text feature extraction unit, a visual feature extraction unit, and an audio feature extraction unit. These units extract features and perform linear projection on text, visual, and audio inputs respectively, mapping them uniformly to a shared feature space. Hierarchical Temporal Memory Network (HTMN): It maintains a three-layer memory structure (STM, MTM, LTM) for visual and audio modalities, with each layer updated using an exponential moving average (EMA). It captures emotional dynamics at different time scales through differentiated decay rates. ； in, Indicates at time , No. The memory vector of the layer; These represent short-term, medium-term, and long-term memory, respectively. For each layer of memory projection function; It is the feature vector at the current moment; By integrating historical information with the current state through exponential moving averages, a comprehensive sentiment representation is formed; The decay rate for different levels determines the effective time window of the corresponding memory layer; based on the mathematical properties of EMA, the decay rate... The corresponding effective window length is approximately It is used to capture information at different time scales; the three layers of memory together cover the complete time scale from instantaneous to long-term: short-term memory (STM) captures immediate emotional expression, medium-term memory (MTM) maintains emotional trends, and long-term memory (LTM) stores the overall emotional baseline. After the three layers of memory are updated independently, they are combined to form a unified memory pool for subsequent attention retrieval; the visual and audio modal memory pools are defined as follows: ； ； in These are unified memory pool matrices for visual and audio modalities, respectively, and are derived from the three-layer memory of the visual modality in the formula. and audio modality three-layer memory The data is obtained by concatenating rows; each row corresponds to a memory vector at a given time scale, for a total of 3 rows; indexes... The visual and audio modalities are identified respectively, and the superscripts STM, MTM, and LTM are used to identify the short-term, medium-term, and long-term memory layers respectively. This is a stacking operator for concatenation along the row direction; The Cross-Time Attention Recall (CTAR) module employs scaled dot product attention. Using the current time-stack features as the query vector and the memory pool as a key-value pair vector, it adaptively retrieves the historical sentiment context most relevant to the current emotional state, projecting the current features as the query vector and the memory pool as key and value vectors. Attention scores are calculated through the dot product of the query and key, and scaled to stabilize the gradient. Attention weights are obtained after Softmax normalization. ； The final recalled historical sentiment context is a weighted sum of attention weights and the value vector: ； ； in Recall context for visual modalities; It extracts the visual features of the current moment from video frames; with the help of the adaptive ability of this cross-temporal attention mechanism, when there is a sudden change in emotional state, attention will automatically shift to STM; when it is necessary to grasp the overall trend, more weight will be allocated to the medium memory MTM and the long memory LTM. For the recall context of the audio modality; It extracts auditory features from speech segments at the current moment; with the help of the adaptive ability of this cross-temporal attention mechanism, when there are instantaneous fluctuations in the tone of voice, attention will automatically shift to short-term memory (STM) to capture fine-grained features; when it is necessary to understand the overall emotional tone of the speaker, more weight will be allocated to medium-term memory (MTM) and long-term memory (LTM). These are the query matrix, key matrix, and value matrix of the scaled dot product attention mechanism, respectively. For row Normalization operation; To query the dot product score matrix of the key; The key vector dimension is used to avoid excessively large dot product values ​​in high-dimensional dimensions. Gradient saturation; Learnable visual modalities Projection matrix; Learnable audio modalities Projection matrix; Gated fusion layer: The gated signal is obtained by concatenating the current feature and the recall context, followed by linear transformation and sigmoid activation. ； ； in The Sigmoid activation function is used. For gated projection matrix; This indicates vector concatenation; These are the corresponding biases, and the fused visual and audio features are as follows: ； ； in and This is the intermediate representation after gating and fusion; ⊙ represents element-wise multiplication; This represents the fusion weights that control the visual modalities. This represents the fusion weights that control the acoustic modes; subsequently, through... and Regularization yields the final visual and audio feature representations: ； ； When g≈1, the model considers the current emotion to be significantly different from the historical emotion, and mainly relies on the current information; when g≈0, the model considers the emotion to be continuous, and mainly relies on the historical context; when g≈0.5, the two are integrated in a balanced way. Multimodal fusion and sentiment prediction: integrating text features Enhanced visual features Enhanced audio features The data is then concatenated and processed by a three-layer fully connected MLP regressor to output the predicted sentiment intensity value. : ； The fused multimodal features are input into a three-layer fully connected MLP regressor, which ultimately outputs a predicted sentiment intensity value. ； The layer-by-layer dimensionality reduction design of MLP can achieve a smooth mapping from high-dimensional multimodal representation to scalar emotion intensity; Multi-task joint training strategy: total loss function Fusion regression loss Knowledge distillation loss Comparative learning loss with cross-modal modes : ； in and Hyperparameters are used to balance the contributions of each loss term.

2. The method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis according to claim 1, characterized in that: The text feature extraction unit uses text modalities as the core carrier of emotional expression, and extracts text features using a pre-trained BERT model, given an input text sequence. Where 𝑛 represents the sequence length, the BERT encoder converts it into a hidden state sequence: ； in For the hidden layer dimension of BERT, the hidden state corresponding to the [CLS] position is taken as the sentence-level feature, and mapped to the unified encoding space through a linear projection layer: ； in, The projected text sentence-level feature vector; For learnable projection matrices, For bias vectors, To unify the coding dimensions.

3. The method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis according to claim 1, characterized in that: Visual Feature Extraction Unit: Visual modalities contain rich facial expressions and body movement information. Facial action unit features are pre-extracted using the Facet tool, given a visual feature sequence. ,in For visual frames, , To encode the dimensions of visual features, each frame's features are projected into a unified space using a two-layer MLP, and then average pooling is used to obtain the global visual representation: ； ； in The visual features after projection of the i-th frame; This is the current visual global representation obtained through average pooling; Let be the projection matrix. For hidden layer dimensions; For the first Original visual features of the frame; These are the corresponding biases; the superscript cur indicates the feature representation at the current moment, used to distinguish it from historical information in subsequent memory mechanisms.

4. The method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis according to claim 1, characterized in that: Audio Feature Extraction Unit: The audio modality contains rich paralinguistic features such as pitch, speech rate, and volume. The COVAREP tool is used to pre-extract acoustic features, including fundamental frequency, Mel-frequency cepstral coefficients, and glottal parameters. Given an audio feature sequence... Where l is the number of audio frames, , For the audio feature encoding dimension, a projection strategy consistent with that of the visual modality is adopted: ； ； in The audio features are those projected onto the i-th frame. This is the global audio representation of the current moment obtained through average pooling. This is the projection matrix; through the above extraction process, the three modal features are uniformly mapped to... The current audio features are obtained by sharing the representation space.

5. The method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis according to claim 1, characterized in that: The knowledge distillation loss is based on the KL divergence constraint, which approximates the frozen teacher network output using the student network output. ； ； ； in For modal indexing, distillation is performed only for the visual and audio modalities; For student networks, in modal The original output vector of the last level of the branch; For pre-trained and parameter-frozen teacher networks in modality The original output vector on the branch, which is directly obtained by the teacher network through forward propagation of the input data; , These are the probability distributions of the student and teacher networks after temperature softening, respectively. Divergence, defined as .

6. The method for constructing a cross-temporal hierarchical temporal memory network for multimodal sentiment analysis according to claim 1, characterized in that: The cross-modal contrastive learning loss uses text features as semantic anchors to achieve strong semantic alignment between visual, audio, and text features, thus suppressing modal noise. ； ； in These represent the contrast loss for the visual modality and the audio modality, respectively. Temperature hyperparameter used to control the smoothness of the temperature distribution; superscript They represent Inner A sample; cross-modal feature pairs of the same sample constitute a positive sample. Other sample features constitute negative samples. The sum of the visual and audio contrast loss components; the numerator is the similarity of positive samples to cross-modal features of the same sample, and the denominator is... The sum of all candidates.

Images