Multi-modal temporal sentiment consistency recognition method and system based on continuous-time dynamic modeling

Abstract

The application discloses a kind of multi-modal timing emotional consistency identification method and system based on continuous time dynamic modeling.Firstly, the characteristics of multi-modal data are extracted and preprocessed;Secondly, feature projection and spatial alignment are performed, and an independent linear mapping layer is constructed to project heterogeneous features to a unified latent space dimension;Then, a CT-RNN encoder is constructed to perform differential evolution of hidden states in continuous time space;Thirdly, a time constant network is used to dynamically calculate a channel-level time constant vector to adaptively regulate the evolution rate to align the emotional semantics of different modalities;Then, based on the similarity of semantics and vision, high-conflict difficult examples are screened, and a random mask is used to calculate consistency loss;Finally, the features are fused and combined with CCC Loss to output the emotional prediction result.The application uses continuous modeling and adaptive alignment to solve the problem of discrete sampling information loss and asynchronization, and significantly improves the robustness in complex scenarios through difficult example mining.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and affective computing, and in particular to a method and system for recognizing multimodal temporal affective consistency based on continuous-time dynamic modeling. Background Technology

[0002] Multimodal sentiment analysis is one of the core technologies in the fields of artificial intelligence and human-computer interaction today. It aims to analyze complex human psychological states by fusing heterogeneous data such as text, acoustics, and vision. This technology can capture emotional cues from multiple dimensions, including language content, voice tone, and facial expressions, and has significant application value in scenarios such as intelligent customer service, mental health monitoring, and virtual digital human interaction. Accurate multimodal sentiment recognition not only requires the model to understand the semantics of a single modality but also emphasizes modeling the temporal synergy and consistency of multi-source information to determine whether a user is insincere or in a state of emotional conflict.

[0003] Existing multimodal sentiment analysis methods primarily rely on deep learning frameworks for temporal modeling, such as the widespread use of Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRUs), or Transformer architectures. However, these existing technologies still have shortcomings when dealing with non-stationary multimodal data in real-world scenarios, specifically in the following aspects:

[0004] First, existing technologies are generally based on discrete time-steps for calculation, making it difficult to fit the continuous dynamic evolution of emotions. Human emotional expression is essentially a continuous process at the physical level, and the rate of signal change varies greatly between different modalities (for example, semantic expressions typically last for several seconds, while micro-expressions or abrupt changes in tone may only last for hundreds of milliseconds). Existing methods forcibly cut continuous signals into fixed discrete segments, causing high-frequency transient emotional features (such as micro-expressions) to be smoothed or lost during sampling, making it impossible to achieve an accurate fit of the emotional flow from a mathematical perspective.

[0005] Secondly, existing multimodal alignment mechanisms lack adaptability and struggle to handle "non-equal-length emotion duration windows." To address the issue of inconsistent sampling rates across different modalities, current techniques typically employ forced alignment or zero-padding strategies. Forced alignment often assumes that speech and visual signals are strictly bound to text boundaries, ignoring the independence of non-verbal signals; while zero-padding introduces significant artificial noise, leading to decreased robustness when processing long-tailed data. Current models lack a mechanism to automatically adjust the receptive field or step size based on the physical rate of change of the input content.

[0006] Furthermore, existing sentiment consistency judgments primarily focus on the fusion of global features, lacking fine-grained mining of local temporal conflicts. In actual interactions, emotional conflicts often occur within extremely short time segments (such as a momentary frown while speaking). Existing models mostly employ global pooling or static attention mechanisms, which are easily masked by the dominant modality (such as text semantics), leading to limited accuracy in recognizing complex emotional states such as "irony" and "sarcasm." Summary of the Invention

[0007] To address the issues that existing modeling methods based on discrete time steps cannot accurately fit the continuous dynamic evolution of emotions, and that forced alignment strategies introduce artificial noise and lack the ability to adapt to windows with unequal duration of emotions, this invention proposes a multimodal temporal emotion consistency recognition method and system based on continuous time dynamic modeling.

[0008] The first aspect of this invention provides a multimodal temporal sentiment consistency recognition method based on continuous-time dynamic modeling, comprising the following steps:

[0009] Data acquisition and preprocessing: Acquire multimodal data, including text sequences, acoustic sequences, and visual sequences; and preprocess the multimodal data to obtain heterogeneous feature sequences;

[0010] Feature projection and spatial alignment: Construct three independent linear mapping layers to project the heterogeneous feature sequences onto a unified latent spatial dimension to obtain heterogeneous feature sequences with a unified dimension;

[0011] Continuous-time dynamic modeling: Construct a continuous-time recurrent neural network based on the constant differential equation of the nervous system, input the heterogeneous feature sequence of the same dimension into the continuous-time recurrent neural network, and perform differential evolution of the hidden state in the continuous time space;

[0012] Adaptive temporal alignment: Based on the current input feature vector and hidden state vector, the time constant network dynamically calculates the channel-level time constant vector and adaptively adjusts the evolution rate of the hidden state to achieve sentiment semantic alignment of different modalities in continuous time.

[0013] Difficult example mining and fusion discrimination: Calculate the cosine similarity between text feature vectors and visual feature vectors, filter high-conflict difficult examples through dynamic Top-K filtering, apply random mask perturbation to non-text modalities, and then perform nonlinear fusion on the hidden states that have been aligned in continuous time to output the final sentiment consistency prediction result.

[0014] A second aspect of the present invention provides a multimodal temporal sentiment consistency recognition system based on continuous-time dynamic modeling, comprising:

[0015] The data acquisition and preprocessing module is used to acquire multimodal data, including text sequences, acoustic sequences, and visual sequences; and to preprocess the multimodal data to obtain heterogeneous feature sequences.

[0016] The feature projection and spatial alignment module is used to construct three independent linear mapping layers to project the heterogeneous feature sequences onto a unified latent spatial dimension, thereby obtaining heterogeneous feature sequences with a unified dimension.

[0017] The continuous-time dynamic modeling module is used to construct a continuous-time recurrent neural network based on the neural ordinary differential equation. The heterogeneous feature sequence of the same dimension is input into the continuous-time recurrent neural network to perform differential evolution of the hidden state in the continuous time space.

[0018] The adaptive temporal alignment module is used to dynamically calculate the channel-level time constant vector based on the current input feature vector and hidden state vector through a time constant network, and adaptively adjust the evolution rate of the hidden state to achieve sentiment semantic alignment of different modalities in continuous time.

[0019] The difficult example mining and fusion discrimination module is used to calculate the cosine similarity between text feature vectors and visual feature vectors. It uses dynamic Top-K filtering to select high-conflict difficult examples, applies random mask perturbation to non-text modalities, and then performs nonlinear fusion on the hidden states that have been aligned in continuous time to output the final sentiment consistency prediction result.

[0020] The beneficial effects of this invention are as follows: This invention utilizes neural constant differential equations to achieve continuous-time dynamic modeling, accurately capturing high-frequency features such as transient micro-expressions; it proposes an adaptive time constant mechanism, which can achieve physical-level accurate synchronization of heterogeneous modal non-uniform length windows without manual alignment, eliminating filling noise; combined with a high-conflict difficult example mining strategy, it effectively solves the problem of recognizing inconsistencies between semantic and non-linguistic signals, significantly improving robustness in complex scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the overall architecture of a multimodal temporal sentiment consistency recognition method based on continuous-time dynamic modeling provided in an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the core method flow provided in the embodiments of the present invention;

[0024] Figure 3 This is a comparison chart of the training convergence performance of the embodiments of the present invention on the real CMU-MOSEI dataset with that of the prior art (LSTM). Detailed Implementation

[0025] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0026] like Figure 1 and Figure 2 As shown, this embodiment provides a multimodal temporal sentiment consistency recognition method based on continuous-time dynamic modeling, which mainly includes steps such as data acquisition and preprocessing, feature projection, continuous-time dynamic modeling, adaptive temporal alignment, and fusion discrimination and verification. Specifically:

[0027] S1: Data Acquisition and Feature Preprocessing

[0028] S1 transforms unstructured multimodal raw data into a computer-processable structured feature tensor, wherein the multimodal data includes text sequences, acoustic sequences, and visual sequences.

[0029] Further, for text modality: First, the raw text data (e.g., user comments, conversation logs) is acquired. A tokenizer is used to segment the text, converting it into a token sequence. In this embodiment, the tokenizer of the BERT (Bidirectional Encoder Representations from Transformers) pre-trained model is preferably used. Next, a maximum sequence length L (e.g., 50) is set. Sequences longer than L are truncated, and sequences shorter than L are padded with masks. Subsequently, features are extracted using a pre-trained BERT-base-uncased model, and the output of its last hidden state is taken as the text feature sequence, denoted as... Its dimensions are (L, 768).

[0030] Furthermore, regarding audio modality: the original audio signal is first resampled to a uniform sampling rate (e.g., 16kHz). To extract acoustic features containing emotional information such as intonation and prosody, this embodiment preferably uses a pre-trained Wav2Vec 2.0 model. This model can extract frame-level features with contextual information from the original waveform. The extracted acoustic feature sequence is denoted as... Its dimensions are ,in 74 represents the number of audio frames and 74 represents the feature dimensions (including MFCCs, fundamental frequency F0, glottal parameters, etc.).

[0031] Furthermore, for visual modality: a sequence of video frames synchronized with the audio is acquired. Each frame is analyzed using the OpenFace 2.0 toolkit to extract facial action units (AUs), gaze direction, and head pose. In this embodiment, the 35-dimensional features (containing 17 AU intensity values) with the strongest emotional representation are selected to construct the visual feature sequence. , dimension .

[0032] Data Normalization: Due to the significant differences in the feature value distributions of the three modalities mentioned above, and the fact that the neural ordinary differential equation is quite sensitive to the scale of the input values, this embodiment adopts the following normalization strategy to prevent gradient instability:

[0033] For text features, maximum value normalization is used to map them to... interval;

[0034] For acoustic and visual features, a Z-Score normalization strategy is used, and the calculation formula is as follows:

[0035]

[0036] in and These are the mean and standard deviation of the training set statistics, respectively. To prevent small constants from being divided by zero (such as...) ).

[0037] S2: Alignment of feature projection with latent space

[0038] In this embodiment, since the original feature dimensions of text, acoustic, and visual are 768, 74, and 35 respectively, they cannot be directly fused. Step S2 projects the heterogeneous feature sequences to a unified latent spatial dimension by constructing three independent linear projection layers.

[0039] Specifically, three fully connected layers are defined, with their weight matrices as follows: , and .in The hidden layer dimension is preset, and in this embodiment, it is preferably 64.

[0040] The projection calculation formula is as follows:

[0041]

[0042]

[0043]

[0044] in For random deactivation operation (drop rate set to 0.2); This is the bias term. After projection, the feature sequence dimension of all modalities is unified to . This laid the foundation for subsequent continuous-time modeling.

[0045] S3: Continuous-Time Dynamic Modeling

[0046] Existing techniques typically use RNNs or Transformers to model discrete time steps, assuming time intervals. It is fixed. However, multimodal data often exhibits a non-uniform distribution over physical time. This embodiment introduces Neural ODEs to construct a continuous-time recurrent neural network (CT-RNN). Heterogeneous feature sequences of the same dimension are input into the CT-RNN to perform differential evolution of the hidden states in continuous time space.

[0047] Furthermore, S3 hides the state vector. Defined as about time It is a continuous function. Its state update is no longer a discrete addition, but follows the following first-order ordinary differential equation:

[0048]

[0049] in, express The hidden state vector at time step; express Input characteristics at any given time; It is a nonlinear function parameterized by a neural network (e.g., a fully connected layer activated by Tanh) used to fit the driving force of the input on the state; These are network parameters.

[0050] in The natural decay representing the hidden state simulates the "leaky integrator" characteristic of biological neurons; This represents activation from external inputs. This modeling approach enables the network to remember historical information and evolve continuously over time.

[0051] Furthermore, in order to obtain the state at the next moment... In this embodiment, a numerical integration method is used to solve the above differential equation. Preferably, to ensure a balance between accuracy and computational efficiency, the Runge-Kutta 4th order method (RK4) is used to solve the neural ordinary differential equation. The hidden state vector at the next time step is updated by calculating a weighted average of multiple slope estimates within the time step. The specific calculation steps are as follows:

[0052] The first step is to calculate the initial slope: ;

[0053] The second step is to calculate the midpoint prediction slope 1: ;

[0054] The third step is to calculate the midpoint prediction slope 2: ;

[0055] Step 4: Calculate the final state prediction slope: ;

[0056] Step 5, update the status: .

[0057] Therefore, through high-precision integral solving, this embodiment enables the model to accurately fit the subtle changes in emotion over continuous time and space.

[0058] S4: Adaptive timing alignment mechanism

[0059] This embodiment designs a dedicated "Time Constant Network". Unlike the coarse approach of uniformly scaling all features in existing technologies, it proposes a channel-wise adaptive mechanism.

[0060] Specifically, the network uses the input feature vector at the current time. and hidden state vector The optimal time constant is calculated independently for each dimension of the hidden layer. The calculation formula is as follows:

[0061]

[0062] in, This indicates the concatenation of vectors. For learnable parameters, It is a dimension The vector.

[0063] In this embodiment, in order to prevent numerical instability ( Too small (leading to gradient explosion) and too slow reaction ( (Too large leads to underfitting) Set , .

[0064] In this embodiment, the physical meaning of the adaptive temporal alignment mechanism lies in the fact that the rate of change of human emotional expression in different dimensions is asynchronous (for example, changes in facial micro-expressions often outpace the unfolding of semantic content). This is achieved by outputting a vector form... The model achieves "feature decoupled alignment"—that is, it allows certain rapidly changing feature channels (such as visual AU) to be captured with a smaller time constant, while allowing slowly changing feature channels (such as background context) to maintain a larger time constant to preserve long-range memory. This fine-grained dynamic alignment is significantly better than traditional scalar scaling.

[0065] S5: Difficult Example Mining and Fusion Judgment

[0066] High-conflict difficult example mining: During training, text feature vectors are calculated in real time. With visual feature vectors Cosine similarity:

[0067]

[0068] This embodiment argues that the "insincere" nature of emotions is primarily manifested in the inconsistency between linguistic content (Text) and facial expressions (Visual) (e.g., saying harsh words while smiling), while acoustic features often contain significant environmental noise or uncertainty. Therefore, this embodiment focuses on calculating the feature distance between text and visual representations to pinpoint the samples with the most intense semantic conflicts.

[0069] Specifically, a dynamic Top-K filtering strategy is adopted. A preset ratio is set. (In this embodiment) =0.3), sort all samples in the current batch by text-visual similarity, and select the 30% of samples with the lowest similarity. These samples are marked as "high-conflict difficult examples".

[0070] Random Masking Perturbation and Consistency Validation: For the hard examples discovered, random masking perturbs are applied to their non-textual modalities. Specifically, a binary mask matrix with the same shape as the feature vector is generated (0 is generated with a probability of 0.1, and the rest are 1). The feature tensor is then multiplied by the mask. The perturbed sample is then input back into the model, and the mean squared error (MSE) between its output and the original output is calculated. This MSE serves as the consistency loss, forcing the model to maintain the stability of its predictions even in the face of modal noise.

[0071] Multimodal fusion and output: Nonlinear interaction fusion of heterogeneous modal states aligned to continuous time is performed to resolve complex dependencies between modes (such as complementarity, conflict, inhibition, etc.). This includes the following steps:

[0072] (1) State splicing

[0073] Furthermore, in this embodiment, at the end of the continuous-time evolution... The hidden state vectors of the three independent channels—text, acoustic, and visual—are extracted separately. Because it has been adaptive The mechanism has achieved physical time alignment, at which point the three state vectors are strictly synchronized at the semantic level. This embodiment concatenates these three vectors along the channel dimension to construct a global feature descriptor containing multi-source information:

[0074]

[0075] (2) Nonlinear feature interaction

[0076] Furthermore, to capture non-linear relationships between modalities (such as the polarity reversal of semantics and expression in irony), this embodiment inputs the concatenated vectors into a multilayer perceptron (MLP). This module includes:

[0077] Feature compression layer: Maps high-dimensional concatenated vectors to a low-dimensional manifold to extract the most significant cross-modal features.

[0078] Nonlinear activation (ReLU): Introducing nonlinear factors enables the model to fit complex XOR logic and identify modal conflicts.

[0079] Dropout: Randomly blocks some neurons during training to prevent the model from over-relying on a dominant modality (such as relying solely on text) and forces the model to make comprehensive use of multimodal information.

[0080] (3) Final judgment and loss constraint

[0081] Furthermore, the features processed by the MLP are input into the final sentiment regression layer (Fully Connected Layer), which outputs a continuous scalar value. (Emotion Intensity Prediction). To ensure that the prediction results are both accurate and consistent with the patterns of emotion change, this embodiment designs a dual loss constraint, including:

[0082] MSE Loss: Mean Squared Error Loss, which constrains the numerical distance between the predicted value and the true label to ensure accuracy.

[0083]

[0084] CCC Loss: Consistency Correlation Coefficient Loss, used to constrain the distribution trend of the predicted sequence and the true sequence to ensure "similarity".

[0085]

[0086] in, , These are the mean values ​​of the predicted and actual values, respectively. These are the standard deviations of the predicted and actual values, respectively. The Pearson correlation coefficient between predicted and actual values ​​is calculated using the following formula:

[0087]

[0088] Since the goal is to maximize CCC (i.e., make it close to 1), its complement is usually taken when used as the loss.

[0089]

[0090] Furthermore, in this specific embodiment, to meet the high computing power requirements of the deep learning model, the experimental hardware environment is configured as follows: CPU is Intel Xeon Gold 6240, GPU is NVIDIA GeForce RTX 3070 (8GB VRAM), and memory is 2TB DDR5. The software environment is based on the Ubuntu 20.04 LTS operating system, using the PyTorch 1.10 deep learning framework and the torchdiffeq 0.2.3 differential equation library.

[0091] Experimental verification and result analysis

[0092] Furthermore, to fully verify the effectiveness of the technical solution of this invention, this embodiment conducted detailed experiments on the publicly available authoritative dataset CMU-MOSEI. CMU-MOSEI is one of the largest multimodal sentiment analysis datasets currently available, containing 23,453 annotated sentences and exhibiting significant non-uniform length and asynchronous characteristics.

[0093] The experimental parameters were set as follows:

[0094] (1) Batch Size: 32;

[0095] (2) Number of training epochs: 100;

[0096] (3) Optimizer: AdamW, Weight Decay coefficient is ;

[0097] (4) Learning rate strategy: The initial learning rate is set to 0.001, and the StepLR strategy is adopted (decaying by 0.5 every 10 rounds) to perform fine-grained search in the later stage of training;

[0098] (5) Loss Function: This embodiment uses a composite loss function:

[0099]

[0100] in The mean squared error loss for the main task. The Consistency Correlation Coefficient Loss (CCC Loss) is used to maximize the correlation between the predicted distribution and the true distribution, significantly improving... index; The consistency constraint loss (MSE) for difficult cases. In this embodiment, the weighting coefficient is set to... .

[0101] Evaluation Indicators Explanation:

[0102] To comprehensively evaluate the model's performance, this embodiment selects the following five core metrics:

[0103] (1) MAE (Mean Absolute Error): Measures the average distance between the predicted sentiment intensity value and the true label value. The lower the value, the more accurate the regression prediction of the model.

[0104] (2) Pearson Correlation (Corr): Measures the linear correlation between the predicted sequence and the actual sequence in terms of their changing trends. The closer the value is to 1, the more sensitively the model can capture the fluctuations in sentiment over time (Trend).

[0105] (3) Score (R²): Measures the model's ability to explain the variance of the data. A value closer to 1 indicates a better fit; a value close to 0 or negative indicates that the model cannot effectively fit the data distribution. This is a key indicator for verifying whether the model has learned the data distribution pattern through CCC Loss.

[0106] (4) HC Subset MAE (High Conflict Subset Error): The MAE is specifically calculated for the selected "high conflict difficult cases". This metric is used to evaluate the prediction error of the model when dealing with difficult samples such as "insincere statements".

[0107] (5) HC Subset Accuracy: Maps the continuous sentiment intensity values ​​output by the model to positive and negative polarities (e.g., positive for values ​​greater than 0 and negative for values ​​less than 0), and calculates its binary classification accuracy on a set of difficult examples. The higher the value, the more robust the model is in semantic conflict scenarios.

[0108] Comparative experimental setup (Ablation Study):

[0109] To refine the verification of the contributions of each module (especially adaptive alignment and hard example mining), this embodiment sets up three hierarchical sets of models:

[0110] (1) Baseline: The industry-standard LSTM model is used, and zero padding is used to process variable-length sequences. There is no continuous time modeling.

[0111] (2) Variant group: Fixed CT-RNN, containing only continuous-time modeling (fixed) =2.0), the adaptive alignment mechanism is not enabled, and the difficult case mining and perturbation verification modules are not enabled;

[0112] (3) Ours: Adaptive CT-RNN, which includes a complete adaptive alignment and high-conflict hard case mining optimization strategy.

[0113] Experiment 1: Global Performance Comparison

[0114] Before comparing the final performance metrics in detail, in order to intuitively verify the training efficiency and convergence stability of the model, Figure 3 The MAE Loss curves of different models during training are shown. Figure 3As shown, the baseline LSTM model (red dashed line) converges slowly and has a high final error; the Fixed CT-RNN variant (blue dotted line) without adaptive mechanism decreases quickly in the early stages, but due to the lack of fine alignment for non-equal-length windows, it exhibits significant oscillations in the later stages; while the complete model proposed in this invention (green solid line) benefits from the channel-level adaptive alignment mechanism, and its loss curve shows a steep and continuous downward trend, converging to the lowest extreme value (MAE < 0.1) around the 80th round, and maintaining extremely high stability in subsequent training, fully demonstrating its robustness and fitting advantage under long-term training.

[0115] First, the overall performance of each model was evaluated on the complete test set, and the results are shown in Table 1.

[0116] Table 1. Performance metrics of different algorithms on the CMU-MOSEI dataset

[0117]

[0118] Experiment 2: Robustness Verification of High-Conflict Difficult Examples

[0119] To specifically verify the "difficult example mining" module's ability to identify complex samples such as "insincere", this invention constructs a "high-conflict test subset".

[0120] Subset definition: Select the 30% of samples in the test set with the lowest cosine similarity between text features and visual features. These samples usually contain irony or complex emotions, making them extremely difficult to recognize.

[0121] Verification objective: To demonstrate that by incorporating a consistency constraint, this invention can effectively prevent the model from being misled by a single text modality and improve the discrimination accuracy.

[0122] The experimental results are shown in Table 2.

[0123] Table 2 Performance Comparison of High-Conflict Difficulty Example Sets

[0124]

[0125] Furthermore, in-depth analysis of the results and technological advantages:

[0126] 1. A qualitative change in global goodness of fit ( Score

[0127] As shown in Table 1, the present invention The score reached 0.0516, representing a performance leap of approximately 5 times compared to the baseline LSTM (0.0103). This data strongly demonstrates that traditional discrete models struggle to capture the complex distribution of multimodal data. This invention, by introducing Consistent Correlation Coefficient Loss (CCC Loss) in conjunction with neural frequent differential equations, successfully overcomes the "central tendency effect" common in regression tasks, resulting in a qualitative leap in the model's ability to interpret the variance of sentiment intensity.

[0128] 2. Significantly enhanced trend-following capabilities (Pearson Corr)

[0129] On the Pearson correlation coefficient (Corr) metric, this invention achieved 0.2824, significantly outperforming the baseline group (0.1932) and the variant group (0.2439). This demonstrates that the complete Adaptive CT-RNN model not only fits static numerical values ​​but also keenly captures the dynamic trends of emotion evolution over time. This is a direct manifestation of the "adaptive temporal alignment" mechanism—dynamically adjusting the time constant. The model accurately synchronizes the nonlinear change rhythm of heterogeneous modes.

[0130] 3. Breakthrough in discrimination under high-conflict scenarios (HC Subset)

[0131] The data in Table 2 reveals the model's significant advantage in handling insincere samples. The baseline group achieved an accuracy of only 59.3% on high-conflict subsets, indicating its susceptibility to being misled by a single dominant modality. Ours, however, significantly improved accuracy to 68.7% (an improvement of nearly 10 percentage points), and reduced subset error to 0.8142. This result demonstrates that by introducing random mask perturbation and MSE consistency loss, the model is forced to simulate modality-deficient scenarios during training, thereby learning to consider non-verbal modal cues rather than blindly following the text when faced with conflicting information, achieving a dual improvement in robustness and accuracy.

[0132] in conclusion:

[0133] As can be seen from Tables 1 and 2, this invention, while ensuring the best global prediction accuracy (lowest Global MAE of 0.8141), significantly improves the fitting ability of complex sentiment trends through CCC Loss, and achieves a substantial breakthrough in the identification of highly conflicting difficult cases through dynamic difficult case mining strategy, perfectly achieving the unity of robustness, accuracy and consistency.

[0134] Based on the same concept as the above method embodiments, this invention also proposes a multimodal temporal sentiment consistency recognition system based on continuous-time dynamic modeling, including:

[0135] The data acquisition and preprocessing module is used to acquire multimodal data, including text sequences, acoustic sequences, and visual sequences; and to preprocess the multimodal data to obtain heterogeneous feature sequences.

[0136] The feature projection and spatial alignment module is used to construct three independent linear mapping layers to project the heterogeneous feature sequences onto a unified latent spatial dimension, thereby obtaining heterogeneous feature sequences with a unified dimension.

[0137] The continuous-time dynamic modeling module is used to construct a continuous-time recurrent neural network based on the neural ordinary differential equation. The heterogeneous feature sequence of the same dimension is input into the continuous-time recurrent neural network to perform differential evolution of the hidden state in the continuous time space.

[0138] The adaptive temporal alignment module is used to dynamically calculate the channel-level time constant vector based on the current input feature vector and hidden state vector through a time constant network, and adaptively adjust the evolution rate of the hidden state to achieve sentiment semantic alignment of different modalities in continuous time.

[0139] The difficult example mining and fusion discrimination module is used to calculate the cosine similarity between text feature vectors and visual feature vectors. It uses dynamic Top-K filtering to select high-conflict difficult examples, applies random mask perturbation to non-text modalities, and then performs nonlinear fusion on the hidden states that have been aligned in continuous time to output the final sentiment consistency prediction result.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal temporal sentiment consistency recognition method based on continuous-time dynamic modeling, characterized in that, Includes the following steps: Data acquisition and preprocessing: Acquire multimodal data, including text sequences, acoustic sequences, and visual sequences; and preprocess the multimodal data to obtain heterogeneous feature sequences; Feature projection and spatial alignment: Construct three independent linear mapping layers to project the heterogeneous feature sequences onto a unified latent spatial dimension to obtain heterogeneous feature sequences with a unified dimension; Continuous-time dynamic modeling: Construct a continuous-time recurrent neural network based on the constant differential equation of the nervous system, input the heterogeneous feature sequence of the same dimension into the continuous-time recurrent neural network, and perform differential evolution of the hidden state in the continuous time space; Adaptive temporal alignment: Based on the current input feature vector and hidden state vector, the time constant network dynamically calculates the channel-level time constant vector and adaptively adjusts the evolution rate of the hidden state to achieve sentiment semantic alignment of different modalities in continuous time. Difficult example mining and fusion discrimination: Calculate the cosine similarity between the text feature vector and the visual feature vector, filter high-conflict difficult examples through dynamic Top-K, apply random mask perturbation to the non-text modality, and then perform nonlinear fusion on the hidden states that have been aligned in continuous time to output the final sentiment consistency prediction result.

2. The method according to claim 1, characterized in that, The heterogeneous feature sequences include text feature sequences, acoustic feature sequences, and visual feature sequences, wherein: The text modality is segmented using the BERT word segmenter to obtain the text feature sequence; The acoustic modality is obtained by resampling the original audio signal to a uniform sampling rate using the Wav2Vec 2.0 model and extracting the acoustic feature sequence. The visual modality is generated by analyzing the video frame sequence synchronized with the audio using the OpenFace 2.0 toolkit.

3. The method according to claim 1 or 2, characterized in that, A continuous-time recurrent neural network based on the neural ordinary differential equation is constructed. During the continuous-time dynamic modeling, the hidden state vector is defined as a continuous function of time, following the neural ordinary differential equation, as shown in the following formula: in, express The hidden state vector at time step; express Input characteristics at any given time; It is a nonlinear function parameterized by a neural network, used to fit the driving force of the input on the state; For network parameters; This represents the optimal time constant vector.

4. The method according to claim 3, characterized in that, When solving the aforementioned neural ordinary differential equation using the RK4 method, the hidden state vector for the next time step is updated by calculating the weighted average of multiple slope estimates within the time step.

5. The method according to claim 1, characterized in that, The adaptive timing alignment calculates a time constant vector using a time constant network. The calculation formula is as follows: in, This represents the concatenated vector of the hidden state and the input features; These are learnable weights; For bias; For activation functions; and These are the preset upper and lower threshold values ​​for the time constant; where the time constant vector... Dimensions and hidden state vectors Dimensional consistency is used to achieve channel-level adaptive time scaling.

6. The method according to claim 1, characterized in that, When using the dynamic Top-K filtering method to identify high-conflict difficult examples, a preset ratio is set to sort the text-visual similarity of all samples in the current batch, and the top 30% of samples with the lowest similarity are selected and marked as high-conflict difficult examples.

7. The method according to claim 6, characterized in that, When applying a random mask perturbation to the non-textual modality of the high-collision difficulty example, the following steps are performed: First, generate a binary mask matrix with the same shape as the feature; Secondly, multiply the feature tensor with the mask; Finally, the perturbed samples are input into the model again, and the mean square error between the output value and the original output value is calculated as the consistency loss.

8. The method according to claim 1 or 7, characterized in that, When performing nonlinear fusion on the hidden states that have been aligned over consecutive time periods, the following steps are included: State concatenation: At the end of continuous time evolution, extract the hidden state vectors of the three independent channels of text, acoustic and visual, and concatenate them along the channel dimension. Nonlinear feature interaction: The concatenated vector is input into a multilayer perceptron to capture the nonlinear relationships between modes; Final discrimination and loss constraint: The features processed by the multilayer perceptron are input into the sentiment regression layer, which outputs a continuous scalar value, and a dual loss constraint is designed.

9. The method according to claim 8, characterized in that, The dual loss constraint includes: MSE Loss: Mean Squared Error Loss, used to constrain the numerical distance between the predicted value and the true label; CCC Loss: Consistency Correlation Coefficient Loss, used to constrain the distribution trend of the predicted sequence and the true sequence.

10. A multimodal temporal sentiment consistency recognition system based on continuous-time dynamic modeling, characterized in that, include: The data acquisition and preprocessing module is used to acquire multimodal data, including text sequences, acoustic sequences, and visual sequences; The multimodal data is preprocessed to obtain heterogeneous feature sequences; The feature projection and spatial alignment module is used to construct three independent linear mapping layers to project the heterogeneous feature sequences onto a unified latent spatial dimension, thereby obtaining heterogeneous feature sequences with a unified dimension. The continuous-time dynamic modeling module is used to construct a continuous-time recurrent neural network based on the neural ordinary differential equation. The heterogeneous feature sequence of the same dimension is input into the continuous-time recurrent neural network to perform differential evolution of the hidden state in the continuous time space. The adaptive temporal alignment module is used to dynamically calculate the channel-level time constant vector based on the current input feature vector and hidden state vector through a time constant network, and adaptively adjust the evolution rate of the hidden state to achieve sentiment semantic alignment of different modalities in continuous time. The difficult example mining and fusion discrimination module is used to calculate the cosine similarity between text feature vectors and visual feature vectors. It uses dynamic Top-K filtering to select high-conflict difficult examples, applies random mask perturbation to non-text modalities, and then performs nonlinear fusion on the hidden states that have been aligned in continuous time to output the final sentiment consistency prediction result.

Images